Pilot is started by APF wrapper( You can get a copy of the wrapper. Here is an example to run pilot.

Run pilot with ES
Here is an example I run ES on lxplus. Before run ES, you may need to define some ES tasks( on one site. Then you can run on your local tasks without affecting prod tasks.
           cd /tmp
           dir=$(mktemp -d)
           cd $dir
           echo $dir

           export ATLAS_LOCAL_ROOT_BASE=/cvmfs/
           source ${ATLAS_LOCAL_ROOT_BASE}/user/

           source /cvmfs/

           export OSG_GRID=/cvmfs/
           export VO_ATLAS_SW_DIR=/cvmfs/
           source $OSG_GRID/

           export RUCIO_ACCOUNT=$USER
           rm -f pilot.tar.gz
           # prod dev pilot           
           #wget -O pilot.tar.gz
           # pilot release candidate(pre-prod pilot)
           #wget -O pilot.tar.gz
           # prod pilot
           wget -O pilot.tar.gz
           tar xzf pilot.tar.gz

           #python -s BNL_PROD -h BNL_PROD-condor  -w -p 25443 -d /usatlas/u/wguan/panda/test -u ptest
           #python -s IN2P3-LPSC_CLOUD_MCORE -h IN2P3-LPSC_CLOUD_MCORE -w -p 25443 -d /tmp/wguan/test2
           #python -s $site -h $queue -w -p 25443 -d $dir -v AP_VALI

           # "-u ptest" is special option to test pilots. When a task is defined with 'prodSourceLabel: ptest', prod pilot will not run it. Then we can use our local pilot with this option to run these tasks.
           python -s $site -h $queue -w -p 25443 -d $dir -u ptest

Run pilot without wrapper
Here is an example.
           cd /tmp/wguan
           export OSG_GRID=/cvmfs/
           export VO_ATLAS_SW_DIR=/cvmfs/
           source $OSG_GRID/

           #export COPYTOOL=gfal-copy
           #export COPYTOOLIN=gfal-copy
           #export RUCIO_ACCOUNT=wguan

           rm -f pilot.tar.gz
           wget -O pilot.tar.gz
           tar xzf pilot.tar.gz

           #python -s NERSC_Edison -h NERSC_Edison  -w -p 25443 -d /scratch2/scratchdirs/wguan/Edison/hpcf/edison02/wguan-pilot-dev-HPC_merge/ -N 2 -Q debug -u ptest
            python -s NERSC_Edison -h NERSC_Edison  -w -p 25443 -d /global/cscratch1/sd/wguan/pilot/hpcf/cori19/test1 -N 6 -Q debug

Run pilot example
Here is an example to start pilot on NERSC Edision.
            #cat /project/projectdirs/atlas/pilot/
           # if cvmfs
           export OSG_GRID=/cvmfs/
           export VO_ATLAS_SW_DIR=/cvmfs/
            # NERSC
            # export OSG_GRID=/global/project/projectdirs/atlas/pilot/grid_env
            # export VO_ATLAS_SW_DIR=/project/projectdirs/atlas

            rm -f
            wget -O

            chmod +x

            # export COPYTOOL=gfal-copy
            # export COPYTOOLIN=gfal-copy

            ./  --wrapperloglevel=debug --wrappergrid=OSG --wrapperwmsqueue=ANALY_WISC_ATLAS --wrapperbatchqueue=ANALY_WISC_ATLAS --wrappervo=ATLAS --wrappertarballurl=  --wrapperpilotcodeurl= --wrapperpilotcode=pilotcode-PICARD,pilotcode-rc  -s ANALY_WISC_ATLAS -q ANALY_WISC_ATLAS  -w -p 25443 -d /tmp/wguan/

In this example, settting up COPYTOOLIN and COPYTOOLIN environment is for staging input files from SE and staging out output log files to SE. I only tested gfal site mover on HPC cray system. " -d /scratch2/scratchdirs/wguan/Edison" is the pilot working directory. You need to change it to your writable directory.

Run pilot example 2
Here is an example to start pilot using, you can try it on Lxplus.
           export OSG_GRID=/cvmfs/
           export VO_ATLAS_SW_DIR=/cvmfs/
            rm -f
            wget -O

            chmod +x

            # define pilot code. If it's not defined, default one will be used.
            # export PILOT_HTTP_SOURCES=

            ./  -s CERN-P1 -h CERN-P1-OpenStack -p 25443 -w  

Run pilot on Grid
In ATLAS we are using condor to submit pilots to Grid. From this link you can find the production apf condor machines. There is another machine aipanda020 which is used for developing(pilot, apf). Here is an example on aipanda020.
           [wguan@aipanda020 RemoteTests]$ cat Logs/gfal-copy/FZK-LCG2-all-prod-CEs/submit.jdl 
           CodeDir          = /data/wguan/Panda/pilot/RemoteTests
           LogDir           = /data/wguan/Panda/pilot/RemoteTests/Logs/gfal-copy/FZK-LCG2-all-prod-CEs
           notify_user      =
           grid_resource    = cream sge sl6
           environment      = " COPYTOOL=gfal-copy "
           x509userproxy = /tmp/x509up_u23959_EU
           executable       = $(CodeDir)/wrapper/
           arguments        = -s FZK-LCG2 -h FZK-LCG2-all-prod-CEs -p 25443 -w -j false -k 3500 -u ptest
           transfer_input_files = $(CodeDir)/wrapper/
           #globusrsl       = (jobtype=single)(queue=paul_test_q)
           globusrsl        = (jobtype=single)
           #+RACF_Group = "dq2test"
           universe         = grid
           output           = $(LogDir)/$(Cluster).$(Process).output
           error            = $(LogDir)/$(Cluster).$(Process).error
           log              = $(LogDir)/$(Cluster).$(Process).log
           stream_output    = False
           stream_error     = False
           notification     = Error
           transfer_executable = True
           should_transfer_files = YES
           when_to_transfer_output = ON_EXIT_OR_EVICT
           periodic_hold    = GlobusResourceUnavailableTime =!= UNDEFINED &&(CurrentTime-GlobusResourceUnavailableTime>30)
           periodic_remove = (JobStatus == 5 && (CurrentTime - EnteredCurrentStatus) > 3600) || (JobStatus == 1 && globusstatus =!= 1 && (CurrentTime - EnteredCurrentStatus) > 86400)
           +Nonessential = True
           copy_to_spool = false

           globusrsl = (=atlasXL) 
           globusrsl = (jobtype=single)(=atlasXL)  
           queue 1

-- WenGuan - 2015-02-24
Topic attachments
I Attachment History Action Size Date Who Comment
Unix shell scriptsh r1 manage 0.7 K 2018-08-16 - 10:15 WenGuan  
Unix shell scriptsh r1 manage 2.4 K 2018-08-16 - 10:15 WenGuan  
Unix shell scriptsh r1 manage 2.5 K 2018-08-16 - 10:15 WenGuan  
Edit | Attach | Watch | Print version | History: r7 < r6 < r5 < r4 < r3 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r7 - 2018-08-16 - WenGuan
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    PanDA All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback