-- LimingChen - 10-Oct-2010

Grid >>>>>>>

prun on grid

source before submit

source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
source /afs/cern.ch/atlas/offline/external/GRID/ddm/DQ2Clients/setup.sh
source /afs/cern.ch/atlas/offline/external/GRID/DA/panda-client/latest/etc/panda/panda_setup.sh
export LFC_HOST=lfc-atlas.cern.ch
export PATHENA_GRID_SETUP_SH=/afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
#export PATH=/afs/cern.ch/sw/lcg/external/sqlite/3.4.0/i686-slc5-gcc43-opt/bin:$PATH
#export PATH=/afs/cern.ch/sw/lcg/external/sqlite/3.4.0/slc3_ia32_gcc323/bin:$PATH
voms-proxy-init -voms atlas -hours 1000 --debug
#From Shu Li

How to kill

kill(JobID, upperJobID=None, useJobsetID=True)
    Kill all subJobs in JobID or JobsetID. 
    Jobsets between JobID and UpperJobID will be killed if upperJobID is given. 
    Set useJobsetID=False if you want to kill JobIDs between JobID and UpperJobID
      >>> kill(15)
      >>> kill(15,20)

How to retry

retry(JobID, newSite=True)
retry(JobID, upperJobID=None, newSite=False, newOpts={}, noSubmit=False, ignoreDuplication=False, useJobsetID=True, retryBuild=False)
      >>> retry(15)
      >>> retry(15,20)
      >>> retry(15,newSite=True)
      >>> retry(15,newSite=True,newOpts={'nGBperJob':10})
      >>> retry(15,ignoreDuplication=True)

How to deal with CANCELLED job

Cannot retry it! resubmit it again, but make sure everthing not is channged.

For help


pbook --gui

It is more useful if you type:

pbook  --gui

Details can be found at: Bookkeeping for Panda analysis jobs

Grid Links

Panda archive output files

When you have a lot of output files on WN, you may want to archive them.

$ prun --outputs "abc.data,JiveXML_*.xml" ...

this will produce DatasetName_SerialNumber_abc.data and DatasetName_SerialNumber_JiveXML_XYZ.xml.tgz.
The latter will contain all JiveXML _*.xml files. You need " or \ to disable shell-globing when the wild-card is used.

Data transfer and delete

  • Data sets delete (the same certification when you did data transfering)

for j in `dq2-ls -L $Site user.limingchen.Slim_Skim_TOP_D3PD.physics_*_Period*V4/`
  echo ""
  echo "j = "$j
  for i in `dq2-list-datasets-container $j`
        echo "i = "$i
        dq2-delete-replicas -d $i $Site

DQ2 Links

1. DQ2 Clients How To

  • Download a single file from a dataset: dq2-get -f FILENAME DATASETNAME
  • Download a sample of n random files from a dataset: dq2-get -n NUMBEROFFILES DATASETNAME
  • Download a dataset from a specific site: dq2-get -s SITE DATASETNAME

  • The location of the dataset: dq2-ls -r DATASET
  • List datasets in a Dataset Container: dq2-list-datasets-container CONTAINERNAME
  • List detail dq2-list-dataset-replicas-container DATASET

  • example: dq2-ls -f data11_7TeV.00182726.physics_Egamma.merge.NTUP_TOPEL.r2603_p659_p694_p722/ | sort
2. The DQ2 Tutorial

3. stderr:Using grid catalog type: UNKNOWN

Lxplus >>>>>>>

Batch Jobs On Lxplus

recover your lxplus files or doucuments

  • cp /afs/cern.ch/ubackup/s/surname/filename $HOME/filename

Change the right of AFS directory permission on lxplus

  • e.g. to make /tmp/lchen readable by others
    • chmod 0755 /tmp/lchen
    • recursively: chmod -R 0755 *

  • But that doesn't let other to read the content of the file, to change that use
    • To see the permission of the local directory fs la
    • Set one directory for one user: fs setacl -dir ~ -acl czhu rl
    • Set one directory for one user: fs setacl -dir ~lchen/myAnalysisCode/ -acl czhu rl
    • Set one directory for any user: fs setacl -dir ~lchen/myAnalysisCode/ -acl system:anyuser rl
    • removes all rights for user "abc1234" in your public_html directory: fs sa ~/public_html abc1234 none
    • recursively for one user: find . -type d -print -exec fs setacl {} czhu rl \;
    • recursively for anyone : find . -type d -print -exec fs setacl {} system:anyuser rl \;
    • recursively for one user: find . -type d -print -exec fs setacl {} czhu none \;
    • recursively for anyone : find . -type d -print -exec fs setacl {} system:anyuser none \;

Copy file on lxplus /tmp

  • ssh
  • Copy a file from lchen@lxplus315.cern.ch:/tmp/lchen to lchen@lxplus415.cern.ch:/tmp/lchen (assume at lxplus415)
  • scp lchen@lxplus315:/tmp/lchen/filename /tmp/lchen
  • scp -r lxplus432:/tmp/czhu/ /tmp/lchen/zhu3

The AtlasSetup environment setup package

Set athena

For users of the sh/zsh/bash shells (i.e. Bourne-like), add the following to the ${HOME}/.shrc, ${HOME}/.zshrc or ${HOME}/.bashrc file:

export AtlasSetup=/afs/cern.ch/atlas/software/dist/AtlasSetup
alias asetup='source $AtlasSetup/scripts/asetup.sh'

In .asetup

[defaults] testArea = <pwd>

1. dq2-get error:

user.jsearcy.mc10_7TeV.105200.T1_McAtNlo_Jimmy.merge.AOD.e598_s933_s946_r1652_r1700.D3PDTest000102.101202103245/user.jsearcy.002238.D3PD._00009.root: external failed:

stderr:Using grid catalog type: UNKNOWN
Using grid catalog : (null)
VO name: atlas
Checksum type: None
Trying SURL srm://dcsrm.usatlas.bnl.gov:8443/srm/managerv2?SFN=/pnfs/usatlas.bnl.gov/atlasuserdisk/user.jsearcy/user.jsearcy.mc10_7TeV.105200.T1_McAtNlo_Jimmy.merge.AOD.e598_s933_s946_r1652_r1700.D3PDTest000102.101202103245_sub013206469/user.jsearcy.002238.D3PD._00009.root ...

[SE][Ls][] httpg://dcsrm.usatlas.bnl.gov:8443/srm/managerv2: CGSI-gSOAP running on lxplus415.cern.ch reports Error reading token data header: Connection closed

lcg_cp: Communication error on send 

You should soruce this (bash)

#source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
source /afs/cern.ch/atlas/offline/external/GRID/ddm/DQ2Clients/setup.sh 
voms-proxy-init -voms atlas

and should not source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh

In zsh, do this

source /afs/cern.ch/atlas/offline/external/GRID/ddm/DQ2Clients/setup.zsh
voms-proxy-init -voms atlas

2. OSError

OSError: [Errno 30] Read-only file system: '/afs/cern.ch/atlas/software/releases/16.7.0/AtlasOffline/16.7.0/InstallArea/python/TrackD3PDMaker'
make[2]: *** [TrackD3PDMakerConfinstall] Error 1
make[1]: *** [TrackD3PDMakerConf] Error 2


it seems that your <pwd> did not make it into your CMTPATH and hence it tries to write to somewhere where it is not allowed.

Can you add "here" to the asetup command (and you need to issue it in the dir where you want to work).

i.e. do:
asetup 16.7.0,here
in this dir:

or somewhere else but not really in the cmt dir of the package (even though you can use that as working dir but it's a bit unusual ...).

AFS directories with name ._afs

  • Question: There are in some AFS directories strange files with names starting with .__afs
  • Example:
    • rm: cannot remove `chen_top//.__afs2EC2': Device or resource busy
    • rm: cannot remove `chen_top//.__afsBBFE': Device or resource busy
  • Answer:
    • These .__afs#### names are created when a file is deleted while another (or the same) process is still writing to it. This is then normally cleaned up at the time the process which had it open closes it. .
    • Some older version of the AFS client had a bug such that the files were not always deleted. This bug had been fixed quite some time ago in the openAFS 1.2.6 client ( today (oct 2009), centrally maintained machines are running version 1.4.8 of openAFS ).
    • If you cannot delete these files, please contact afs.support: afs.support@cern.ch

CRA: change the default shell of lxplus

If you have any problem, you can contact ais.support@cern.ch

ssh is a login shell. .bashrc is not executed for login shells. In order to excute .bashrc in a login shell you need to add [ -f ~/.bashrc ] && . ~/.bashrc in your ~/.bash_profile


  • Convert to PEM Keypair
  • Copy the above PKCS#12 myCert.p12 file to the computer where you will run grid-proxy-init.
  • Extract your certificate (which contains the public key) and the private key:
    • Certificate: openssl pkcs12 -in myCert.p12 -clcerts -nokeys -out $HOME/.globus/usercert.pem
    • To get the encrypted private key: openssl pkcs12 -in myCert.p12 -nocerts -out $HOME/.globus/userkey.pem
  • You must set the mode on your userkey.pem file to read/write only by the owner, otherwise grid-proxy-init will not use it ( chmod go-rw $HOME/.globus/userkey.pem).
  • Delete the myCert.p12 file created above to avoid security issues.
  • renew VO



My Link

atlasmon project: https://svnweb.cern.ch/project/atlasmon/view/

WebSVN Web Interface: https://svnweb.cern.ch/cern/wsvn/atlasmon

Basic command

  • Request a project, Like testWork
  • svn help svn help [command]
  • check out (co) svn co https://svn.cern.ch/reps/testWork/ .
  • add
    • svn add topCSCvsKLFitter
    • svn ci -m "Normal / test / and so on"
  • move
    • svn move name1 name2
    • svn ci -m "rename"
    • Rename a directory svn mv name1 name2
  • remove
    • svn remove topCSCvsKLFitter
    • svn ci -m "Normal / test / and so on"
  • Creat a new directory
    • svn mkdir -m "new dir." https://svn.cern.ch/reps/atlasmon/newdir See here.
  • delete
    • svn delete -m "delete newdir" https://svn.cern.ch/reps/atlasmon/newdir See here.
  • commit (Go to the dir of muonSelection.h)
    • svn diff -r HEAD muonSelection.h
    • svn commit -m "new chnages " muonSelection.h

some links

  • SVN Twiki
  • WorkBookSVN
  • svn/howto.php
  • An example, Download the package from svn. Like here, You can chage the second line according to your package path.
    • export svnoff=svn+ssh://svn.cern.ch/reps/atlasoff
    • svn co $svnoff/PhysicsAnalysis/TopPhys/TopD3PDSelection/trunk

How to make tags from trunk

Before committing any changes back into the repository, you should always first update your local copy to the trunk or the head of the working branch (whichever was specified when checking it out). The command to do this is: svn update

The command to commit your locally checked out changes back to the repository is: svn ci [-m "message"] . If you don't specify a message using the -m "message" option, the editor will be activated.

The recommended command to apply a tag to your package is:

svn cp $SVNROOT/<path-to-package>/<package>/[trunk|branches/<branch-tag>] -r <rev> $SVNROOT/<path-to-package>/<package>/tags/<tag> -m "message"

  • Where:
    • <path-to-package>/ is the directory hierarchy to your package (which might be empty but is MyTopLevel/MyDirectory/ in this example)
    • <package> is the name of your package ( MyPackage in this example)
    • [trunk|branches/<branch-tag>] specify either trunk or branches/<branch-tag> depending on whether your commit was made to the trunk or to a branch.
    • -r <rev> denotes the revision number that was displayed from your previous commit command.
    • tags/ is the tags directory within your package (which is reserved for this use)
    • <tag> is the new package tag (e.g. MyPackage-01-02-03)
    • -m "message" is an optional message. If you don't supply this, the editor will be activated.
    • example
         svn update
         svn ci -m "test"
         svn cp svn+ssh://svn.cern.ch/reps/atlasmon/ObjectSelectionD3PD/MuonSelectionD3PD/trunk svn+ssh://svn.cern.ch/reps/atlasmon/ObjectSelectionD3PD/MuonSelectionD3PD/tags/MuonSelectionD3PD-00-00-01 -m "The first tag SF only"


  • This is an example of how to cooperate in code developing with "svn merge"

  • Downloading JetSelectionD3PD-00-00-06 to your working copy JetSelectionD3PD
  • if user wants to modify this package:
    • cd JetSelectionD3PD
    • svn import -m"start of JetSelectionD3PD-00-00-06 branch JetSelectionD3PD-01" svn+ssh://svn.cern.ch/reps/atlasmon/JetSelectionD3PD/branches/JetSelectionD3PD-01 //e.g it is rev 100
    • svn switch svn+ssh://svn.cern.ch/reps/atlasmon/JetSelectionD3PD/branches/JetSelectionD3PD-01
    • svn ci -m"...."
  • I will merge your modification into my JetSelectionD3PD/trunk
    • cd JetSelectionD3PD // this is my developing copy which may also changed a lot
    • svn merge svn+ssh://svn.cern.ch/reps/atlasmon/JetSelectionD3PD/branches/JetSelectionD3PD-01@100 svn+ssh://svn.cern.ch/reps/atlasmon/JetSelectionD3PD/branches/JetSelectionD3PD-01
    • If no conflicts ( both didn't modify the same line, it's OK, otherwise you can choose option (p), and edit after merging.
    • svn ci -m"dev"
    • make new tag
  • be careful, for each new downloaded tagged package, you should make a new branch, the old branch will be discarded.


  • errors when checksum mismatch for file ...
  • solution:
    • backup file
    • svn delete --force file
    • cp file back
    • svn add file

  • error message in commit:
svn: Commit failed (details follow):
svn: Directory '/tmp/czhu/forR16Data_4/externalPac/RootCore/python/.svn' containing working copy admin area is missing

  • solution
    • mv it out and svn delete the directory
    • mv it back and svn add

  • 2
svn: Commit failed (details follow):
svn: No repository found in 'svn+ssh://svn.cern.ch/reps'

  • solution no very good solution now svn diff to find which files need to be committed svn ci XXXXfile -m"" to check in the files one by one. remove this working copy and re-checkout all the copy.


svn: Directory 'externalPac/RootCore/python/.svn' containing working copy admin area is missing

  • solution rm externalPac/RootCore/python svn del --force externalPac/RootCore/python
but this will induce the problem 2 above.

  • 4
Transmitting file data .svn: Commit failed (details follow):
svn: Commit blocked by pre-commit hook (exit code 1) with output:
Some parts of your commit look suspiciously like merge
conflict markers.  Please double-check your diff and try
committing again.

  • solution some contents of the files are rejected by svn. you can commit files one by one to see which file does match the problem


  • Permissions on your personal area
    • eos chmod -r 750 /eos/atlas/user/l/login makes the directory readable by the whole ATLAS group, but not by other users
    • eos attr set user.acl="u::rx" /eos/atlas/user/l/login give access to any of your directories to specific users

  • Getting your quota on EOS:
    • eos quota | grep -A 4 "Quota Node: /eos/atlas/user/$" | head -5

  • How to list a directory in EOS:
    • eos ls [-l] /eos/atlas/user/l/lchen

  • Copy file to EOS:
    • xrdcp local.file root://eosatlas//eos/atlas/user/l/lchen/

  • Copy directory to EOS
    • xrdcp -f -R test/ root://eosatlas//eos/atlas/user/l/lchen/

  • Reading files from your EOS:
    • xrdcp root://eosatlas//eos/atlas/user/l/lchen/file.name .
    • TFile *file = TFile::Open("root://eosatlas//eos/atlas/user/[l]/[login]/file.name");

  • xrdcp will create automatically al non-existing directories specified in your path:
    • xrdcp local.file root://eosatlas//eos/atlas/user/[l]/[login]/dir1/dir2/dest_file

  • If you just need to create an empty directory, you can choose two options:
    • xrd eosatlas mkdir /eos/atlas/user/l/lchendir1/dir2/...
    • eos mkdir /eos/atlas/user/l/lchen/dir1/dir2/...

  • Deleting files and directories on EOS
    • Delete files: eos rm /eos/atlas/.../filename
    • Delete directories eos rm -r /eos/atlas/.../directory_to_remove

  • Don't use mv (move) on eos
      "Moving" files or directories on EOS is NOT supported.
      You can copy from one plece to another, you can delete but you cannot move.
      This is like this by design: moving would imply an actual recalculation of the accounting
      (which is based on paths in EOS) and this would crash the system.

  • Go to EOS:
      [lxplus439] /afs/cern.ch/user/l/lchen/Dilep_cutflow/mySrc > eos
      EOS Console [root://eosatlas.cern.ch] |/> cd /eos/atlas/user/l/lchen
      EOS Console [root://eosatlas.cern.ch] |/eos/atlas/user/l/lchen/> ls
      EOS Console [root://eosatlas.cern.ch] |/eos/atlas/user/l/lchen/> .q

  • mount EOS
      You can also mount EOS on your filesystem and copy there with xrdcp or 
      dq2-get (writing on a mounted filesystem is somewhat slower than xrdcp 
      on a remote server, but for small datasets it's really handy):
      [gnegri@lxplus310 ~] mkdir eos
      [gnegri@lxplus310 ~] eosmount eos
      [gnegri@lxplus310 ~] dq2-get -H eos/atlas/user/l/lchen/dest_dir/<dataset_name>
      Like this, you don't have to care wher your dataset is on the grid (dq2-get will care for you).

  • EOS help: eos help

Castor is outdate (15/09/2011)

  • ordinary /castor/cern.ch/user/l/lchen
  • grid one /castor/cern.ch//grid/atlas/users/limingchen

    • xrdcp ~/myAnalysisCode/mmega2008.tar.gz root://castoratlas//castor/cern.ch/user/c/czhu/code

  • access castor from lxplus: copy dataset from CERN-PROD_SCRATCHDISK:
    • xrdcp root://castoratlas//castor/cern.ch/grid/atlas/atlasscratchdisk/user10/chengguangzhu .
    • dq2-get --site=CERN-PROD_SCRATCHDISK user.zhuchengguang.cint_201012130938/

  • How to read: root://castoratlas//castor/cern.ch/user/l/lchen

  • list
    • nsls -l /castor/cern.ch/user/l/lchen
    • rfdir /castor/cern.ch/user/l/lchen
    • nsls -T /castor/cern.ch/user/l/lchen/private

  • Removing a CASTOR file
    • nsrm or rfrm commands. Both commands support the -r flag for recursively removing a whole directory tree
    • rfrm /castor/cern.ch/user/l/lchen/AtlFast_ttbar.ntuple.root
    • nsrm -r /castor/cern.ch/user/l/lchen/user

  • New directoriescreated: nsmkdir or rfmkdir commands. Both support the -p option for recursive creation of non-existing directories.
    • nsmkdir /castor/cern.ch/user/l/lchen/KLFitterTestRoot
    • nsmkdir -p /castor/cern.ch/user/l/lchen/higgs/muons // Do not forget -p if higgs/ file does not exit

  • renamed and/or moved between directories using the nsrename or rfrename commands

Others >>>>>>>

Edit | Attach | Watch | Print version | History: r66 < r65 < r64 < r63 < r62 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r66 - 2018-05-06 - KnutZoch
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback