-- LimingChen - 10-Oct-2010

Grid >>>>>>>

prun on grid

source before submit

source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
source /afs/cern.ch/atlas/offline/external/GRID/ddm/DQ2Clients/setup.sh
source /afs/cern.ch/atlas/offline/external/GRID/DA/panda-client/latest/etc/panda/panda_setup.sh
export LFC_HOST=lfc-atlas.cern.ch
export PATHENA_GRID_SETUP_SH=/afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
#export PATH=/afs/cern.ch/sw/lcg/external/sqlite/3.4.0/i686-slc5-gcc43-opt/bin:$PATH
#export PATH=/afs/cern.ch/sw/lcg/external/sqlite/3.4.0/slc3_ia32_gcc323/bin:$PATH
voms-proxy-init -voms atlas -hours 1000 --debug
#From Shu Li

How to kill

kill(JobID, upperJobID=None, useJobsetID=True)
    Kill all subJobs in JobID or JobsetID. 
    Jobsets between JobID and UpperJobID will be killed if upperJobID is given. 
    Set useJobsetID=False if you want to kill JobIDs between JobID and UpperJobID
    
    example:
      >>> kill(15)
      >>> kill(15,20)

How to retry

retry(JobID)
retry(JobID, newSite=True)
retry(JobID, upperJobID=None, newSite=False, newOpts={}, noSubmit=False, ignoreDuplication=False, useJobsetID=True, retryBuild=False)
    example:
      >>> retry(15)
      >>> retry(15,20)
      >>> retry(15,newSite=True)
      >>> retry(15,newSite=True,newOpts={'nGBperJob':10})
      >>> retry(15,ignoreDuplication=True)

How to deal with CANCELLED job

Cannot retry it! resubmit it again, but make sure everthing not is channged.

For help

pbook
help()
help(retry)

pbook --gui

It is more useful if you type:

pbook  --gui

Details can be found at: Bookkeeping for Panda analysis jobs

Grid Links

• atlas VO Registration
• Re-sign the Grid and VO AUPs: Note the nickname(limingchen).

Panda archive output files

When you have a lot of output files on WN, you may want to archive them.

$ prun --outputs "abc.data,JiveXML_*.xml" ...

this will produce DatasetName_SerialNumber_abc.data and DatasetName_SerialNumber_JiveXML_XYZ.xml.tgz.
The latter will contain all JiveXML _*.xml files. You need " or \ to disable shell-globing when the wild-card is used.

• How to submit Athena jobs to Panda
• How to submit ROOT/general jobs to Panda
• Panda monitor

Data transfer and delete

• Data trnasfer
  • DaTRI: Data_Transfer_Request

• Data sets delete (the same certification when you did data transfering)

#!/bin/sh

Site=IN2P3-CPPM_LOCALGROUPDISK
for j in `dq2-ls -L $Site user.limingchen.Slim_Skim_TOP_D3PD.physics_*_Period*V4/`
do 
  echo ""
  echo "j = "$j
  for i in `dq2-list-datasets-container $j`
  do
        echo "i = "$i
        dq2-delete-replicas -d $i $Site
  done
done 

DQ2 Links

1. DQ2 Clients How To

• Download a single file from a dataset: dq2-get -f FILENAME DATASETNAME
• Download a sample of n random files from a dataset: dq2-get -n NUMBEROFFILES DATASETNAME
• Download a dataset from a specific site: dq2-get -s SITE DATASETNAME

• The location of the dataset: dq2-ls -r DATASET
• List datasets in a Dataset Container: dq2-list-datasets-container CONTAINERNAME
• List detail dq2-list-dataset-replicas-container DATASET

• example: dq2-ls -f data11_7TeV.00182726.physics_Egamma.merge.NTUP_TOPEL.r2603_p659_p694_p722/ | sort

2. The DQ2 Tutorial

3. stderr:Using grid catalog type: UNKNOWN

Lxplus >>>>>>>

Batch Jobs On Lxplus

recover your lxplus files or doucuments

• cp /afs/cern.ch/ubackup/s/surname/filename $HOME/filename

Change the right of AFS directory permission on lxplus

• e.g. to make /tmp/lchen readable by others
  • chmod 0755 /tmp/lchen
  • recursively: chmod -R 0755 *

• But that doesn't let other to read the content of the file, to change that use
  • To see the permission of the local directory fs la
  • Set one directory for one user: fs setacl -dir ~ -acl czhu rl
  • Set one directory for one user: fs setacl -dir ~lchen/myAnalysisCode/ -acl czhu rl
  • Set one directory for any user: fs setacl -dir ~lchen/myAnalysisCode/ -acl system:anyuser rl
  • removes all rights for user "abc1234" in your public_html directory: fs sa ~/public_html abc1234 none
  • recursively for one user: find . -type d -print -exec fs setacl {} czhu rl \;
  • recursively for anyone : find . -type d -print -exec fs setacl {} system:anyuser rl \;
  • recursively for one user: find . -type d -print -exec fs setacl {} czhu none \;
  • recursively for anyone : find . -type d -print -exec fs setacl {} system:anyuser none \;

• Reference 1
• Reference 2

Copy file on lxplus /tmp

• ssh 137.138.210.232
• Copy a file from lchen@lxplus315.cern.ch:/tmp/lchen to lchen@lxplus415.cern.ch:/tmp/lchen (assume at lxplus415)
• scp lchen@lxplus315:/tmp/lchen/filename /tmp/lchen
• scp -r lxplus432:/tmp/czhu/ /tmp/lchen/zhu3

The AtlasSetup environment setup package

Set athena

For users of the sh/zsh/bash shells (i.e. Bourne-like), add the following to the ${HOME}/.shrc, ${HOME}/.zshrc or ${HOME}/.bashrc file:

export AtlasSetup=/afs/cern.ch/atlas/software/dist/AtlasSetup
alias asetup='source $AtlasSetup/scripts/asetup.sh'

In .asetup

[defaults] testArea = <pwd>

1. dq2-get error:

user.jsearcy.mc10_7TeV.105200.T1_McAtNlo_Jimmy.merge.AOD.e598_s933_s946_r1652_r1700.D3PDTest000102.101202103245/user.jsearcy.002238.D3PD._00009.root: external failed:

stdout:
stderr:Using grid catalog type: UNKNOWN
Using grid catalog : (null)
VO name: atlas
Checksum type: None
Trying SURL srm://dcsrm.usatlas.bnl.gov:8443/srm/managerv2?SFN=/pnfs/usatlas.bnl.gov/atlasuserdisk/user.jsearcy/user.jsearcy.mc10_7TeV.105200.T1_McAtNlo_Jimmy.merge.AOD.e598_s933_s946_r1652_r1700.D3PDTest000102.101202103245_sub013206469/user.jsearcy.002238.D3PD._00009.root ...

[SE][Ls][] httpg://dcsrm.usatlas.bnl.gov:8443/srm/managerv2: CGSI-gSOAP running on lxplus415.cern.ch reports Error reading token data header: Connection closed

lcg_cp: Communication error on send 

You should soruce this (bash)

#source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
source /afs/cern.ch/atlas/offline/external/GRID/ddm/DQ2Clients/setup.sh 
voms-proxy-init -voms atlas
#https://twiki.cern.ch/twiki/bin/view/Main/HowToRun

and should not source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh

In zsh, do this

source /afs/cern.ch/atlas/offline/external/GRID/ddm/DQ2Clients/setup.zsh
voms-proxy-init -voms atlas

2. OSError

OSError: [Errno 30] Read-only file system: '/afs/cern.ch/atlas/software/releases/16.7.0/AtlasOffline/16.7.0/InstallArea/python/TrackD3PDMaker'
make[2]: *** [TrackD3PDMakerConfinstall] Error 1
make[1]: *** [TrackD3PDMakerConf] Error 2

Answer

it seems that your <pwd> did not make it into your CMTPATH and hence it tries to write to somewhere where it is not allowed.

Can you add "here" to the asetup command (and you need to issue it in the dir where you want to work).

i.e. do:
asetup 16.7.0,here
in this dir:
/afs/cern.ch/user/l/lchen/
not
/afs/cern.ch/user/l/lchen/D3PDMaker/PhysicsAnalysis/D3PDMaker/TrackD3PDMaker/cmt

or somewhere else but not really in the cmt dir of the package (even though you can use that as working dir but it's a bit unusual ...).

AFS directories with name ._afs

• Question: There are in some AFS directories strange files with names starting with .__afs
• Example:
  • rm: cannot remove `chen_top//.__afs2EC2': Device or resource busy
  • rm: cannot remove `chen_top//.__afsBBFE': Device or resource busy
• Answer:
  • These .__afs#### names are created when a file is deleted while another (or the same) process is still writing to it. This is then normally cleaned up at the time the process which had it open closes it. .
  • Some older version of the AFS client had a bug such that the files were not always deleted. This bug had been fixed quite some time ago in the openAFS 1.2.6 client ( today (oct 2009), centrally maintained machines are running version 1.4.8 of openAFS ).
  • If you cannot delete these files, please contact afs.support: afs.support@cern.ch

CRA: change the default shell of lxplus

If you have any problem, you can contact ais.support@cern.ch

ssh is a login shell. .bashrc is not executed for login shells. In order to excute .bashrc in a login shell you need to add [ -f ~/.bashrc ] && . ~/.bashrc in your ~/.bash_profile

CA at CERN

• Convert to PEM Keypair
• Copy the above PKCS#12 myCert.p12 file to the computer where you will run grid-proxy-init.
• Extract your certificate (which contains the public key) and the private key:
  • Certificate: openssl pkcs12 -in myCert.p12 -clcerts -nokeys -out $HOME/.globus/usercert.pem
  • To get the encrypted private key: openssl pkcs12 -in myCert.p12 -nocerts -out $HOME/.globus/userkey.pem
• You must set the mode on your userkey.pem file to read/write only by the owner, otherwise grid-proxy-init will not use it ( chmod go-rw $HOME/.globus/userkey.pem).
• Delete the myCert.p12 file created above to avoid security issues.
• renew VO

• CPPM/CNRS
  • revoke address https://igc.services.cnrs.fr/usercert/?CA=GRID2-FR&lang=fr
  • Convert a certificate from PEM format

SVN

Notice:

• If the web is: https://svnweb.cern.ch/trac/atlasoff/browser/ export svnoff=svn+ssh://svn.cern.ch/reps/atlasoff
• if the web is: https://svnweb.cern.ch/reps/atlasgrp/browser/ export svnroot=svn+ssh://svn.cern.ch/reps/atlasgrp

My Link

atlasmon project: https://svnweb.cern.ch/project/atlasmon/view/

WebSVN Web Interface: https://svnweb.cern.ch/cern/wsvn/atlasmon

Basic command

• Request a project, Like testWork
• svn help svn help [command]
• check out (co) svn co https://svn.cern.ch/reps/testWork/ .
• add
  • svn add topCSCvsKLFitter
  • svn ci -m "Normal / test / and so on"
• move
  • svn move name1 name2
  • svn ci -m "rename"
  • Rename a directory svn mv name1 name2
• remove
  • svn remove topCSCvsKLFitter
  • svn ci -m "Normal / test / and so on"
• Creat a new directory
  • svn mkdir -m "new dir." https://svn.cern.ch/reps/atlasmon/newdir See here.
• delete
  • svn delete -m "delete newdir" https://svn.cern.ch/reps/atlasmon/newdir See here.
• commit (Go to the dir of muonSelection.h)
  • svn diff -r HEAD muonSelection.h
  • svn commit -m "new chnages " muonSelection.h

some links

• SVN Twiki
• WorkBookSVN
• svn/howto.php
• An example, Download the package from svn. Like here, You can chage the second line according to your package path.
  • export svnoff=svn+ssh://svn.cern.ch/reps/atlasoff
  • svn co $svnoff/PhysicsAnalysis/TopPhys/TopD3PDSelection/trunk

How to make tags from trunk

• How to Commit changes and Tag them

Before committing any changes back into the repository, you should always first update your local copy to the trunk or the head of the working branch (whichever was specified when checking it out). The command to do this is: svn update

The command to commit your locally checked out changes back to the repository is: svn ci [-m "message"] . If you don't specify a message using the -m "message" option, the editor will be activated.

The recommended command to apply a tag to your package is:

svn cp $SVNROOT/<path-to-package>/<package>/[trunk|branches/<branch-tag>] -r <rev> $SVNROOT/<path-to-package>/<package>/tags/<tag> -m "message"

• Where:
  • <path-to-package>/ is the directory hierarchy to your package (which might be empty but is MyTopLevel/MyDirectory/ in this example)
  • <package> is the name of your package ( MyPackage in this example)
  • [trunk|branches/<branch-tag>] specify either trunk or branches/<branch-tag> depending on whether your commit was made to the trunk or to a branch.
  • -r <rev> denotes the revision number that was displayed from your previous commit command.
  • tags/ is the tags directory within your package (which is reserved for this use)
  • <tag> is the new package tag (e.g. MyPackage-01-02-03)
  • -m "message" is an optional message. If you don't supply this, the editor will be activated.
  • example

         svn update
         svn ci -m "test"
         svn cp svn+ssh://svn.cern.ch/reps/atlasmon/ObjectSelectionD3PD/MuonSelectionD3PD/trunk svn+ssh://svn.cern.ch/reps/atlasmon/ObjectSelectionD3PD/MuonSelectionD3PD/tags/MuonSelectionD3PD-00-00-01 -m "The first tag SF only"

merge

• This is an example of how to cooperate in code developing with "svn merge"

• Downloading JetSelectionD3PD-00-00-06 to your working copy JetSelectionD3PD
• if user wants to modify this package:
  • cd JetSelectionD3PD
  • svn import -m"start of JetSelectionD3PD-00-00-06 branch JetSelectionD3PD-01" svn+ssh://svn.cern.ch/reps/atlasmon/JetSelectionD3PD/branches/JetSelectionD3PD-01 //e.g it is rev 100
  • svn switch svn+ssh://svn.cern.ch/reps/atlasmon/JetSelectionD3PD/branches/JetSelectionD3PD-01
  • svn ci -m"...."
• I will merge your modification into my JetSelectionD3PD/trunk
  • cd JetSelectionD3PD // this is my developing copy which may also changed a lot
  • svn merge svn+ssh://svn.cern.ch/reps/atlasmon/JetSelectionD3PD/branches/JetSelectionD3PD-01@100 svn+ssh://svn.cern.ch/reps/atlasmon/JetSelectionD3PD/branches/JetSelectionD3PD-01
  • If no conflicts ( both didn't modify the same line, it's OK, otherwise you can choose option (p), and edit after merging.
  • svn ci -m"dev"
  • make new tag
• be careful, for each new downloaded tagged package, you should make a new branch, the old branch will be discarded.

problem

• errors when checksum mismatch for file ...
• solution:
  • backup file
  • svn delete --force file
  • cp file back
  • svn add file

• error message in commit:

svn: Commit failed (details follow):
svn: Directory '/tmp/czhu/forR16Data_4/externalPac/RootCore/python/.svn' containing working copy admin area is missing

• solution
  • mv it out and svn delete the directory
  • mv it back and svn add

• 2

svn: Commit failed (details follow):
svn: No repository found in 'svn+ssh://svn.cern.ch/reps'

• solution no very good solution now svn diff to find which files need to be committed svn ci XXXXfile -m"" to check in the files one by one. remove this working copy and re-checkout all the copy.

*3

svn: Directory 'externalPac/RootCore/python/.svn' containing working copy admin area is missing

• solution rm externalPac/RootCore/python svn del --force externalPac/RootCore/python

but this will induce the problem 2 above.

• 4

Transmitting file data .svn: Commit failed (details follow):
svn: Commit blocked by pre-commit hook (exit code 1) with output:
Some parts of your commit look suspiciously like merge
conflict markers.  Please double-check your diff and try
committing again.

• solution some contents of the files are rejected by svn. you can commit files one by one to see which file does match the problem

EOS

• Permissions on your personal area
  • eos chmod -r 750 /eos/atlas/user/l/login makes the directory readable by the whole ATLAS group, but not by other users
  • eos attr set user.acl="u::rx" /eos/atlas/user/l/login give access to any of your directories to specific users

• Getting your quota on EOS:
  • eos quota | grep -A 4 "Quota Node: /eos/atlas/user/$" | head -5

• How to list a directory in EOS:
  • eos ls [-l] /eos/atlas/user/l/lchen

• Copy file to EOS:
  • xrdcp local.file root://eosatlas//eos/atlas/user/l/lchen/

• Copy directory to EOS
  • xrdcp -f -R test/ root://eosatlas//eos/atlas/user/l/lchen/

• Reading files from your EOS:
  • xrdcp root://eosatlas//eos/atlas/user/l/lchen/file.name .
  • TFile *file = TFile::Open("root://eosatlas//eos/atlas/user/[l]/[login]/file.name");

• xrdcp will create automatically al non-existing directories specified in your path:
  • xrdcp local.file root://eosatlas//eos/atlas/user/[l]/[login]/dir1/dir2/dest_file

• If you just need to create an empty directory, you can choose two options:
  • xrd eosatlas mkdir /eos/atlas/user/l/lchendir1/dir2/...
  • eos mkdir /eos/atlas/user/l/lchen/dir1/dir2/...

• Deleting files and directories on EOS
  • Delete files: eos rm /eos/atlas/.../filename
  • Delete directories eos rm -r /eos/atlas/.../directory_to_remove

• Don't use mv (move) on eos

      "Moving" files or directories on EOS is NOT supported.
      You can copy from one plece to another, you can delete but you cannot move.
      This is like this by design: moving would imply an actual recalculation of the accounting
      (which is based on paths in EOS) and this would crash the system.

• Go to EOS:

      [lxplus439] /afs/cern.ch/user/l/lchen/Dilep_cutflow/mySrc > eos
      EOS Console [root://eosatlas.cern.ch] |/> cd /eos/atlas/user/l/lchen
      EOS Console [root://eosatlas.cern.ch] |/eos/atlas/user/l/lchen/> ls
      data11_7TeV.00180710.physics_Egamma_E
      mc10_7TeV.105200.T1_McAtNlo_Jimmy.merge.NTUP_TOP.e598_s933_s946_r2302_r2300_p572
      EOS Console [root://eosatlas.cern.ch] |/eos/atlas/user/l/lchen/> .q

• mount EOS

      You can also mount EOS on your filesystem and copy there with xrdcp or 
      dq2-get (writing on a mounted filesystem is somewhat slower than xrdcp 
      on a remote server, but for small datasets it's really handy):
      [gnegri@lxplus310 ~] mkdir eos
      [gnegri@lxplus310 ~] eosmount eos
      [...]
      [gnegri@lxplus310 ~] dq2-get -H eos/atlas/user/l/lchen/dest_dir/<dataset_name>
      Like this, you don't have to care wher your dataset is on the grid (dq2-get will care for you).

• EOS help: eos help

Castor is outdate (15/09/2011)

Others >>>>>>>

• expansion of lxpuls

• e group e-groups.cern.ch
• LSF problem