-- VitaliChoutko - 19-Apr-2011

-- BaosongShan - 19-Feb-2013

Batch Jobs Submission

On ams local cluster (Since Feb. 2013)


All non-interactive jobs should be submitted only via PBS system. Please use man qsub and refs wherein to control jobs submission.

The public queues include:

q8mn (default)
To minimize the efforts on the user side of the migration from LSF to PBS, there are some wrappers written to simulate the LSF commands:

Please always specify number of threads of your job by

qsub -l nodes=1:ppn=NT
bsub -n NT -R "span[hosts=1]" 
option when submitting multi-thread (NT == number_of_threads) jobs.

In general, please refrain of running jobs with NT > 4 .

Data Access

root & raw files access

via local disks DisksNameSOC



User Disk space


Scratch Area


Conditiononal DataBase (aka $AMSDataDir)


/cvmfs/ams.cern.ch/Offline/AMSDataDir (ReadOnly, Recommended)

on lxplus (recommended)

ssh user@lxplus

All noninteractive jobs should be submitted only via lsf. Please use man bsub and refs wherein to control jobs submission

Please always specify number of threads of your job by bsub -n NT -R "span[hosts=1]" option when submitting multi-thread (NT == number_of_threads) jobs.

In general, please refrain of running jobs with NT > 4 .

Data Access

Recommended File Access Mode for castor

After some discussion with our castor support, we would like everybody restricts their mode to access castor files by the following procedure:

1. Create a filelist of files to be accessed. An example can be found at /afs/cern.ch/work/c/choutko/txt/filelistexample.txt. Note there is no wildcards in the file list.

2. Access files from castor by using AMSChain::AddFromFile function only.

  /// Add TFiles to the chain from a text file 

   \brief Conditionally add the TFiles from the filelist to an  amschain
\param[in]  rootfilelist  a (txt) file with full path names of the files to be chained. example is in /afs/cern.ch/work/c/choutko/txt/filelistexample.txt

\param[in] beg first  file line  (started from 0) to be chained

\param[in] end next to the last file line to be chained

\param[in] stagedonly  if true only take staged files if files are on castor.  The rejected filenames will be written in the rootfilelist_beg_end_STAGEIN file

\param[in] pattern if not NULL only process files with names matched with pattern

\retval -1 if rootfilelist can not be open
\retval 0  otherwise


  int AddFromFile(const char* rootfilelist,int beg=0,int end=INT_MAX, bool stagedonly=false,char *pattern=0);

root file access

To avoid getting castor stuck, please kindly try to avoid using full pass4 data set unless absolutely necessarily.

Instead, please use selected streams.

Please promptly report any problems with selected streams if you think you need to access to full pass4 dataset.

Please also ensure you minimize your i/o rate per job as much as possible by

1. using fStatus branch

2. rejection events on early stages using nXYZ() functions from AMSEventR

Also to avoid castor stuck, from Feb. 2013, rfio should be completely AVOIDED in accessing castor. Instead, you should use xrootd.

xrootd: /afs/cern.ch/ams/Offline/vdev/doc/xrootd.doc

see also CastorXrootd

raw files access ( in gbatch only)

  • export NtupleDir = {scratch area }

  • export TransferRawBy="/afs/cern.ch/ams/Offline/root/Linux/527.icc64/bin/xrdcp -f root://castorpublic.cern.ch//"

  • export TransferSharedLib=/afs/cern.ch/ams/Offline/root/Linux/527.gcc64/lib:/afs/cern.ch/ams/local/castor/lib

User Disk space

Home afs directories (up to 2 Gb)

2 Tb user area on /afs/ams.cern.ch/user (ask A. Eline to get a space)

Scratch Area

2 Tb scratch area available on /afs/ams.cern.ch/scratch

afs central scratch space (to be filled by A. Eline)

Moving back all your output using scp (recommended)

  • on lxplus run
ssh-keygen -t rsa

  • Make sure you have empty passphrase

  • now check you are able to login into say pcamsf2.cern.ch without password
ssh pcamsf2

  • submit test job on lsbatch

echo "test" >test.txt
scp test.txt `whoami`@ams.cern.ch:/f2users/`whoami`
scp test.txt `whoami`@ams.cern.ch:/scratch/`whoami`

  • if everything all right, you can see your files back on /f2users and /scratch

Mounting any user directories ( /f2users etc ) on lxplus using sshfs (see man sshfs on lxplus)
mkdir /tmp/f2users
sshfs pcamsr0(f2,r1).cern.ch:/f2users   /tmp/f2users
Unmounting sshfs mounted filesystems
  • NB (!) sshfs DOES NOT unmount volume automatically after session /job/ termination so it should be unmount manually by issuing
      fusermount -u /tmp/f2users   

icc compiler on lxplus

available in /afs/ams.cern.ch/opt/intel

to use it

set path=( /afs/ams.cern.ch/opt/intel/Compiler/11.1/073/bin/intel64 $path  )

setenv LD_LIBRARY_PATH .:/afs/ams.cern.ch/opt/intel/Compiler/11.1/073/idb/lib/intel64/:/afs/ams.cern.ch/opt/intel/Compiler/11.1/073/lib/intel64/:$ROOTSYS/lib:/afs/ams.cern.ch/opt/intel/compiler80/lib            

RootFiles Description

Analysis Examples

Edit | Attach | Watch | Print version | History: r11 < r10 < r9 < r8 < r7 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r11 - 2013-02-19 - BaosongShan
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    AMS All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback