CRAB at CAF/LSF at CERN
Complete:
Detailed Review status
This document describes how to use CRAB at CERN for direct submission to the batch system LSF or CERN Analysis Facility (CAF). The responsable of CAF is Peter Kreuzer.
Useful information of CAF can be found in the
CAF twiki page.
Prerequisites
- Dataset you want to access has to be available at the CAF, so it must be registered in the CAF DBS
- If you run on CAF, you have to be authorized to do so. In this page: https://twiki.cern.ch/twiki/bin/view/CMS/CAF#User_Permissions you can find the sub-groups and the correspond leader. If you know your sub-group, you can contact the leader for the authorization.
- CRAB StandAlone (direct submission)
- Jobs has to be submitted from an afs directory, from a node with LSF access for exemple on lxplus
- Since in this case you are effectively using CRAB as a convenience tool to do LSF submission from your shell, you need to setup the environment as usual:
- Please note that you must be sure to have enough quota on your afs area. Large output should be put on castor (look at CAF stageout below)
- Even if you decided to send the output to castor, the stdout/err and the Framework Job Report will be returned back to your afs area in any case.
- Removes the requirement to use an AFS directory and a host with LSF access, so can also submit from your desktop/laptop
Running
The workflow is exactly the same of that you would follow to access data on the Grid (see:
CRAB Tutorial). So you setup your CMSSW area, you develop your code, test it on a (small) part of a dataset and then you configure CRAB to create and submit identical jobs to CAF to analyze the full Dataset.
In the crab.cfg configuration file, you have just to put under the
[CRAB]
section:
scheduler = caf
The available CAF queues are:
cmscaf1nh
cmscaf1nd
cmscaf1nw
Running on the CAF, using
caf
as scheduler instead of
lsf
, the longest queue will be selected automatically (cmscaf1nw).
If you need to select a different queue you can fill the parameter
queue
under the
[CAF]
section with either cmscaf1nh or cmscaf1nd (i.e.
queue = cmscaf1nh
).
If you know that your jobs are short, it should be more efficient to use shorter queues.
CAF stageout
If you are running jobs at CAF then the required stageout configuration is:
- Stage out into CAF user area (T2_CH_CAF is the offical site name for CAF):
[USER]
copy_data = 1
storage_element=T2_CH_CAF
user_remote_dir=xxx
the path where data will be stored is /castor/cern.ch/cms/store/caf/user/<username>/<user_remote_dir>
There is no support for staging out to the CAF-T1 from the GRID. The above instructions only apply for jobs running on the CAF itself.
Further details on CRAB and Stage out configurations available at
this page.
CAF publication
You need the following in crab configuration file:
- (NOTE: the storage element where the data are copied have to be T2_CH_CAF):
[USER]
copy_data = 1
storage_element=T2_CH_CAF
publish_data=1
publish_data_name = data-name-to-publish (e.g. publish_data_name = JohnSmithTestDataVersion666 )
dbs_url_for_publication = https://cmsdbsprod.cern.ch:8443/cms_dbs_caf_analysis_01_writer/servlet/DBSServlet
The path where data will be stored is /castor/cern.ch/cms/store/caf/user/<username>/<primarydataset>/<publish_data_name>/<PSETHASH>
Review status
Responsible:
MarcoCalloni
Last reviewed by: