CRAB at CAF/LSF at CERN

Complete:
Detailed Review status

Newsbox
under review

---

This document describes how to use CRAB at CERN for direct submission to the batch system LSF or CERN Analysis Facility (CAF). The responsable of CAF is Peter Kreuzer. Useful information of CAF can be found in the CAF twiki page.

Prerequisites

• Dataset you want to access has to be available at the CAF, so it must be registered in the CAF DBS
• If you run on CAF, you have to be authorized to do so. In this page: https://twiki.cern.ch/twiki/bin/view/CMS/CAF#User_Permissions you can find the sub-groups and the correspond leader. If you know your sub-group, you can contact the leader for the authorization.
• CRAB StandAlone (direct submission)
  • Jobs has to be submitted from an afs directory, from a node with LSF access for exemple on lxplus
  • Since in this case you are effectively using CRAB as a convenience tool to do LSF submission from your shell, you need to setup the environment as usual:
    • make sure you setup the environment in the following order *

      1.source /afs/cern.ch/cms/LCG/LCG-2/UI/cms_ui_env.sh
      2. cmsenv
      3. source /afs/cern.ch/cms/ccs/wm/scripts/Crab/crab.sh
      

      • in the above replace sh with csh if you are using tcsh
  • Please note that you must be sure to have enough quota on your afs area. Large output should be put on castor (look at CAF stageout below)
  • Even if you decided to send the output to castor, the stdout/err and the Framework Job Report will be returned back to your afs area in any case.
  • Removes the requirement to use an AFS directory and a host with LSF access, so can also submit from your desktop/laptop

Running

The workflow is exactly the same of that you would follow to access data on the Grid (see: CRAB Tutorial). So you setup your CMSSW area, you develop your code, test it on a (small) part of a dataset and then you configure CRAB to create and submit identical jobs to CAF to analyze the full Dataset. In the crab.cfg configuration file, you have just to put under the [CRAB] section:
scheduler = caf

The available CAF queues are:

cmscaf1nh
cmscaf1nd
cmscaf1nw

Running on the CAF, using caf as scheduler instead of lsf, the longest queue will be selected automatically (cmscaf1nw). If you need to select a different queue you can fill the parameter queue under the [CAF] section with either cmscaf1nh or cmscaf1nd (i.e. queue = cmscaf1nh). If you know that your jobs are short, it should be more efficient to use shorter queues.

CAF stageout

If you are running jobs at CAF then the required stageout configuration is:

• Stage out into CAF user area (T2_CH_CAF is the offical site name for CAF):

[USER]
copy_data = 1
storage_element=T2_CH_CAF
user_remote_dir=xxx

the path where data will be stored is /castor/cern.ch/cms/store/caf/user/<username>/<user_remote_dir>

There is no support for staging out to the CAF-T1 from the GRID. The above instructions only apply for jobs running on the CAF itself.

Further details on CRAB and Stage out configurations available at this page.

CAF publication

You need the following in crab configuration file:

• (NOTE: the storage element where the data are copied have to be T2_CH_CAF):

 
[USER]
copy_data = 1
storage_element=T2_CH_CAF
publish_data=1
publish_data_name = data-name-to-publish  (e.g. publish_data_name = JohnSmithTestDataVersion666 )
dbs_url_for_publication = https://cmsdbsprod.cern.ch:8443/cms_dbs_caf_analysis_01_writer/servlet/DBSServlet

The path where data will be stored is /castor/cern.ch/cms/store/caf/user/<username>/<primarydataset>/<publish_data_name>/<PSETHASH>

Review status

Reviewer/Editor and Date (copy from screen)	Comments
MarcoCalloni - 18 Dec 2008	linked to workbook, needs to be renamed swguide -> workbook
StefanoLacaprara - 13 Mar 2008	created the page

Responsible: MarcoCalloni
Last reviewed by: