CRAB @ FNAL Tutorial, CMS101++, June 2008 These instructions were intended only in preparation for a tutorial. They are now out of date and should not be used. Please follow the official CMS instructions as indicated in these pages.


This tutorial will concentrate on Grid submission of CMSSW jobs. We will use CRAB to submit jobs to the grid and give an introduction to the most common use cases for CRAB.

Prerequisites and Requirements

We assume that the participants in this tutorial are either familiar with CMSSW analysis in interactive mode or took the CMSSW & PAT tutorial yesterday. Additionally, there are several prerequisites that must be satisfied in advance. Many of these require human intervention, so doing them during the tutorial is just not possible. In order to be able to submit Grid jobs from Fermilab, please make sure you have:

  • A valid UAF/LPC login account
  • A valid Grid Certificate
  • A certificate identifying you as belonging to the CMS VO

The instructions at CRABatFNALPrerequisites explain how to do all of these things. Please make sure that before attending the tutorial you can log into cmsplc, issue the commands

voms-proxy-init -voms cms
voms-proxy-info -all
and see output which contains the line
attribute : /cms/Role=NULL/Capability=NULL

Login and prepare CMSSW interactive analysis

Note: whenever you see [c]sh you should use "csh" if your shell is csh or tcsh and "sh" if your shell is ash or bash. These instructions are for a simple, prepared analysis. If you have your own analysis, you are welcome to try that, but you may run into problems if the data you would like to run on is not widely distributed.

We'll begin by logging into the LPC cluster:

Next, make sure that the CMSSW environment is set up:
source /uscmst1/prod/sw/cms/[c]shrc prod
Now create a CMSSW project area:
scramv1 project CMSSW CMSSW_2_1_0_pre6
Lastly, unpack the sample analysis into this new area:
cd CMSSW_2_1_0_pre6/src
tar xvfz ~ewv/CrabSample2008Jun.tgz

This sample analysis is for Higgs -> ZZ; ZZ -> 4 mu, but without any muon ID. It creates a few histograms just so you can see it's working.

Build & run CMS interactive analysis

Build the CMSSW project:

scramv1 build
Look at the input file in config file. This file is just a part of the dataset we will be ultimately be running over. Lastly set up the CMSSW environment and run cmsRun with:
cmsenv # Formerly "eval scramv1 runtime" etc.
cd Demo/MyTrackAnalyzer/src/

When cmsRun finishes, you should have a file called "histograms.root" in the Demo/MyTrackAnalyzer/src/ directory. It has just a few histograms in it. You should check it with root to verify that things look OK. You always want to run a short job locally before submitting your analysis to the Grid for two reasons:

  1. Make sure the output is what you expect (for a small input set)
  2. If you have problems with your submission, it is nice to know that your code is not the problem

Locate the dataset and prepare CRAB submission

Now that we've verified that our analysis works OK, we're ready to run over the whole dataset. But we have to find it first. We'll use DBS to find the dataset. The example analysis is for Higgs -> 4 mu decays. The closest we can find that's been produced recently is Higgs -> 4 leptons. Point your web browser at DBS has lots of ways to find datasets, but in this case:

  • Select CMSSW_2_1_0_pre5 under "Software release" (yes, different than our code version)
  • Press find

For this exercise, we are looking for the Higgs -> ZZ -> 4 leptons skim. The full name of our dataset is /RelValH200ZZ4L/CMSSW_2_1_0_pre5-RelVal-1212539383-STARTUP-2nd-02-STARTUP/GEN-SIM-DIGI-RAW-HLTDEBUG-RECO. After finding the dataset, DBS can show us now where the data is located. This is the information CRAB needs to look up. As an extra benefit, DBS will help us configure CRAB. Click on crab.cfg under the dataset name. Copy and paste this output to a new file named crab.cfg in the src/ directory in the cmslpc session you used to run cmsRun. The crab.cfg file should look like this:


jobtype                 = cmssw
scheduler               = glitecoll


datasetpath             = /RelValH200ZZ4L/CMSSW_2_1_0_pre5-RelVal-1212539383-STARTUP-2nd-02-STARTUP/GEN-SIM-DIGI-RAW-HLTDEBUG-RECO
pset                    = 
total_number_of_events  = -1
events_per_job          = 1000
output_file             = 

return_data             = 1
use_central_bossDB      = 0
use_boss_rt             = 0

rb                      = CERN 
proxy_server            = 
virtual_organization    = cms
retry_count             = 0
lcg_catalog_type        = lfc
lfc_host                =
lfc_home                = /grid/cms

We need to make a few changes and fill in the blanks in this CRAB configuration file:

  • We're going to start off using a different grid scheduler, so replace the scheduler line with
scheduler               = condor_g
  • We need to tell CRAB what our input CMSSW config and output files are, so change those two lines to read
pset                    =
output_file             = histograms.root
  • We also don't want to run over the complete dataset for this exercise. So we restrict CRAB to run only over the first 5000 events by changing:
total_number_of_events = 5000
Because events_per_job = 1000 we expect CRAB to send 5 jobs.

Submit an analysis to the Grid

Now that we've prepared everything, we're ready to run the jobs following the four CRAB steps of creating, submitting, checking (as needed), and retrieving the output of our jobs.

Set up crab

To use the latest production release of CRAB, set it up like this:

source /uscmst1/prod/grid/CRAB/crab.[c]sh
If you need an earlier version of CRAB or a pre-release, look in /uscmst1/prod/grid/ to see what is available, but please use the default version if possible.

Job creation

The job creation uses the CRAB configuration file you just wrote to check the availability of the selected dataset and prepares the jobs for submission according to the selected job splitting:

crab -create

which uses the default name for a CRAB configuration file: crab.cfg. If a different CRAB configuration file should be used, the command becomes

crab -create -cfg <configuration file>

If you haven't used your Grid certificate in the last day or two, you'll be asked for the password to create a new proxy.

The creation process creates a CRAB project directory (format =crab_<number>_<date>_<time>) in the current working directory. It can be used later to distinguish multiple CRAB projects in the same directory.

The used CRAB configuration file is copied into the CRAB project directory, so the original can be changed and used again without interfering with the already created project.

This is an example of the output of the creation command:

crab. crab (version 2.3.0) running on Mon Jun 16 16:36:45 2008

crab. Working options:
  scheduler           condor_g
  job type            CMSSW
  working directory   /uscms/home/ewv/CMSSW_2_1_0_pre6/src/Demo/MyTrackAnalyzer/src/crab_0_080616_163645/

crab. `voms-proxy-info -timeleft 2>/dev/null`
   failed with exit code 256=0(signal)+1(status)
crab. No valid proxy found or remaining time of validity of already existing proxy shorter than 10 hours!
 Creating a user proxy with default length of 192h

Cannot find file or dir: /uscms/home/ewv/.glite/vomses
Enter GRID pass phrase:
Your identity: /DC=org/DC=doegrids/OU=People/CN=Eric Vaandering 768123
Creating temporary proxy .................................... Done
Contacting [/DC=ch/DC=cern/OU=computers/] "cms" Done
Creating proxy ........................................... Done
Your proxy is valid until Tue Jun 24 16:36:50 2008
crab. Contacting Data Discovery Services ...
crab. Requested dataset: /CSA07Muon/CMSSW_1_6_7-CSA07-Stew-P1-PDMuon-Skims2-higgsToZZ4LeptonsRECOSIM/USER has 154587 events in 5 blocks.

crab. 10 job(s) can run on 10000 events.

crab. List of jobs and available destination sites:

Block     1: jobs                 1-10: sites:,,,,,,,

crab. Creating 10 jobs, please wait...

crab. Total of 10 jobs created.

crab. Log-file is /uscms/home/ewv/CMSSW_2_1_0_pre5/src/Demo/MyTrackAnalyzer/src/crab_0_080616_163645/log/crab.log

If there are problems, a good place to look for more information is the CRAB log file, crab_?_*_*/log/crab.log.

Job submission

The following command submits the previously created jobs:

crab -submit all -c

where -c specifies that CRAB uses the latest CRAB project in the current directory and all means to submit all created jobs. You can also specify and combination of jobs and job-ranges separated by commas (Example: =1,2,4-6). You can force CRAB to use a different CRAB project than the most recently created one:

crab -submit all -c <crab_directory>

The output of this command should look like:

crab. crab (version 2.3.0) running on Mon Jun 16 16:41:27 2008

crab. Working options:
  scheduler           condor_g
  job type            CMSSW
  working directory   /uscms/home/ewv/CMSSW_2_1_0_pre5/src/Demo/MyTrackAnalyzer/src/crab_0_080616_163645/

crab. Checking available resources...
crab. Found  compatible site(s) for job 1
crab. 1 blocks of jobs will be submitted
                                     Submitting 10 jobs
100% [==================================================================]
                                         please wait
crab. Total of 10 jobs submitted.
crab. Log-file is /uscms/home/ewv/CMSSW_2_1_0_pre5/src/Demo/MyTrackAnalyzer/src/crab_0_080616_163645/log/crab.log

Checking job status

Check the status of the jobs in the latest CRAB project with the following command:

crab -status 
or to check an earlier project:
crab -status -c  <crab_directory>

which should produce output similar to:

crab. crab (version 2.3.0) running on Mon Jun 16 17:11:05 2008

crab. Working options:
  scheduler           condor_g
  job type            CMSSW
  working directory   /uscms/home/ewv/CMSSW_2_1_0_pre5/src/Demo/MyTrackAnalyzer/src/crab_0_080616_163645/

crab. Checking the status of all jobs: please wait
ID       STATUS             E_HOST                                   EXE_EXIT_CODE JOB_EXIT_STATUS
1        Running  
2        Running  
3        Running  
4        Running  
5        Submitted
6        Submitted
7        Submitted
8        Done     
9        Done     
10       Done     

>>>>>>>>> 10 Total Jobs

crab. Log-file is /uscms/home/ewv/CMSSW_2_1_0_pre5/src/Demo/MyTrackAnalyzer/src/crab_0_080616_163645/log/crab.log

In this case, we can tell that there are three jobs that have completed (successfully, we hope) and whose output is ready to be retrieved. Four jobs are still running, and three have not started yet. We can start to gather output now or wait for the rest of the jobs to run and finish.

Retrieving job output

Once a job has status "Done" it's possible to retrieve the output back to the UI. The following command retrieves the output of all jobs with status done of the last created CRAB project:

crab -getoutput 
or to get the output of a specific project:
crab -getoutput -c  <dir name>

This command can be repeated as long as there are jobs in "Done" status.

The job results will be copied into the res/ subdirectory of your crab project. You should see output like this:

crab. crab (version 2.3.0) running on Tue Jun 17 09:43:23 2008

crab. Working options:
  scheduler           condor_g
  job type            CMSSW
  working directory   /uscms/home/ewv/CMSSW_2_1_0_pre5/src/Demo/MyTrackAnalyzer/src/crab_0_080617_090412/

crab. Results of Jobs # 8 are in /uscms/home/ewv/CMSSW_2_1_0_pre5/src/Demo/MyTrackAnalyzer/src/crab_0_080617_090412/res/
crab. Results of Jobs # 9 are in /uscms/home/ewv/CMSSW_2_1_0_pre5/src/Demo/MyTrackAnalyzer/src/crab_0_080617_090412/res/
crab. Results of Jobs # 10 are in /uscms/home/ewv/CMSSW_2_1_0_pre5/src/Demo/MyTrackAnalyzer/src/crab_0_080617_090412/res/
crab. Log-file is /uscms/home/ewv/CMSSW_2_1_0_pre5/src/Demo/MyTrackAnalyzer/src/crab_0_080617_090412/log/crab.log

Making the final plot

Each job produces a histogram output file which can be combined with the others using ROOT in the res directory:

cd crab_directory/res
ls *.root
hadd histograms.root histograms_*.root

The final histograms.root opened in ROOT contains the final plot:

root histograms.root
(or use the TBrowser if you prefer).

Using the CRAB Server and storing data on a storage element

Next, we're going to change two things about our grid job:

  • We'll change the scheduler and use CRAB's server mode so that we can run our jobs in multiple locations and with less interaction from us
  • Instead of collecting the output root file, we'll have it automatically written to Fermilab's dCache system

Prepare a dCache area at FNAL for storage element interaction

For CRAB to be able to write into your dCache user directory /pnfs/cms/WAX/resilient/<username>

we have to create a destination directory and change the file permissions:

mkdir /pnfs/cms/WAX/resilient/<username>
mkdir /pnfs/cms/WAX/resilient/<username>/tutorial 
chmod +775 /pnfs/cms/WAX/resilient/<username>/tutorial

replacing <username> with your username. The permission change allows the CMS group to write into your directory, not just you. Grid jobs run as a generic user in the CMS group, so you must allow write access to the whole group.

Changing crab.cfg

To change to server mode, modify the line

scheduler               = condor_g
in the [CRAB] section to instead read
scheduler               = glite
and add a line for server operation:
server_name = bari
and these two lines to the [USER] section
thresholdLevel          = 100
eMail                   =
since we want an e-mail when our job is done so we don't have to keep checking the status. You can replace 100 by a "percent done" to get the e-mail earlier.

The second round of changes to crab.cfg enables the writing of data to dCache. Replace

return_data             = 1
return_data            = 0
copy_data              = 1
storage_element        =
storage_path           = /srm/managerv1?SFN=/resilient/<username>/tutorial
# Set SRM version to match "managerv1" above. 
srm_version = 1
This instructs CRAB to copy the output files listed in
output_file = 
to dCache instead of making them available when you do crab -getoutput. This allows you to write much bigger output files (2GB vs. 50 MB) and is recommended for CMSSW ROOT files, but here we'll just use it for our histogram files. The rest of the output (stderr, stdout, etc.) is still retrieved with
crab -getoutput

Submit and check on the job with the same commands as before:

crab -create
crab -submit all
crab -status 
crab -getoutput
but in addition, you should see your output (histogram) files begin to arrive in dCache
ls -l /pnfs/cms/WAX/resilient/<username>/tutorial 

A note on CRAB Server

CRAB Server is a rather new development, so expect this to be more fully fleshed out in the future. Currently only gLite can be used as the scheduler with the server. We hope to have Condor support soon. Also, very few servers are currently supported. In the future, there will be one or two CRAB Servers per analysis group.

Local submission to the cmslpc CAF

If the data we're interested in is stored locally at FNAL, we can use the local resources instead. In the past, one needed to write their own Condor scripts and JDL files and submit them. Now recent versions of CRAB can do all this for you. The changes are quite simple:

From our first crab.cfg file, change the line

scheduler               = condor_g
in the [CRAB] section to instead read
scheduler               = condor

When these jobs start up, they do not have the CMSSW environment set up unless you set it up for your bash shell. Most of us use the csh shell instead, so you have to create a .profile file in your home directory:

echo "source /uscmst1/prod/sw/cms/shrc prod" > ~/.profile

Now you can submit jobs to the cmslpc cluster just like jobs to the Grid using the same CRAB commands:

crab -create
crab -submit all
crab -status 
crab -getoutput

Using CRAB's copy_data option in this local mode is a little more complex and is still being worked on. For up to date information, see CRABonLPCCAF.

Going from here and more information

A full exploration of CRAB's capabilities is too much for this tutorial, so here are some things you can try on your own:

  • Use DBS to find your favorite dataset at a Tier2
  • Use CRAB for MC generation: Set dataset to "None" and use seeds as described in crab -help
  • Check out other schedulers in local mode
  • Investigate CRAB's debugging options
    • crab -kill and crab -resubmit to kill and resubmit jobs
    • crab -postMortem returns Grid error messages
    • Realize that things will go wrong with the Grid as the load increases and as Grid experts adapt

There are a number of resources to help you

-- EricVaandering - 18 Jun 2008

Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r8 - 2010-11-03 - EricVaandering
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback