CRAB @ FNAL Tutorial, LPC J-Term, January 2008


This brief tutorial will concentrate on Grid submission of CMSSW jobs. We will use CRAB to submit jobs to the grid and give an introduction to the most common use cases for CRAB.

Prerequisites and Requirements

We assume that the participants in this tutorial are either familiar with CMSSW analysis in interactive mode or took the CMSSW tutorial yesterday. Additionally, there are several prerequisites that must be satisfied in advance. Many of these require human intervention, so doing them during the tutorial is just not possible. In order to be able to submit Grid jobs from Fermilab, please make sure you have:

  • A valid UAF/LPC login account
  • A valid Grid Certificate
  • A certificate identifying you as belonging to the CMS VO

CRABatFNALTutorialOctober2007 explains how to do all of these things. Please make sure that before attending the tutorial you can log into cmsplc, issue the commands

voms-proxy-init -voms cms
voms-proxy-info -all
and see output which contains the line
attribute : /cms/Role=NULL/Capability=NULL

Login and prepare CMSSW interactive analysis

Note: whenever you see [c]sh you should use "csh" if your shell is csh or tcsh and "sh" if your shell is ash or bash. These instructions are for a simple, prepared analysis. If you have your own analysis, you are welcome to try that, but you may run into problems if the data you would like to run on is not widely distributed.

We'll begin by logging into the LPC cluster:

Next, make sure that the CMSSW environment is set up:
source /uscmst1/prod/sw/cms/[c]shrc prod
Now create a CMSSW project area:
scramv1 project CMSSW CMSSW_1_6_8
Lastly, unpack the sample analysis into this new area:
cd CMSSW_1_6_8/src
tar xvfz ~ewv/CrabSample2008Jan.tgz

This sample analysis is for Higgs -> ZZ; ZZ -> 4 mu, but without any muon ID. It creates just a few histograms so you can see it's working.

Build & run CMS interactive analysis

Build the CMSSW project:

scramv1 build
Look at the input file in higgs.cfg This file is just a part of the dataset we will be ultimately be running over. Lastly set up the CMSSW environment for 1_6_8 and run cmsRun with:
eval `scramv1 runtime -[c]sh`
cd Demo/MyTrackAnalyzer/src/
cmsRun higgs.cfg

When cmsRun finishes, you should have a file called "histograms.root" in the Demo/MyTrackAnalyzer/src/ directory. It has just a few histograms in it. You should check it with root to verify that things look OK. You always want to run a short job locally before submitting your analysis to the Grid for two reasons:

  1. Make sure the output is what you expect (for a small input set)
  2. If you have problems with your submission, it is nice to know that your code is not the problem

Locate the dataset and prepare CRAB submission

Now that we've verified that our analysis works OK, we're ready to run over the whole dataset. But we have to find it first. We'll use DBS to find the dataset. In this case, we're interested in Higgs -> 4 mu decays. The closet we can find that's been produced recently is Higgs -> 4 leptons. Point your web browser at DBS has lots of ways to find datasets, but in this case:

  • Select CMSSW_1_6_7 under "Software release"
  • Select CSA07Muon under "Primary dataset"
  • Press find

DBS shows us now all primary muon datasets reconstructed in CMSSW_1_6_7. For this exercise, we are looking for the Higgs→ZZ→4 leptons skim of the Stew soup (pythia with generator level filter). The full name of our dataset is /CSA07Muon/CMSSW_1_6_7-CSA07-Stew-P1-PDMuon-Skims2-higgsToZZ4LeptonsRECOSIM/USER. After finding the dataset, DBS can show us now where the data is located. This is information we have to give to CRAB. As an extra benefit, DBS will help us configure CRAB. Click on more (just to the right of the dataset name) and then click on crab.cfg. Copy and paste this output to a new file named crab.cfg in the src/ directory in the cmslpc session you used to run cmsRun. The file should look like this:


jobtype                 = cmssw
scheduler               = glitecoll


datasetpath             = /CSA07Muon/CMSSW_1_6_7-CSA07-Stew-P1-PDMuon-Skims2-higgsToZZ4LeptonsRECOSIM/USER
pset                    = 
total_number_of_events  = -1
events_per_job          = 1000
output_file             = 

return_data             = 1
use_central_bossDB      = 0
use_boss_rt             = 0

rb                      = CERN 
proxy_server            = 
virtual_organization    = cms
retry_count             = 0
lcg_catalog_type        = lfc
lfc_host                =
lfc_home                = /grid/cms

We need to make a few changes and fill in the blanks in this CRAB configuration file:

  • We're going to start off using a different scheduler, so replace the scheduler line with
scheduler               = condor_g
  • We need to tell CRAB what our input CMSSW .cfg and output files are, so change those two lines to read
pset                    = higgs.cfg
output_file             = histograms.root
  • Lastly, we need to tell condor_g where to submit the jobs, so to the [EDG] section, add these line
se_white_list =
but replace with the SE name of one of the four Tier2 sites that DBS says has this dataset. This will submit only to computing and storage elements whose names match "" Warning: for some Tier2 sites you may have to specify a CE part to be more specific (for example, there are two CEs at FNAL). CRAB will tell you during the submission process if this is the case. Just repeat the steps after adding to the [EDG] section:

ce_white_list =
  • We also don't want to run over the complete dataset for this exercise. So we restrict CRAB to run only over the first 10000 events by changing:
total_number_of_events = 10000
and because we set events_per_job = 1000 we expect CRAB to send 10 jobs.

Submit an analysis to the Grid

Now that we've prepared everything, we're ready to run the jobs following the four CRAB steps of creating, submitting, checking (as needed), and retrieving the output of our jobs.

Set up crab

To use the latest production release of CRAB, set it up like this:

source /uscmst1/prod/grid/CRAB/crab.[c]sh
If you need an earlier version of CRAB or a pre-release, look in /uscmst1/prod/grid/ to see what is available.

Job creation

The job creation uses the CRAB configuration file you just wrote to check the availability of the selected dataset and prepares the jobs for submission according to the selected job splitting:

crab -create

which uses the default name for a CRAB configuration file: crab.cfg. If a different CRAB configuration file should be used, the command becomes

crab -create -cfg <configuration file>

If you haven't used your Grid certificate in the last day or two, you'll be asked for the password to create a new proxy.

The creation process creates a CRAB project directory (format =crab_<number>_<date>_<time>) in the current working directory. It can be used later to distinguish multiple CRAB projects in the same directory.

The used CRAB configuration file is copied into the CRAB project directory, so the original can be changed and used again without interfering with the already created projects.

This is an example of the output of the creation command:

crab. crab (version 2.0.4) running on Thu Jan 10 15:20:22 2008

crab. Working options:
  scheduler           condor_g
  job type            CMSSW
  working directory   /uscms/home/gutsche/work/software/Tutorial/CMSSW_1_6_8/src/Demo/MyTrackAnalyzer/src/crab_0_080110_152022/

crab. Contacting Data Discovery Services ...
crab. Requested dataset: /CSA07Muon/CMSSW_1_6_7-CSA07-Stew-P1-PDMuon-Skims2-higgsToZZ4LeptonsRECOSIM/USER has 154587 events in 5 blocks.

crab. 10 job(s) can run on 10000 events.

crab. List of jobs and available destination sites:

Block     1: jobs                 1-10: sites:

crab. Creating 10 jobs, please wait...

crab. Total of 10 jobs created.

crab. Log-file is /uscms/home/gutsche/work/software/Tutorial/CMSSW_1_6_8/src/Demo/MyTrackAnalyzer/src/crab_0_080110_152022/log/crab.log

If there are problems, a good place to look for more information is the CRAB log file, crab_?_*_*/log/crab.log.

Job submission

The following command submits the previously created jobs:

crab -submit all -c

where -c specifies that CRAB uses the latest CRAB project in the current directory and all means to submit all created jobs. You can also specify and combination of jobs and job-ranges separated by commas (Example: =1,2,4-6). You can force CRAB to use a different CRAB project than the most recently created one:

crab -submit all -c <crab_directory>

The output of this command should look like:

crab. crab (version 2.0.4) running on Thu Jan 10 15:21:21 2008

crab. Working options:
  scheduler           condor_g
  job type            CMSSW
  working directory   /uscms/home/gutsche/work/software/Tutorial/CMSSW_1_6_8/src/Demo/MyTrackAnalyzer/src/crab_0_080110_152022/

crab. Found 1 compatible site(s) for job 1

Submitting 10 jobs
please wait

crab. Total of 10 jobs submitted.

crab. Total of 10 jobs submitted.
crab. Log-file is /uscms/home/gutsche/work/software/Tutorial/CMSSW_1_6_8/src/Demo/MyTrackAnalyzer/src/crab_0_080110_152022/log/crab.log

ignoring warnings.

Checking job status

Check the status of the jobs in the latest CRAB project with the following command:

crab -status 
or to check an earlier project:
crab -status -c  <crab_directory>

which should produce output similar to:

crab. crab (version 2.0.4) running on Thu Jan 10 15:25:23 2008

crab. Working options:
  scheduler           condor_g
  job type            CMSSW
  working directory   /uscms/home/gutsche/work/software/Tutorial/CMSSW_1_6_8/src/Demo/MyTrackAnalyzer/src/crab_0_080110_152022/

crab. Checking the status of all jobs: please wait
Chain    STATUS             E_HOST                                   EXE_EXIT_CODE JOB_EXIT_STATUS
1        Running                                                       
2        Running                                                       
3        Running                                                       
4        Running                                                       
5        Running                                                       
6        Running                                                       
7        Running                                                       
8        Running                                                       
9        Running                                                       
10       Running                                                       

>>>>>>>>> 10 Total Jobs 

>>>>>>>>> 10 Jobs Running
          List of jobs: 1,2,3,4,5,6,7,8,9,10
crab. Log-file is /uscms/home/gutsche/work/software/Tutorial/CMSSW_1_6_8/src/Demo/MyTrackAnalyzer/src/crab_0_080110_152022/log/crab.log

In this case, we can tell that there is one job that has completed successfully and whose output is ready to be retrieved. Five jobs are still running. We can start to gather output now or wait for the rest of the jobs to finish.

Retrieving job output

Once a job has status "Done" it's possible to retrieve the output back to the UI. The following command retrieves the output of all jobs with status done of the last created CRAB project:

crab -getoutput 
or to get the output of a specific project:
crab -getoutput -c  <dir name>

This command can be repeated as long as there are jobs in "Done" status.

The job results will be copied into the res/ subdirectory of your crab project. You should see output like this:

crab. crab (version 2.0.4) running on Thu Jan 10 15:28:38 2008

crab. Working options:
  scheduler           condor_g
  job type            CMSSW
  working directory   /uscms/home/gutsche/work/software/Tutorial/CMSSW_1_6_8/src/Demo/MyTrackAnalyzer/src/crab_0_080110_152022/

crab. Results of Job # 1 are in /uscms/home/gutsche/work/software/Tutorial/CMSSW_1_6_8/src/Demo/MyTrackAnalyzer/src/crab_0_080110_152022/res/
crab. Results of Job # 2 are in /uscms/home/gutsche/work/software/Tutorial/CMSSW_1_6_8/src/Demo/MyTrackAnalyzer/src/crab_0_080110_152022/res/
crab. Results of Job # 3 are in /uscms/home/gutsche/work/software/Tutorial/CMSSW_1_6_8/src/Demo/MyTrackAnalyzer/src/crab_0_080110_152022/res/
crab. Results of Job # 4 are in /uscms/home/gutsche/work/software/Tutorial/CMSSW_1_6_8/src/Demo/MyTrackAnalyzer/src/crab_0_080110_152022/res/
crab. Results of Job # 5 are in /uscms/home/gutsche/work/software/Tutorial/CMSSW_1_6_8/src/Demo/MyTrackAnalyzer/src/crab_0_080110_152022/res/
crab. Results of Job # 6 are in /uscms/home/gutsche/work/software/Tutorial/CMSSW_1_6_8/src/Demo/MyTrackAnalyzer/src/crab_0_080110_152022/res/
crab. Results of Job # 7 are in /uscms/home/gutsche/work/software/Tutorial/CMSSW_1_6_8/src/Demo/MyTrackAnalyzer/src/crab_0_080110_152022/res/
crab. Results of Job # 8 are in /uscms/home/gutsche/work/software/Tutorial/CMSSW_1_6_8/src/Demo/MyTrackAnalyzer/src/crab_0_080110_152022/res/
crab. Results of Job # 9 are in /uscms/home/gutsche/work/software/Tutorial/CMSSW_1_6_8/src/Demo/MyTrackAnalyzer/src/crab_0_080110_152022/res/
crab. Results of Job # 10 are in /uscms/home/gutsche/work/software/Tutorial/CMSSW_1_6_8/src/Demo/MyTrackAnalyzer/src/crab_0_080110_152022/res/
crab. Log-file is /uscms/home/gutsche/work/software/Tutorial/CMSSW_1_6_8/src/Demo/MyTrackAnalyzer/src/crab_0_080110_152022/log/crab.log

Making the final plot

All 10 jobs produce a histogram output file which can be combined using ROOT in the res directory:

cd crab_directory/res
ls *.root
hadd histograms.root histograms_*.root

The final histograms.root opened in ROOT contains the final plot:

root histograms.root
(or use the TBrowser if you prefer).

A more sophisticated Grid Analysis

Next, we're going to change two things about our grid job:

  • Instead of collecting the output, we'll have it automatically written to Fermilab's dCache system
  • We'll change the scheduler and use CRAB's server mode so that we can run our jobs in multiple locations and with less interaction from us

Prepare a dCache area at FNAL for storage element interaction

For CRAB to be able to write into your dCache user directory /pnfs/cms/WAX/resilient/<username>

we have to create a destination directory and change the file permissions:

mkdir /pnfs/cms/WAX/resilient/<username>
mkdir /pnfs/cms/WAX/resilient/<username>/tutorial 
chmod +775 /pnfs/cms/WAX/resilient/<username>/tutorial

replacing <username> with your username. The permission change allows the CMS group to write into your directory, not just you.

Changing crab.cfg

To change to server mode, modify the line

scheduler               = condor_g
in the [CRAB] section to instead read
scheduler               = glite
and add two lines for server operation
server_mode =1
server_name =
We have to delete (or comment them out by putting # at the start of each line) the two lines you added earlier
ce_white_list =
se_white_list =
since we no longer want to restrict where the jobs will run. We also wan an e-mail when our job is done so we don't have to keep checking the status. Put these two lines in the [USER] section:
thresholdLevel          = 100
eMail                   =
. You can replace 100 by a "percent done" to get the e-mail earlier.

The second round of changes to crab.cfg enables the writing of data to dCache. Replace

return_data             = 1
return_data            = 0
copy_data              = 1
storage_element        =
storage_path           = /srm/managerv1?SFN=/resilient/<username>/tutorial
This instructs CRAB to copy the output files listed in
output_file = 
to dCache instead of making them available when you do crab -getoutput. This allows you to write much bigger output files (2GB vs. 50 MB) and is recommended for CMSSW ROOT files, but here we'll just use it for our histogram files. The rest of the output (stderr, stdout, etc.) is still retrieved with
crab -getoutput

Submit and check on the job with the same commands as before:

crab -create
crab -submit all
crab -status 
crab -getoutput
but in addition, you should see your output (histogram) files begin to arrive in dCache
ls -l /pnfs/cms/WAX/resilient/<username>/tutorial 

Going from here and more information

A full exploration of CRAB's capabilities is too much for this tutorial, so here are some things you can try on your own:

  • Use DBS to find your favorite dataset coming from CSA07 to a Tier2 (Physics Group CSA07)
  • Use CRAB for MC generation: Set dataset to "None" and use seeds as described in crab -help
  • Check out other schedulers in local mode
    • EDG: Going away, but still sometimes useful
    • glitecoll: A bulk submission mode for glite
    • Server mode recommended for runs of more than about 25 jobs
  • Investigate CRAB's debugging options
    • crab -kill and crab -resubmit to kill and resubmit jobs
    • crab -postMortem returns Grid error messages
    • Realize that things will go wrong with the Grid as the load increases and as Grid experts adapt

There are a number of resources to help you

A number of talks and tutorials from the June 2007 workshop may be quite useful. This was a full day tutorial, so it was much more comprehensive. Talks were given on each aspect of the computing model and how they relate.

-- EricVaandering - 02 Jan 2008

Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r4 - 2008-01-10 - OliverGutsche
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback