Do NOT use this tutorial. It is woefully out of date. Use the current CRAB tutorial instead:

CRAB JTerm Pre-tutorial


This tutorial will concentrate on Grid submission of CMSSW jobs. We will use CRAB in server mode to submit jobs to the grid and give an introduction to the most common use cases for CRAB. This data and analysis of this tutorial is based on the CRAFT tutorial by Jonatan Piedra given at the December CMS week.

Prerequisites and Requirements

We assume that the participants in this tutorial are familiar with CMSSW analysis in interactive mode. Additionally, there are several prerequisites that must be satisfied in advance. Many of these require human intervention, so the process may take a few days. In order to be able to submit Grid jobs from Fermilab, please make sure you have:

  • A valid UAF/LPC login account
  • A valid Grid Certificate
  • A certificate identifying you as belonging to the CMS VO

The instructions at and explain how to do all of these things. Please make sure that before attempting this tutorial you can log into cmsplc, issue the commands

voms-proxy-init -voms cms
voms-proxy-info -all
and see output which contains the line
attribute : /cms/Role=NULL/Capability=NULL

Login and prepare CMSSW interactive analysis

Note: whenever you see [c]sh you should use "csh" if your shell is csh or tcsh and "sh" if your shell is ash or bash. These instructions are for a simple, prepared analysis. If you have your own analysis, you are welcome to try that, but you may run into problems if the data you would like to run on is not widely distributed.

We'll begin by logging into the LPC cluster:

Next, make sure that the CMSSW environment is set up:
source /uscmst1/prod/sw/cms/[c]shrc prod
Now create a CMSSW project area:
scram project CMSSW CMSSW_2_1_12
Lastly, unpack the sample analysis into this new area:
cd CMSSW_2_1_12/src
tar xvfz ~ewv/CrabSample2009Pre.tgz

This sample analysis is for CRAFT data. It creates a number of histograms just so you can see it's working.

Build & run CMS interactive analysis

Build the CMSSW project:

scram build
Look at the input file in config file. This file is just a part of the dataset we will be ultimately be running over. Lastly set up the CMSSW environment and run cmsRun with:
cd UserArea/CraftAnalysis/test

When cmsRun finishes, you should have a file called "test.root" in the UserArea/CraftAnalysis/test/ directory. It has a number of histograms in it. You should check it with root to verify that things look OK. You always want to run a short job locally before submitting your analysis to the Grid for two reasons:

  1. Make sure the output is what you expect (for a small input set).
  2. If you have problems with your submission, it is nice to know that your code is not the problem.

Locate the dataset and prepare CRAB submission

Now that we've verified that our analysis works OK, we're ready to run over the whole dataset. But we have to find it first. We'll use DBS to find the dataset. Point your web browser at DBS has lots of ways to find datasets, but in this case:

  • change the search parameter to read
    find dataset where dataset = /Cosmics/Commissioning08_CRAFT_ALL_V4_ReReco-v1/RECO
  • Press "Search"

After finding the dataset, DBS can show us now where the data is located. This is the information CRAB needs to look up. As an extra benefit, DBS will help us configure CRAB. Click on crab.cfg under the dataset name. Copy and paste this output to a new file named crab.cfg in the src/ directory in the cmslpc session you used to run cmsRun. The crab.cfg file should look like this:

jobtype = cmssw
scheduler = glite
server_name = bari

events_per_jobs = 1000

return_data = 1

We need to make a few changes and fill in the blanks in this CRAB configuration file:

  • As of the writing of this tutorial, there is a bug in the crab.cfg returned by DBS. Change "events_per_jobs" to "events_per_job"
  • We need to tell CRAB what our input CMSSW config and output files are, so change/add those two lines to read
pset                    =
output_file             = test.root
(put the output_file line in the CMSSW section).

  • We also don't want to run over the complete dataset for this exercise. So we restrict CRAB to run only over the first 50000 events by changing:
total_number_of_events = 50000
events_per_job = 10000
Because events_per_job = 10000 we expect CRAB to send 5 jobs.

We can also change the server we're using. A list of servers for user analysis is maintained.

Submit an analysis to the Grid

Now that we've prepared everything, we're ready to run the jobs following the four CRAB steps of creating, submitting, checking (as needed), and retrieving the output of our jobs.

Set up crab

To use the latest production release of CRAB, set it up like this:

source /uscmst1/prod/grid/CRAB/crab.[c]sh
If you need an earlier version of CRAB or a pre-release, look in /uscmst1/prod/grid/ to see what is available, but please use the default version if possible.

Job creation

The job creation uses the CRAB configuration file you just wrote to check the availability of the selected dataset and prepares the jobs for submission according to the selected job splitting:

crab -create

which uses the default name for a CRAB configuration file: crab.cfg. If a different CRAB configuration file should be used, the command becomes

crab -create -cfg <configuration file>

If you haven't used your Grid certificate in the last day or two, you'll be asked for the password to create a new proxy.

The creation process creates a CRAB project directory (format =crab_<number>_<date>_<time>) in the current working directory. It can be used later to distinguish multiple CRAB projects in the same directory.

The used CRAB configuration file is copied into the CRAB project directory, so the original can be changed and used again without interfering with the already created project.

This is an example of the output of the creation command:

crab. crab (version 2.4.3) running on Fri Dec 26 10:21:25 2008

crab. Working options:
  scheduler           glite
  job type            CMSSW
  working directory   /uscms/home/ewv/Tutorial/CMSSW_2_1_12/src/UserArea/CraftAnalysis/test/crab_craft/

crab. Downloading config files for WMS:
Cannot find file or dir: /uscms/home/ewv/.glite/vomses
Enter GRID pass phrase:
Your identity: /DC=org/DC=doegrids/OU=People/CN=Eric Vaandering 768123
Creating temporary proxy ..................................... Done
Proxy Verify OK
Your proxy is valid until: Fri Jan  2 10:21:41 2009
A proxy valid for 168 hours (7.0 days) for user /DC=org/DC=doegrids/OU=People/CN=Eric Vaandering 768123 now exists on
crab. Contacting Data Discovery Services ...
crab. Requested dataset: /Cosmics/Commissioning08-PromptReco-v2/RECO has 397440487 events in 1297 blocks.

crab. 20 job(s) can run on 200000 events.

crab. List of jobs and available destination sites:

Block     1: jobs                 1-20: sites:

crab. Creating 20 jobs, please wait...

crab. Total of 20 jobs created.

If there are problems, a good place to look for more information is the CRAB log file, crab_?_*_*/log/crab.log.

Job submission

The following command submits the previously created jobs:

crab -submit

You can also specify specific jobs to submit (see crab -help). You can also force CRAB to use a different CRAB project than the most recently created one:

crab -submit -c <crab_directory>

The output of this command should look like:

crab. crab (version 2.4.3) running on Mon Jun 16 16:41:27 2008

crab. Working options:
  scheduler           glite
  job type            CMSSW
  working directory   /uscms/home/ewv/Tutorial/CMSSW_2_1_12/src/UserArea/CraftAnalysis/test/crab_craft/

crab. Checking available resources...
crab. Found  compatible site(s) for job 1
crab. 1 blocks of jobs will be submitted
                                     Submitting 10 jobs
100% [==================================================================]
                                         please wait
crab. Total of 10 jobs submitted.

Checking job status

Check the status of the jobs in the latest CRAB project with the following command:

crab -status 
or to check an earlier project:
crab -status -c  <crab_directory>

which should produce output similar to:

crab. crab (version 2.4.3) running on Fri Dec 26 14:59:14 2008

crab. Working options:
  scheduler           glite
  job type            CMSSW
  working directory   /uscms/home/ewv/Tutorial/CMSSW_2_1_12/src/UserArea/CraftAnalysis/test/crab_0_081226_145325/

ID     STATUS             E_HOST                               EXE_EXIT_CODE JOB_EXIT_STATUS  ENDED
1      Done                                                        N
2      Done                                                       N
3      Done                                                        N
4      Done                                                       N
5      Done                                                        N
6      Running                                                    N
7      Running                                                     N
8      Running                                                     N
9      Scheduled                                                  N
10     Scheduled                                                  N
11     Ready                                                      N
12     Scheduled                                                  N
13     Scheduled                                                  N
14     Scheduled                                                  N
15     Ready                                                       N
16     Scheduled                                                  N
17     Ready                                                       N
18     Scheduled                                                  N
19     Scheduled                                                  N
20     Scheduled                                                  N
21     Ready                                                       N

>>>>>>>>> 21 Total Jobs

>>>>>>>>> 3 Jobs Ready

>>>>>>>>> 3 Jobs Running

>>>>>>>>> 10 Jobs Scheduled

>>>>>>>>> 5 Jobs Done
          Retrieve them with: crab -getoutput <List of jobs>
          List of jobs: 1-5

In this case, we can tell that there are five jobs that have completed (successfully, we hope) and whose output is ready to be retrieved. Other jobs are still running, or have not started yet. We can start to gather output now or wait for the rest of the jobs to run and finish.

Retrieving job output

Once a job has status "Done" it's possible to retrieve the output back to the UI. The following command retrieves the output of all jobs with status done of the last created CRAB project:

crab -getoutput 
or to get the output of a specific project:
crab -getoutput -c  <dir name>

This command can be repeated as long as there are jobs in "Done" status.

The job results will be copied into the res/ subdirectory of your crab project. You should see output like this:

crab. crab (version 2.4.3) running on Fri Dec 26 15:52:00 2008

crab. Working options:
  scheduler           glite
  job type            CMSSW
  working directory   /uscms/home/ewv/Tutorial/CMSSW_2_1_12/src/UserArea/CraftAnalysis/test/crab_0_081226_154005/

crab. Starting retrieving output from server
crab. Results of Jobs # 1 are in /uscms/home/ewv/Tutorial/CMSSW_2_1_12/src/UserArea/CraftAnalysis/test/crab_0_081226_154005/res/
crab. Results of Jobs # 2 are in /uscms/home/ewv/Tutorial/CMSSW_2_1_12/src/UserArea/CraftAnalysis/test/crab_0_081226_154005/res/
crab. Results of Jobs # 10 are in /uscms/home/ewv/Tutorial/CMSSW_2_1_12/src/UserArea/CraftAnalysis/test/crab_0_081226_154005/res/
crab. Log-file is /uscms/home/ewv/Tutorial/CMSSW_2_1_12/src/UserArea/CraftAnalysis/test/crab_0_081226_154005/log/crab.log

Making the final plot

Each job produces a histogram output file which can be combined with the others using ROOT in the res directory:

cd crab_directory/res
ls *.root
hadd test.root test_*.root

The final test.root opened in ROOT contains the final plot:

root test.root
(or use the TBrowser if you prefer).

Storing data on a storage element

Next, we're going to change our grid job to store data on a storage element (not on local disk).

Prepare a dCache area at FNAL for storage element interaction

For CRAB to be able to write into your dCache user directory /pnfs/cms/WAX/resilient/<username>

we have to create a destination directory and change the file permissions:

mkdir /pnfs/cms/WAX/resilient/<username>
mkdir /pnfs/cms/WAX/resilient/<username>/tutorial 
chmod +775 /pnfs/cms/WAX/resilient/<username>/tutorial

replacing <username> with your username. The permission change allows the CMS group to write into your directory, not just you. Grid jobs run as a generic user in the CMS group, so you must allow write access to the whole group.

Changing crab.cfg

We have to change crab.cfg to enable writing data to dCache. Replace

return_data             = 1
return_data            = 0
copy_data              = 1
storage_element        =
storage_path           = /srm/managerv2?SFN=/resilient/<username>/tutorial
user_remote_dir    = run1
This instructs CRAB to copy the output files listed in
output_file = 
to dCache instead of making them available when you do crab -getoutput. This allows you to write much bigger output files (2GB vs. 50 MB) and is recommended for CMSSW ROOT files, but here we'll just use it for our histogram files. The rest of the output (stderr, stdout, etc.) is still retrieved with
crab -getoutput

Submit and check on the job with the same commands as before:

crab -create
crab -submit
crab -status 
crab -getoutput
but in addition, you should see your output (histogram) files begin to arrive in dCache
ls -l /pnfs/cms/WAX/resilient/<username>/tutorial/run1/ 

Going from here and more information

A full exploration of CRAB's capabilities is too much for this tutorial, so here are some things you can try on your own:

  • Use DBS to find your favorite dataset at a Tier2
  • Use CRAB for MC generation: Set dataset to "None" and use seeds as described in crab -help
  • Check out running CRAB in local mode, including "scheduler = condor" on the LPC CAF
  • Investigate CRAB's debugging options
    • crab -kill and crab -resubmit to kill and resubmit jobs
    • crab -postMortem returns Grid error messages
    • Realize that things will go wrong with the Grid as the load increases and as Grid experts adapt

There are a number of resources to help you

Edit | Attach | Watch | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r9 - 2012-04-12 - EricVaandering
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback