CRAB @ FNAL Tutorial 12 October 2007

Introduction

This brief (two hour) tutorial will concentrate on Grid submission of CMSSW jobs. We will use CRAB to submit jobs to the grid and give an introduction to the most common use cases for CRAB.

Prerequisites and Requirements

We assume that the participants in this tutorial are already familiar with CMSSW analysis in interactive mode. Additionally, there are several prerequisites that must be satisfied in advance. Many of these require human intervention, so doing them during the tutorial is just not possible. In order to be able to submit Grid jobs from Fermilab, please make sure you have:

  • A valid UAF/LPC login account
  • A valid Grid Certificate
  • A certificate identifying you as belonging to the CMS VO

CRABatFNALTutorialOctober2007 explains how to do all of these things. Please make sure that before attending the tutorial you can log into cmsplc, issue the commands

voms-proxy-init -voms cms
and
voms-proxy-info -all
and see output which contains the line
attribute : /cms/Role=NULL/Capability=NULL

Login and prepare CMSSW interactive analysis

Note: whenever you see [c]sh you should use "csh" if your shell is csh or tcsh and "sh" if your shell is ash or bash. These instructions are for a simple, prepared analysis. If you have your own analysis, you are welcome to try that, but you may run into problems if the data you would like to run on is not widely distributed.

We'll begin by logging into the LPC cluster:

slogin cmslpc[11,12].fnal.gov
Next, make sure that the CMSSW environment is set up:
source /uscmst1/prod/sw/cms/[c]shrc prod
Now create a CMSSW project area:
scramv1 project CMSSW CMSSW_1_6_4
Lastly, unpack the sample analysis into this new area:
cd CMSSW_1_6_4/src
tar xvfz ~ewv/CrabSample2007Oct.tgz

This sample analysis is for Higgs -> ZZ; ZZ -> 4 mu, but without any muon ID. It creates just a few histograms so you can see it's working.

Build & run CMS interactive analysis

Build the CMSSW project:

scramv1 build
Look at the input file in higgs.cfg This file is just a part of the dataset we will be ultimately be running over. Lastly set up the CMSSW environment for 1_6_4 and run cmsRun with:
eval `scramv1 runtime -[c]sh`
cd Demo/MyTrackAnalyzer/src/
cmsRun higgs.cfg

When cmsRun finishes, you should have a file called "histograms.root" in the Demo/MyTrackAnalyzer/src/ directory. It has just a few histograms in it. You should check it with root to verify that things look OK. You always want to run a short job locally before submitting your analysis to the Grid for two reasons:

  1. Make sure the output is what you expect (for a small input set)
  2. If you have problems with your submission, it is nice to know that your code is not the problem

Locate the dataset and prepare CRAB submission

Now that we've verified that our analysis works OK, we're ready to run over the whole dataset. But we have to find it first. We'll use DBS to find the dataset. In this case, we're interested in Higgs -> 4 mu decays. The closet we can find that's been produced recently is Higgs -> 4 leptons. Point your web browser at http://cmsdbs.cern.ch/DBS2_discovery/. DBS has lots of ways to find datasets, but in this case:

  • Select CMSSW_1_6_4 under "Software release"
  • Select RelValHiggsZZ4LM190 under "Primary dataset"
  • Press find

DBS now tells us that the full name of our dataset is /RelValHiggsZZ4LM190/CMSSW_1_6_4-RelVal-1191064565/GEN-SIM-DIGI-RECO and shows us where the data is located. This is information we have to give to CRAB. As an extra benefit, DBS will help us configure CRAB. Click on more (just to the right of the dataset name) and then click on crab.cfg. Copy and paste this output to a new file named crab.cfg in the src/ directory in the cmslpc session you used to run cmsRun. The file should look like this:

[CRAB]

jobtype                 = cmssw
scheduler               = glitecoll

[CMSSW]

datasetpath             = /RelValHiggsZZ4LM190/CMSSW_1_6_4-RelVal-1191064565/GEN-SIM-DIGI-RECO
pset                    = 
total_number_of_events  = -1
events_per_job          = 1000
output_file             = 

[USER]
return_data             = 1
use_central_bossDB      = 0
use_boss_rt             = 0

[EDG]
rb                      = CERN 
proxy_server            = myproxy.cern.ch 
virtual_organization    = cms
retry_count             = 0
lcg_catalog_type        = lfc
lfc_host                = lfc-cms-test.cern.ch
lfc_home                = /grid/cms

We need to make a few changes and fill in the blanks in this CRAB configuration file:

  • We're going to start off using a different scheduler, so replace the scheduler line with
scheduler               = condor_g
  • We need to tell CRAB what our input CMSSW .cfg and output files are, so change those two lines to read
pset                    = higgs.cfg
output_file             = histograms.root
  • Lastly, we need to tell condor_g where to submit the jobs, so to the [EDG] section, add these lines
ce_white_list = tier2.edu
se_white_list = tier2.edu
but replace tier2.edu with one of the six Tier2 sites that DBS says has this dataset. This will submit only to computing and storage elements whose names match "tier2.edu." Warning: for some Tier2 sites you may have to specify the CE part to be more specific (there are two CEs at FNAL). CRAB will tell you if this is the case.

Submit an analysis to the Grid

Now that we've prepared everything, we're ready to run the jobs following the four CRAB steps of creating, submitting, checking (as needed), and retrieving the output of our jobs.

Set up crab

For this tutorial,

source /uscmst1/prod/grid/CRAB_1_5_4/crab.[c]sh
in general, you should
source /uscmst1/prod/grid/CRAB/crab.[c]sh
to get the latest production version of CRAB.

Job creation

The job creation uses the CRAB configuration file you just wrote to check the availability of the selected dataset and prepares the jobs for submission according to the selected job splitting:

crab -create

which uses the default name for a CRAB configuration file: crab.cfg. If a different CRAB configuration file should be used, the command becomes

crab -create -cfg <configuration file>

If you haven't used your Grid certificate in the last day or two, you'll be asked for the password to create a new proxy.

The creation process creates a CRAB project directory (format =crab_<number>_<date>_<time>) in the current working directory. It can be used later to distinguish multiple CRAB projects in the same directory.

The used CRAB configuration file is copied into the CRAB project directory, so the original can be changed and used again without interfering with the already created projects.

This is an example of the output of the creation command:

crab. crab (version 1.5.4) running on Tue Oct  9 13:42:39 2007

crab. Working options:
  scheduler           condor_g
  job type            CMSSW
  working directory   /uscms/home/ewv/crab-test/CMSSW_1_6_4/src/Demo/MyTrackAnalyzer/src/crab_0_071009_134239/

crab. Contacting Data Discovery Services ...
crab. Requested dataset: /RelValHiggsZZ4LM190/CMSSW_1_6_0-RelVal-1189471579/GEN-SIM-DIGI-RECO has 5025 events in 2 blocks.

crab. 6 job(s) can run on 5025 events.

crab. List of jobs and available destination sites:

Block     1: jobs                  1-5: sites: cmssrm.fnal.gov
Block     2: jobs                    6: sites: cmssrm.fnal.gov

crab. Creating 6 jobs, please wait...

crab. Total of 6 jobs created.

crab. Log-file is /uscms/home/ewv/crab-test/CMSSW_1_6_4/src/Demo/MyTrackAnalyzer/src/crab_0_071009_134239/log/crab.log

If there are problems, a good place to look for output is the CRAB log file in crab_?_*_*/log/crab.log.

Job submission

The following command submits the previously created jobs:

crab -submit all -c

where -c specifies that CRAB uses the latest CRAB project in the current directory and all means to submit all created jobs. You can also specify and combination of jobs and job-ranges separated by commas (Example: =1,2,4-6). You can force CRAB to use a different CRAB project than the most recently created one:

crab -submit all -c <crab_directory>

The output of this command should look like:

crab. crab (version 1.5.4) running on Tue Oct  9 13:48:02 2007

crab. Working options:
  scheduler           condor_g
  job type            CMSSW
  working directory   /uscms/home/ewv/crab-test/CMSSW_1_6_4/src/Demo/MyTrackAnalyzer/src/crab_0_071009_134239/

crab. Found 1 compatible site(s) for job 1
                                   Submitting 6 jobs
100% [===============================================================]
                                       please wait
crab. Total of 6 jobs submitted.

crab. Total of 6 jobs submitted.
crab. Log-file is /uscms/home/ewv/crab-test/CMSSW_1_6_4/src/Demo/MyTrackAnalyzer/src/crab_0_071009_134239/log/crab.log

Checking job status

Check the status of the jobs in the latest CRAB project with the following command:

crab -status 
or to check an earlier project:
crab -status -c  <crab_directory>

which should produce output similar to:

crab. crab (version 1.5.4) running on Tue Oct  9 13:55:58 2007

crab. Working options:
  scheduler           condor_g
  job type            CMSSW
  working directory   /uscms/home/ewv/crab-test/CMSSW_1_6_4/src/Demo/MyTrackAnalyzer/src/crab_0_071009_134239/

crab. Checking the status of all jobs: please wait
Chain    STATUS             E_HOST                                   EXE_EXIT_CODE JOB_EXIT_STATUS
---------------------------------------------------------------------------------------------------
1        Done (Success)     cmsosgce.fnal.gov
2        Running            cmsosgce.fnal.gov
3        Running            cmsosgce.fnal.gov
4        Running            cmsosgce.fnal.gov
5        Running            cmsosgce.fnal.gov
6        Running            cmsosgce.fnal.gov

>>>>>>>>> 6 Total Jobs

>>>>>>>>> 5 Jobs Running
          List of jobs: 2,3,4,5,6

>>>>>>>>> 1 Jobs Done
          List of jobs: 1
          Retrieve them with: crab -getoutput <Jobs list>
crab. Log-file is /uscms/home/ewv/crab-test/CMSSW_1_6_4/src/Demo/MyTrackAnalyzer/src/crab_0_071009_134239/log/crab.log

In this case, we can tell that there is one job that has completed successfully and whose output is ready to be retrieved. Five jobs are still running. We can start to gather output now or wait for the rest of the jobs to finish.

Retrieving job output

Once a job has status "Done" it's possible to retrieve the output back to the UI. The following command retrieves the output of all jobs with status done of the last created CRAB project:

crab -getoutput 
or to get the output of a specific project:
crab -getoutput -c  <dir name>

This command can be repeated as long as there are jobs in "Done" status.

The job results will be copied into the res/ subdirectory of your crab project. You should see output like this:

crab. crab (version 1.5.4) running on Tue Oct  9 15:23:57 2007

crab. Working options:
  scheduler           condor_g
  job type            CMSSW
  working directory   /uscms/home/ewv/crab-test/CMSSW_1_6_4/src/Demo/MyTrackAnalyzer/src/crab_0_071009_134239/

crab. Results of Job # 2 are in /uscms/home/ewv/crab-test/CMSSW_1_6_4/src/Demo/MyTrackAnalyzer/src/crab_0_071009_134239/res/
crab. Results of Job # 3 are in /uscms/home/ewv/crab-test/CMSSW_1_6_4/src/Demo/MyTrackAnalyzer/src/crab_0_071009_134239/res/
crab. Results of Job # 4 are in /uscms/home/ewv/crab-test/CMSSW_1_6_4/src/Demo/MyTrackAnalyzer/src/crab_0_071009_134239/res/
crab. Results of Job # 5 are in /uscms/home/ewv/crab-test/CMSSW_1_6_4/src/Demo/MyTrackAnalyzer/src/crab_0_071009_134239/res/
crab. Results of Job # 6 are in /uscms/home/ewv/crab-test/CMSSW_1_6_4/src/Demo/MyTrackAnalyzer/src/crab_0_071009_134239/res/
1 jobs already cleared

crab. Log-file is /uscms/home/ewv/crab-test/CMSSW_1_6_4/src/Demo/MyTrackAnalyzer/src/crab_0_071009_134239/log/crab.log

Making the final plot

All 10 jobs produce a histogram output file which can be combined using ROOT in the res directory:

cd crab_directory/res
ls *.root
hadd histograms.root histograms_*.root

The final histograms.root opened in ROOT contains the final plot:

root histograms.root
mzz->Draw();
(or use the TBrowser if you prefer).

A more sophisticated Grid Analysis

Next, we're going to change two things about our grid job:

  • Instead of collecting the output, we'll have it automatically written to Fermilab's dCache system
  • We'll change the scheduler so that we can run our jobs in multiple locations (SEs and CEs)

Prepare a dCache area at FNAL for storage element interaction

For CRAB to be able to write into your dCache user directory /pnfs/cms/WAX/resilient/<username>

we have to create a destination directory and change the file permissions:

mkdir /pnfs/cms/WAX/resilient/<username>
mkdir /pnfs/cms/WAX/resilient/<username>/tutorial 
chmod +775 /pnfs/cms/WAX/resilient/<username>/tutorial

replacing <username> with your username. The permission change allows the CMS group to write into your directory, not just you.

Changing crab.cfg

For the scheduler change, modify the line

scheduler               = condor_g
to instead read
scheduler               = glite
and delete (or comment them out by putting # at the start of each line) the two lines you added earlier
ce_white_list = tier2.edu
se_white_list = tier2.edu

The second round of changes to crab.cfg enables the writing of data to dCache. Replace

return_data             = 1
with
return_data            = 0
copy_data              = 1
storage_element        = cmssrm.fnal.gov
storage_path           = /srm/managerv1?SFN=/resilient/<username>/tutorial
This instructs CRAB to copy the output files listed in
output_file = 
to dCache instead of making them available when you do crab -getoutput. This allows you to write much bigger output files (2GB vs. 100 MB) and is recommended for CMSSW ROOT files, but here we'll just use it for our histogram files. The rest of the output (stderr, stdout, etc.) is still retrieved with
crab -getoutput

Submit and check on the job with the same commands as before:

crab -create
crab -submit all
crab -status 
crab -getoutput
but in addition, you should see your output (histogram) files begin to arrive in dCache
ls -l /pnfs/cms/WAX/resilient/<username>/tutorial 

Going from here and more information

A full exploration of CRAB's capabilities is too much for this tutorial, so here are some things you can try on your own:

  • Use DBS to find your favorite dataset coming from CSA07 to a Tier2 (Physics Group CSA07)
  • Use CRAB for MC generation: Set dataset to "None" and use seeds as described in crab -help
  • Check out other schedulers
    • EDG: Going away, but still sometimes useful
    • glitecoll: A bulk submission mode for glite
    • For now use CRAB_1_5_3 on regular cmslpc nodes
  • Investigate CRAB's debugging options
    • crab -kill and crab -resubmit to kill and resubmit jobs
    • Realize that things will go wrong with the Grid as the load increases and as Grid experts adapt

There are a number of resources to help you

A number of talks and tutorials from the June 2007 workshop may be quite useful. This was a full day tutorial, so it was much more comprehensive. Talks were given on each aspect of the computing model and how they relate.

-- EricVaandering - 08 Oct 2007

Edit | Attach | Watch | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r9 - 2007-10-11 - EricVaandering
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback