CRAB Logo

Software Guide on CRAB

Complete: 3
This page contains documentation about CRAB3. The old CRAB2 page has been moved to SWGuideCrab2.

CRAB is a utility to submit CMSSW jobs to distributed computing resources. Currently we are in a transition phase, encouraging users to move from CRAB version 2 towards CRAB version 3 if possible. This page is being updated accordingly to contain documentation about CRAB3. The old CRAB2 page has been moved to SWGuideCrab2.

Introduction

CRAB is a utility to submit CMSSW jobs to distributed computing resources. By using CRAB you will be able to:

  • Access CMS data and Monte-Carlo which are distributed to CMS aligned centres worldwide.
  • Exploit the CPU and storage resources at CMS aligned centres.
Before starting to learn about CRAB, you may want to get an overview of the Grid model and of a typical analysis workflow. For that purpose read the following chapters of the CMS Offline WorkBook: For detailed documentation about CRAB, see section Advanced CRAB Documentation further down in this page.

Differences between CRAB2 and CRAB3

For those who already know about CRAB2 and have to face the transition to CRAB3, here is a list of architecture improvements (or just differences) in the new version.

  • Asynchronous copy of the output files to the destination storage element by an external service called Asynchronous Stage Out (ASO). This allows to have managed file transfers and releases the worker node immediately when the processing job has finished. Impact: the user will no longer have jobs that run and then fail at the end because they cannot move their data.
  • Automatic publication of the output dataset in DBS. No need to run the crab -publish command anymore.
  • Automatic job resubmission for certain failures.
  • Job splitting takes place in the server:
    • The crab -create command is therefore not available anymore and the workflow starts directly from the submission command.
    • The submission command does not accept anymore the range of jobs option (all jobs that are created by the server are also submitted to run).
  • The CRAB3 configuration file is written in python.

There are also features that are not yet implemented in CRAB3:

  • Storage in an arbitrary location is not supported. (CRAB3 only supports storage in /store/user/ and /store/group/.)
  • Local submission is not implemented.

An up-to-date roadmap to what works and what does not in CRAB3 is in CRAB3Functionalities.

The plan for migration user community from Crab2 to CRAB3 is in CRAB3Migration.

How is running CRAB3 is different than CRAB2

  • Commands do not require the - in front of them. That is, a CRAB2 command crab -<command> became crab <command> in CRAB3.
  • The --continue/-c option was replaced by --dir/-d.
  • The option used to specify the configuration file in the submission command is --config/-c.

CRAB3 documentation

CRAB3 releases
Summary of CRAB3 releases since October 2014 with list of new features, improvements and bug fixes.

Help Documentation for beginner users

Quickstart guide for CRAB3
A "cheat sheet" for getting started with CRAB3.
CRAB tutorial (introductory)
This is the most complete and recommended documentation about CRAB3 for beginners.
Help CRAB Troubleshooting
What to do if something does not work. A must read
CRAB tutorial (advanced)
Exercises for CRAB3 Advanced Tutorial 22 May 2015. Covers: dryrun, recovery task, scriptExe, LHE, CRABAPI library
CRAB configuration file
Documentation of the CRAB3 configuration file parameters.
CRAB commands
Documentation of the CRAB3 commands.
CRAB user functions
Documentation of the CRAB3 user functions.
CRAB advanced topics
Provisory page with documentation of different topics about CRAB3 (e.g. generating MC with LHE files, using scriptExe).
Tips and tricks about CRAB
Tips and tricks about CRAB3.
CRAB3 frequently asked questions
You may also want to consult the CRAB2 frequently asked questions page as some questions are common for CRAB2 and CRAB3.

Documentation for more expert users

Data handling in CRAB
An overview of how CRAB3 handles input data, output data and data publication.
CRAB3 task flow
A description of the steps involved in a typical CRAB3 task with description of CRAB3 architecture.
CRAB3 job states
In-depth information about job states in CRAB3.
CRAB3 client API
Details on how to use the CRAB3 Client library API.

Documentation for CRAB3 operators, developers, admins, experts

New operator TODO list
TODO list new operators should go through

Installation

CRAB3 frontend installation
Documentation about how to deploy the CRAB server frontend (i.e. CRAB REST Interface and CRAB Cache).
CRAB3 backend installation
Documentation about how to deploy the CRAB server backend, a.k.a. the CRAB TaskWorker.
CRAB3 Schedd installation and deployment
Documentation for CRAB3 Schedd deployment.
CRAB3 client installation
Documentation for operators of the CRAB3 client.
ASO installation and deployment
Documentation for ASO deployment and operations.
Notes about server deployment
Notes about the current deployment of CRAB3 servers.
CRAB3 Puppet profile
Documentation of the current Puppet profile of the project

Technical documents

Operator debugging tips
Handy commands for operators.
CRAB3 State Machine
Documentation about the Taks status tracking inside CRAB Server Data Base
CMS job exit codes
List of known exit codes for CMSSW and CRAB3.
Notes about CRAB3 service Certificates management
Documentation about the management of all the service certificates in the CRAB3 machines.
Notes about CRAB3 Oracle Database management
Documentation about the management of the oracle database in the CRAB3 project.
Inside Documentation
Miscellaneous collection of useful insights into how CRAB pieces work internally

Code repositories

CrabCodeDevelopment
Short guide to using github for Crab
CRAB Client
CRAB Server
REST, CrabCache, TW, new Publisher
ASO (old)
old ASO + Publisher in separate server
Submission Infrastructure Scripts
includes e.g. all scripts that we run in various ways on CRAB schedd's
Oracle Database management scripts
Puppet

Release management

Notes about CRAB3 release management
Documentation about releasing CRAB3 patches
Notes about CRAB3 validation
Some notes about the deployment procedure and validation process of new CRAB3 versions

#Monitoring

Monitoring

New Monitoring:

Old Monitoring:

Links to relevant projects

FNAL LPC Submission
submission via CRAB3 to FNAL LPC
glideInWMS for CMS documentation
main entry to Submission Infrastructure

Local Submission

CRAB vs CAF in Run 2
Notes about CAF use cases after discussion between Stefano and several DPG people.
Local Resource Provisioning
Notes on the different possibilities to submit to resources dedicated to local users.
Submitting jobs to the CERN HTCondor pool using 'crab preparelocal'
A different approach on how to independently submit CRAB jobs to a local batch system.

Getting started

This section describes how to begin using CRAB to perform your analysis on the Grid.

Prerequisites

To use CRAB to submit CMSSW jobs to the Grid, you must meet some prerequisites, which you can find in the CRAB Prerequisites page.

Basic workflows

Some basic workflows are documented in the CRAB3 Tutorial page. If you have never used CRAB before, it is a good place to learn how to use the tool.

Help Getting support

BEFORE CONTACTING Computing Tools, PLEASE CONSULT THE CRAB TROUBLESHOOTING GUIDE AT THIS LINK:
CRAB3Troubleshoot

and the CRAB3 Frequently Asked Questions page (and maybe even the CRAB2 Frequently Asked Questions page).

A common mistake is to test the code locally only on a small number of events which is not representative of the whole dataset. If the log of a failed job contains a CMSSW error (e.g. exit code 134, 139), please run your code interactively on the very same event on which the CRAB job failed.

All CRAB users must subscribe to the CERN Computing Announcements HyperNews Forum. It is very low traffic and if there is a global problem you will know about it faster in that way.

To send feedback and/or get support about CRAB3, please write to the Computing Tools CMS HyperNews forum. You can do it either via e-mail to hn-cms-computing-tools@cernNOSPAMPLEASE.ch or using the web form https://hypernews.cern.ch/HyperNews/CMS/get/computing-tools.html (the later is recommended, as messages via e-mail have a significant delay to reach other users subscribed to the CMS HyperNews). All CRAB users may find it useful to subscribe to this forum.

When contacting Computing Tools, you must:

  • write a brief description of the problem and copy/paste text of relevant error messages
  • try to be specific ("I have errors, please help me" will not get you an useful reply)
  • give pointers to relevant task information , e.g. by copy/pasting the first few lines form crab status output. In general make sure that you report at least:
    • task name (and a job number, if applicable)
    • the URL printed in green by the crab status command at the line: Task URL to use for HELP
  • if a problem on the client side is suspected, upload the crab.log file (use the crab uploadlog command) corresponding to the task for which you are reporting the problem
  • This command:
    crab status -d  crab_20170616_143152
    Produces:
    CRAB project directory:       /afs/cern.ch/work/b/belforte/CRAB3/TC3/crab_20170616_143152
    Task name:          170616_123158:belforte_crab_20170616_143152
    Grid scheduler:          crab3@vocms0197.cern.ch
    Status on the CRAB server: SUBMITTED
    Task URL to use for HELP:    https://cmsweb.cern.ch/crabserver/ui/task/170616_123158%3Abelforte_crab_20170616_143152
    Dashboard monitoring URL:    http://dashb-cms-job.cern.ch/dashboard/templates/task-analysis/#user=belforte&refresh=0&table=Jobs&p=1&records=25&activemenu=2&status=&site=&tid=170616_123158%3Abelforte_crab_20170616_143152
    Status on the scheduler:    COMPLETED
    
    Jobs status:                    finished      100.0% (2/2)
    So you would have to paste in your mail at least this URL
    https://cmsweb.cern.ch/crabserver/ui/task/170616_123158%3Abelforte_crab_20170616_143152
    while of course also copying/pasting the full text above would be good, and even better.

Other relevant CMS HyperNews forums

Useful links

-- MarcoCalloni - 27-Jan-2010

Edit | Attach | Watch | Print version | History: r159 < r158 < r157 < r156 < r155 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r159 - 2018-11-30 - TodorTrendafilovIvanov



 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback