CRAB3 Cheat Sheet

Perhaps someone's been nagging you to give CRAB3 a try, you're tired of CRAB2 stageout errors plaguing your analysis, or you don't enjoy waiting days for CRAB2 to publish your output datasets. If you would like the bare-minimum information to get started with CRAB3, you've come to the right place.

Environment Setup

Like all CMS software, you need to first start by configuring your environment to see CRAB3. Fortunately any site with CVMFS (which is nearly all sites) will have CRAB3 installed already, you just need to configure the environment for CMS, the grid UI, and CRAB3. If for some reason your site doesn't have CVMFS installed, you will need to request that the admins install CRAB3 manually. Instructions on doing this can be found at the main CRAB3 twiki page. Additionally, keep in mind that CRAB2 and CRAB3 are incompatible, so make sure you don't have both softwares loaded simultaneously (some people source the CRAB2 setup script from their ~/.bashrc, don't do that). Finally, all these instructions assume that you have a valid, registered grid certificate installed in your local home directory.

At many sites, the grid UI already comes with the environment (if you can run grid-proxy-init, then your environment already has the grid UI). If you already have the grid UI, the commands are:

cmsenv
source /cvmfs/cms.cern.ch/crab3/crab.sh

Otherwise, you'll need to also source a grid UI. At CERN, the commands are:

cmsenv
source /afs/cern.ch/cms/LCG/LCG-2/UI/cms_ui_env.sh
source /cvmfs/cms.cern.ch/crab3/crab.sh

But different sites have their UIs stored in different locations (ask your admins!)

CRAB3 Configurations

CRAB3 configurations are written in python (compared with CRAB2s ini-based format). This allows more powerful and flexible configurations, but also means all old configurations will need to be converted to the new format. Please also note that CRAB3 configuration files must end in ".py" to be properly imported.

Fortunately, a CRAB2 to CRAB3 conversion script exists, and will usually produce a working CRAB3 configuration, removing obsolete options and warning about incompatible options. This script is named "crab2cfgTOcrab3py [crab2configName.cfg] [crab3configName.py]" and can be executed after sourcing the CRAB3 environment.

Sample CRAB Configurations

If you would like to start from scratch, the following example configurations are sufficient to get started. Make sure to replace the colored sections with your own values

Show Hide Data processing config.
from WMCore.Configuration import Configuration

config = Configuration()

config.section_("General")
config.General.requestName   = 'test1'
config.General.saveLogs = True

config.section_("JobType")
config.JobType.pluginName  = 'Analysis'
# Name of the CMSSW configuration file
config.JobType.psetName    = 'pset.py'

config.section_("Data")
config.Data.inputDataset = '/GenericTTbar/HC-CMSSW_5_3_1_START53_V5-v1/GEN-SIM-RECO'
config.Data.splitting = 'LumiBased'
config.Data.unitsPerJob = 100
config.Data.publication = True
# This string is used to construct the output dataset name
config.Data.publishDataName = 'CRAB3-tutorial'

# These values only make sense for processing data
#    Select input data based on a lumi mask
config.Data.lumiMask = 'Cert_190456-208686_8TeV_PromptReco_Collisions12_JSON.txt'
#    Select input data based on run-ranges
config.Data.runRange = '190456-194076'

config.section_("Site")
# Where the output files will be transmitted to
config.Site.storageSite = 'T2_US_Nowhere'

Show Hide private Monte Carlo generation config.
from WMCore.Configuration import Configuration

config = Configuration()

config.section_("General")
config.General.requestName   = 'test1'
config.General.saveLogs = True

config.section_("JobType")
config.JobType.pluginName  = 'PrivateMC'
# Name of the CMSSW configuration file
config.JobType.psetName    = 'pset.py'

config.section_("Data")
# This string determines the primary dataset of the newly-produced outputs.
# For instance, this dataset will be named /CrabTestSingleMu/something/USER
config.Data.inputDataset = '/CrabTestSingleMu'
config.Data.splitting = 'EventBased'
config.Data.unitsPerJob = 100
config.Data.totalUnits = 1000
config.Data.publication = True

# This string is used to construct the output dataset name
config.Data.publishDataName = 'CRAB3-tutorial'

config.section_("Site")
# Where the output files will be transmitted to
config.Site.storageSite = 'T2_US_Nowhere'

Run CRAB3

Once you have a CRAB3 configuration and a valid CMSSW configuration, you're ready to run CRAB3! These examples assume that you've named your crab configuration crabConfig.py and left the default value for config.General.requestName.

  • Submit the jobs:
    crab submit -c crabConfig.py
    
  • Check the job status:
    crab status -t crab_test1
    
  • Retrieve the logs from all completed jobs:
    crab getlog -t crab_test1 -q all
    
  • See the list of good lumis for your task:
    crab report -t crab_test1
    

Similar to CRAB2's behavior, submitting a CRAB3 task creates a directory storing all the information about the request. the -t option for the other commands accepts this crab_ directory as an input.

More information

For a more detailed manual, please see the CRAB3 workbook or consult with the experts at the CRAB development HN forum: hn-cms-crabDevelopment@cernNOSPAMPLEASE.ch

Sample CMSSW configurations

If you somehow manage to not have any CMSSW configuration sitting around, you can use these to test CRAB3. They are known to work with CMSSW_5_3_4

import FWCore.ParameterSet.Config as cms
process = cms.Process('NoSplit')
process.source = cms.Source('PoolSource',
  fileNames = cms.untracked.vstring("/store/mc/HC/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0010/FC85224E-EAAD-E111-AB01-0025901D629C.root"),
  skipEvents = cms.untracked.uint32(0),
)

process.dump = cms.EDAnalyzer("EventContentAnalyzer", listContent=cms.untracked.bool(False), getData=cms.untracked.bool(True))
process.load("FWCore.MessageService.MessageLogger_cfi")
process.MessageLogger.cerr.FwkReport.reportEvery = 10

process.maxEvents = cms.untracked.PSet(
    input = cms.untracked.int32(50)
)

process.Timing = cms.Service("Timing",
    useJobReport = cms.untracked.bool(True),
    summaryOnly = cms.untracked.bool(True),
)

process.o = cms.OutputModule("PoolOutputModule", fileName = cms.untracked.string("dumper.root"), outputCommands=cms.untracked.vstring("drop *"))

process.out = cms.EndPath(process.o)

process.p = cms.Path(process.dump)

Show PrivateMC CMSSW configuration... Close
# Auto generated configuration file
# using: 
# Revision: 1.381.2.11 
# Source: /local/reps/CMSSW/CMSSW/Configuration/PyReleaseValidation/python/ConfigBuilder.py,v 
# with command line options: TTbar_Tauola_7TeV_cfi.py -s GEN,FASTSIM,HLT:GRun --conditions=auto:startup_GRun --pileup=NoPileUp --geometry DB --eventcontent=AODSIM --datatier AODSIM -n 10 --no_exec
import FWCore.ParameterSet.Config as cms

process = cms.Process('HLT')

# import of standard configurations
process.load('Configuration.StandardSequences.Services_cff')
process.load('SimGeneral.HepPDTESSource.pythiapdt_cfi')
process.load('FWCore.MessageService.MessageLogger_cfi')
process.load('FastSimulation.Configuration.EventContent_cff')
process.load('FastSimulation.PileUpProducer.PileUpSimulator_NoPileUp_cff')
process.load('FastSimulation.Configuration.Geometries_START_cff')
process.load('Configuration.StandardSequences.MagneticField_38T_cff')
process.load('Configuration.StandardSequences.Generator_cff')
process.load('GeneratorInterface.Core.genFilterSummary_cff')
process.load('FastSimulation.Configuration.FamosSequences_cff')
process.load('IOMC.EventVertexGenerators.VtxSmearedParameters_cfi')
process.load('HLTrigger.Configuration.HLT_GRun_Famos_cff')
process.load('Configuration.StandardSequences.FrontierConditions_GlobalTag_cff')

process.maxEvents = cms.untracked.PSet(
    input = cms.untracked.int32(10)
)

# Input source
process.source = cms.Source("EmptySource")

process.options = cms.untracked.PSet(

)

# Production Info
process.configurationMetadata = cms.untracked.PSet(
    version = cms.untracked.string('$Revision: 1.10 $'),
    annotation = cms.untracked.string('TTbar_Tauola_7TeV_cfi.py nevts:10'),
    name = cms.untracked.string('PyReleaseValidation')
)

# Output definition

process.AODSIMoutput = cms.OutputModule("PoolOutputModule",
    eventAutoFlushCompressedSize = cms.untracked.int32(15728640),
    outputCommands = process.AODSIMEventContent.outputCommands,
    fileName = cms.untracked.string('MyTTBarTauolaTest.root'),
    dataset = cms.untracked.PSet(
        filterName = cms.untracked.string(''),
        dataTier = cms.untracked.string('AODSIM')
    ),
    SelectEvents = cms.untracked.PSet(
        SelectEvents = cms.vstring('generation_step')
    )
)

# Additional output definition

# Other statements
process.genstepfilter.triggerConditions=cms.vstring("generation_step")
process.famosSimHits.SimulateCalorimetry = True
process.famosSimHits.SimulateTracking = True
process.simulation = cms.Sequence(process.simulationWithFamos)
process.HLTEndSequence = cms.Sequence(process.reconstructionWithFamos)
process.Realistic8TeVCollisionVtxSmearingParameters.type = cms.string("BetaFunc")
process.famosSimHits.VertexGenerator = process.Realistic8TeVCollisionVtxSmearingParameters
process.famosPileUp.VertexGenerator = process.Realistic8TeVCollisionVtxSmearingParameters
from Configuration.AlCa.GlobalTag import GlobalTag
process.GlobalTag = GlobalTag(process.GlobalTag, 'auto:startup_GRun', '')

process.generator = cms.EDFilter("Pythia6GeneratorFilter",
    ExternalDecays = cms.PSet(
        Tauola = cms.untracked.PSet(
            UseTauolaPolarization = cms.bool(True),
            InputCards = cms.PSet(
                mdtau = cms.int32(0),
                pjak2 = cms.int32(0),
                pjak1 = cms.int32(0)
            )
        ),
        parameterSets = cms.vstring('Tauola')
    ),
    pythiaPylistVerbosity = cms.untracked.int32(0),
    filterEfficiency = cms.untracked.double(1.0),
    pythiaHepMCVerbosity = cms.untracked.bool(False),
    comEnergy = cms.double(7000.0),
    maxEventsToPrint = cms.untracked.int32(0),
    PythiaParameters = cms.PSet(
        pythiaUESettings = cms.vstring('MSTJ(11)=3     ! Choice of the fragmentation function', 
            'MSTJ(22)=2     ! Decay those unstable particles', 
            'PARJ(71)=10 .  ! for which ctau  10 mm', 
            'MSTP(2)=1      ! which order running alphaS', 
            'MSTP(33)=0     ! no K factors in hard cross sections', 
            'MSTP(51)=10042 ! structure function chosen (external PDF CTEQ6L1)', 
            'MSTP(52)=2     ! work with LHAPDF', 
            'MSTP(81)=1     ! multiple parton interactions 1 is Pythia default', 
            'MSTP(82)=4     ! Defines the multi-parton model', 
            'MSTU(21)=1     ! Check on possible errors during program execution', 
            'PARP(82)=1.8387   ! pt cutoff for multiparton interactions', 
            'PARP(89)=1960. ! sqrts for which PARP82 is set', 
            'PARP(83)=0.5   ! Multiple interactions: matter distrbn parameter', 
            'PARP(84)=0.4   ! Multiple interactions: matter distribution parameter', 
            'PARP(90)=0.16  ! Multiple interactions: rescaling power', 
            'PARP(67)=2.5    ! amount of initial-state radiation', 
            'PARP(85)=1.0  ! gluon prod. mechanism in MI', 
            'PARP(86)=1.0  ! gluon prod. mechanism in MI', 
            'PARP(62)=1.25   ! ', 
            'PARP(64)=0.2    ! ', 
            'MSTP(91)=1      !', 
            'PARP(91)=2.1   ! kt distribution', 
            'PARP(93)=15.0  ! '),
        processParameters = cms.vstring('MSEL      = 0     ! User defined processes', 
            'MSUB(81)  = 1     ! qqbar to QQbar', 
            'MSUB(82)  = 1     ! gg to QQbar', 
            'MSTP(7)   = 6     ! flavour = top', 
            'PMAS(6,1) = 175.  ! top quark mass'),
        parameterSets = cms.vstring('pythiaUESettings', 
            'processParameters')
    )
)


# Path and EndPath definitions
process.generation_step = cms.Path(process.pgen_genonly)
process.reconstruction = cms.Path(process.reconstructionWithFamos)
process.genfiltersummary_step = cms.EndPath(process.genFilterSummary)
process.AODSIMoutput_step = cms.EndPath(process.AODSIMoutput)

# Schedule definition
process.schedule = cms.Schedule(process.generation_step,process.genfiltersummary_step)
process.schedule.extend(process.HLTSchedule)
process.schedule.extend([process.reconstruction,process.AODSIMoutput_step])
# filter all path with the production filter sequence
for path in process.paths:
	getattr(process,path)._seq = process.generator * getattr(process,path)._seq 

# customisation of the process.

# Automatic addition of the customisation function from HLTrigger.Configuration.customizeHLTforMC
from HLTrigger.Configuration.customizeHLTforMC import customizeHLTforMC 

#call to customisation function customizeHLTforMC imported from HLTrigger.Configuration.customizeHLTforMC
process = customizeHLTforMC(process)

# End of customisation functions

-- AndresTanasijczuk - 07 Oct 2014

Edit | Attach | Watch | Print version | History: r10 | r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r1 - 2014-10-07 - AndresTanasijczuk
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback