Basic description and usage of the automatized crab job submission script for the IBs.


Run a script triggered by a cronjob, which sends some 'standard' CMSSW jobs on real data to the grid.


The files are located here:

The cronjob starts Tester/ every 12h. Next to the usual statusFile (containing the time it started, finished), I set up a global statusFile (*now it's in my AFS area*), preventing to start new CAF jobs before the latest has finished.

For the script to run, cmssw and grid environmental variables has to be set (done by standaloneTester).
For the IB runs, we have to use custom CRAB build, since they're not supported by default. I set up one on vocms101 (/build/bbozsogi/CRAB_2_7_5), currently the script runs on this machine.

Basically the script:


I'll try to keep the --help up-to-date smile
By running the script without parameters, it does the things listed above.

You can change the default crab.cfg parameters (see CafQADefaults) by: -c ',CMSSW:events_per_job=2000'

You can send custom jobs to the CAF by using option -inputCmd=' ...' -d='' .
In this case it runs the with -no_exec option and use that config file as an input. For this, you also have to -d '/Electron/Run2010B-WZEG-v2/RAW-RECO' -i ' step2 -s RAW2DIGI,L1Reco,RECO,ALCA:SiStripCalZeroBias+SiStripCalMinBias,DQM 
--data --datatier RECO --eventcontent RECO --conditions auto:com10 --scenario pp --no_exec --magField AutoFromDBCurrent 
--process reRECO --customise Configuration/DataProcessing/ --cust_function customisePPData'


The pickle file contains a dictionary(logData) with the most important infos about the job.
it has a dictionary for every logFile in a stucture like:
logData[logName] = {
   'JobExitCode' = exitcode
   'files' = ['file1.root', 'file2.root', ...]
   'SE_PATH' = '/castor/.../'
   'MSG' = [((startNum, endNum), 'errorMessage'), ...]

The 'MSG' part is the most important, it's basically every block in the logs between %MSG tags.

A way to parse it (used it for the crabCAF.log):

        for key, value in logData.items():
            self.log.write(key+': {\n')
            for key2, value2 in value.items():
                if key2 == 'MSG':
                    map(lambda x: self.log.write('---'+key2+': '+'('+str(x[0][0])+'-'+str(x[0][1])+'):\n'+x[1]), value2)
                elif key2 == 'files':
                    self.log.write('---'+key2+': '+', '.join(value2).split()[0])
                    self.log.write('---'+key2+': '+value2)


Now, the cronjob is running under my user. We have to port it at some point to cmsbuild maybe.
I have sometimes IOError on the statusFiles, need to check why.
Some more error protection...

-- BalazsBozsogi - 17-Nov-2010

This topic: Sandbox > CafQA
Topic revision: r1 - 2010-11-17 - BalazsBozsogi
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback