Tips and tricks about CRAB

Complete: 3 Go to SWGuideCrab

Tips and tricks about CRAB

Retrieving the CERN username in the CRAB configuration file

CRAB provides a function getUsernameFromSiteDB (available in the CRABClient.ClientUtilities module) that retrieves the CERN username from SiteDB. This function can be used in the CRAB configuration file. The most common use case is in the Data.outLFN parameter, where users may need to put their CERN username. Below it is shown how to import and use the function in the CRAB configuration file.

from CRABClient.ClientUtilities import config, getUsernameFromSiteDB
config = config()

## Define a variable with the CMS HyperNews username.
my_cern_username = getUsernameFromSiteDB()

## Use the variable my_cern_username in the Data.outLFN parameter.
config.Data.outLFN = '/store/user/' + my_cern_username + '/bla'

note.gif Note: Using the function within CRAB is safe in terms of proxy validity, because the crab submit command checks before loading the configuration file if the user has a valid proxy and creates one if not. Using the function outside CRAB requires the user to have a valid proxy; if not, the function will raise ProxyException.

Dynamically changing the request name in the CRAB configuration file

The fact the the CRAB configuration file is written in python gives to the user lot of flexibility when defining the CRAB configuration parameter values dynamically. Below we give an example where we added some lines of code to the CRAB configuration file in order to define the parameter General.requestName as tasknumber<index>, where the index is a non-negative integer number that follows a sequence and is helping to organize a bunch of tasks that have otherwise the same name. The code starts by setting the index variable to 0; it then starts a loop where it checks if the CRAB project directory corresponding to the current index already exists, and if it does increments the index by 1 and if it doesn't stops the loop.

from CRABClient.ClientUtilities import config
config = config()

config.General.workArea = 'crab_projects'
## --- Auxiliary code starts here --- ##
## We want to define config.General.requestName = 'tasknumber<index>'
import os
index = 0
base_request_name = 'tasknumber'
while os.path.isdir("%s/crab_%s%s" % (config.General.workArea, base_request_name, index)):
    index += 1
## --- Auxiliary code ends here --- ##
config.General.requestName = base_request_name + str(index)

Slow job release for Hammer Cloud tests (useless for regular users)

Slow job release is a feature intended for lowering the overhead of HC tests. Instead of creating multiple tasks to test the availability of a site, a single task can be submitted with an extraJDL parameter specifying the timeout between job releases. It utilises the HTCondor PRE script DEFER option. Each PRE script will check if the release timeout has been specified, and if it is too soon to be released, it will exit to be retried after timeout * jobId amount of time.

If the timeout is set to be 2 minutes and the task has 3 jobs, the first job will be released after 2 minutes, the second one after 4 minutes and the third one after 6 minutes.

The timeout in seconds can be specified in the crab configuration file like this:

config.Debug.extraJDL = ['+CRAB_JobReleaseTimeout=120']

-- AndresTanasijczuk - 2015-01-08

Edit | Attach | Watch | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r9 - 2017-07-04 - StefanoBelforte
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback