CRAB Logo

CRAB Client: Use, Deployment, Maintenance

Complete: 5 Go to SWGuideCrab

CRAB Client is now deployed by CMSSW automatic release build. A new stable release is made any time the CMSSW master release is updated (typically every 3-4 weeks) and includes whatever version of the client is indicated in the spec files for prod, pre and dev versions, which are independently controlled (by us).

The cmsbuild robot build updates those three crab versions every night in a "parallel universe", where things can be tested, but be aware that those are the same spec files as the actual release build, so be very careful before changing crab-prod.spec.

More on the proper procedure to care for the spec files is below

Using CRABClient

Using CRABClient last release in CMSSW distribution

1. get a proxy
voms-proxy-init -rfc -voms cms -valid 192:00
export X509_USER_PROXY=`voms-proxy-info -path`

2. setup CMSSW

cd CMSSW..
cmsenv

  • note: if you do no thave a CMSSW release at hand, just create any recent one, and it will do:
cmsrel CMSSW_10_4_0
cd CMSSW_10_4_0
cmsenv

3. point to your favourite version executing ONE of these

crab         # which is an alias for  crab-prod
crab-prod  
crab-pre
crab-dev

Usually users should stick to crab i.e. crab-prod unless they want to test, or need to use, some new feature or bug fix only available in pre or dev.

Using CRABClient from nightly CMSSW Integration Build

do this and then proceed as above:
export PATH=/cvmfs/cms-ib.cern.ch/latest/common:$PATH 

Using CRABClient API

In order to use the API, besides setting up cmsenv as above, you need to source this script.
source /cvmfs/cms.cern.ch/common/crab-setup.sh
and make sure that your python script has this before any other import statement:
import CRABClient
That script can be safely sourced at all times, even if you do not plan to use the APO.

By default that will point you to the prod instance of the CRABClient. Otherwise the instance can be indicated as argument:

source /cvmfs/cms.cern.ch/common/crab-setup.sh prod
source /cvmfs/cms.cern.ch/common/crab-setup.sh pre
source /cvmfs/cms.cern.ch/common/crab-setup.sh dev
Or simply
source /cvmfs/cms.cern.ch/common/crab-setup.sh -h

If you are using the nightly IB, you should use instead

source /cvmfs/cms-ib.cern.ch/latest/common/crab-setup.sh 

Examples

the following, usually working, scripts give concrete examples of how to setup things, I usuall execute one of this via a short alias every time I login for CRAB action. My alias also grabs a voms proxy.

/afs/cern.ch/user/b/belforte/public/CRAB/SETUP
/afs/cern.ch/user/b/belforte/public/CRAB/SETUP-IB
/afs/cern.ch/user/b/belforte/public/CRAB/SETUP-LOCAL

Using DBS Python API

DBS Python API is deployed by CMSSW as a CRABClient dependency. Therefore in order to use it in a standalone script, rather than inside CRABClient, you need to setup CMSSW via cmsenv command and execute the crab-setup.sh as indicated above. After that you can use the usual thing in your python scripts:
from dbs.apis.dbsClient import DbsApi

Develop for CRABClient

Using CRABClient directly from your GitHub repository

All binary dependency for CRABClient are now in CMSSW, therefore you only need to point to the python code and its python dependencies:

to do once:

1. go to your favorite development place

cd <some_dir>
GitDir=`pwd`

2. clone repositories and make sure to checkout proper versions (see example beflow for WMCore and DBS). Master HEAD may not always work, but if you want to develop code yoy may have to start there. In addition to CRABClient you will need DBS, WMCore and a couple files from CRABServer. If you want to eventually make git Pull Requests, you will fork first and clone from your fork the repository you want to work on. If that's the case you surely know how to modify following commands. If you simplt want to look around, debug things, or just use latest code, copy/pasting the following should do. But beware that WMCore versions change frequently. You can check crab-*.spec files in cmsdist to see which WMCore version is used with current CRABClient.

git clone https://github.com/dmwm/CRABClient
git clone https://github.com/dmwm/CRABServer
cp CRABServer/src/python/ServerUtilities.py CRABClient/src/python/
cp CRABServer/src/python/RESTInteractions.py CRABClient/src/python/
git clone https://github.com/dmwm/WMCore
cd WMCore; git checkout 1.3.3; cd ..
git clone https://github.com/dmwm/DBS
cd DBS; git checkout 3.10.0; cd ..

to do at each login

1. get a proxy
voms-proxy-init -rfc -voms cms -valid 192:00
export X509_USER_PROXY=`voms-proxy-info -path`

2. setup CMSSW

cd CMSSW..
cmsenv

3. execute these lines

GitDir=<the_directory_where_you_cloned>
MY_DBS=${GitDir}/DBS
MY_CRAB=${GitDir}/CRABClient
MY_WMCORE=${GitDir}/WMCore

export PYTHONPATH=${MY_DBS}/Client/src/python:${MY_DBS}/PycurlClient/src/python:$PYTHONPATH
export PYTHONPATH=${MY_WMCORE}/src/python:$PYTHONPATH
export PYTHONPATH=${MY_CRAB}/src/python:$PYTHONPATH

export PATH=${MY_CRAB}/bin:$PATH
source ${MY_CRAB}/etc/crab-bash-completion.sh

you can see a complete, usually working, example in /afs/cern.ch/user/b/belforte/public/CRAB/SETUP-LOCAL

Managing Versions and dependencies via CMSDIST

Dependencies

CRAB Client needs WMCore, needs a couple files from CRABServer (Client and Server need to stay aligned on some defintions and settings) and carries a dependency from DBS for easy distribution of DBS Python API. The way to manage such dependencies (add ? remove ?) is desribed in https://twiki.cern.ch/twiki/bin/view/CMSPublic/NotesAboutReleaseManagement#Cross_service_dependencies

Where and What

CRAB Client spec files are now in

There are only 3 files which you can/should modify. You must change in your private fork and make a Pull Request At time of writing this twiki those are:

The all have a content like this:

### RPM cms crab-prod 3.3.2001
%define wmcore_version     1.2.8
%define crabserver_version 3.3.2001.rc1
%define dbs_version        3.10.0

Numbers there refer to tags in github. Usually you wll simply want to change the CRAB Client release number in the first line. Example:

from:
### RPM cms crab-prod 3.3.2001
to:
### RPM cms crab-prod  3.3.2002.rc3

  • If needed, we can also modify the following two lines for the WMCore and CRABServer versions respectively:
%define wmcore_version 1.2.9
...
%define crabserver_version 3.3.2002.rc1

It is very rare that DBS Client version needs to change, but it is always better stay with a named version than with repo HEAD

When and How

  • the ideal procedure is
    1. develop and test locally as indicated above
    2. when confident, make a new release in github and update the crab-dev spec
    3. the day after can start checking that build went well and all works in IB
    4. when confident, port the change to crab-pre.spec
    5. it there's no urgency, wait for this to be deployed on "lxplus" for more extensive checking, e.g. involving users
    6. finally update crab-prod-spec
      • at same time it may be a good idea to set crab-dev.spec to same old version as crab-prod was, so if there is a disaster users can roll back by replacing crab with crab-dev command

Tricky technical points

All CRAB Clients instances use a common build file https://github.com/cms-sw/cmsdist/blob/IB/CMSSW_11_1_X/master/crab-build.file

If it is changed via a Pull Request, it will affect all instances (prod, pre, dev) at same time. The way to test changes is to make proper chanced in the crab-dev.spec only. As per this hint from Shahzad:

 first test this change in crab-dev only by adding the following line in crab-dev.spec

%define crabserver_packages RESTInteractions.py ServerUtilities.py

with the above change you do not need to update crab-build.file
see https://github.com/cms-sw/cmsdist/pull/5561#issuecomment-587293336 and https://github.com/cms-sw/cmsdist/pull/5561#issuecomment-587483867

Refs.

Some discussions about client deployment:

Old ways... deprecated, obsoleted, broken....

If you only want to develop some code and do not care about installing the whole client then you should consider sourcing the production client (source /cvmfs/cms.cern.ch/crab3/crab.sh), cloning the client from the GitHub repository (https://github.com/dmwm/CRABClient) and then:

export CLIENT_HOME=/your-local-github-repository/CRABClient
export WMCORE_REPO=/your-local-github-repository/
export PYTHONPATH=$CLIENT_HOME/src/python/:~$WMCORE_REPO/WMCore/src/python/:$PYTHONPATH
export PATH=$CLIENT_HOME/bin:$PATH

You may also test not only your own code like this but also the PRs from other people: For how to checkout locally the PRs from the upstream repository see here: https://gist.github.com/piscisaureus/3342247

$ vi .git/config

[remote "upstream"]
   url = https://github.com/dmwm/CRABClient.git
   fetch = +refs/heads/*:refs/remotes/upstream/*
   fetch = +refs/pull/*/head:refs/remotes/origin/pr/*

$ git fetch upstream
$ git checkout pr/999
When you finish testing the PR in question go back to the original branch

$ git checkout master

Deployment using RPM

The CRAB3 client can be installed via RPM using the following instructions:

Please check http://cmsrep.cern.ch/cmssw/comp/RPMS/slc6_amd64_gcc493/ and select newest or needed version for you. If you are developing, you should be taking always the newest version.

Follow the instructions at point 4) in NotesAboutReleaseManagement#Releasing_a_CRABClient_release_c

Deployment of CRAB client on CVMFS (production and pre-production)

The procedure is very simple thanks to the automated Bockjoo's build/install scripts. This script constantly checks for new rpm of the client, and then installs them on cvmfs. This is how you can trigger the automatic creation of a RPM:

  • Modify the crabclient.spec file to contain the needed (newest) version. The .spec file to use is the one in the comp_gcc630 branch (used to be comp_gcc493 until September 2019) of the cms-sw/cmsdist github repo. Lets say we're ready to deploy in production the April CRABClient (3.3.1604): we change the first line of the .spec file containing the crabclient version to 3.3.1604, which can be accomplished also using the GitHub edit function in the browser. Otherwise, use the listed git commands to push the changes to your fork of cmsdist and them make a PR.
git clone <your-cmsdist-fork> && cd cmsdist && git checkout comp_gcc493
vim crabclient.spec
git add crabclient.spec
git commit -m "Message about commit" 
git push origin

Changes to the crabclient.spec file:

-### RPM cms crabclient 3.3.1604.rc2
+### RPM cms crabclient 3.3.1604

  • If needed, we can also modify the following two lines for the WMCore and CRABServer versions respectively:
%define wmcver 1.0.14_crab_4
...
%define crabserver 3.3.1604.rc1
    • If the specified CRABClient version is of the 3.3.1604 or 3.3.1604.patch## format, it means this is a production version of the client, which will be deployed after the merge and point the crab.sh and crab.csh to the 3.3.1604 or 3.3.1604.patch## client.
    • If the specified CRABClient version ends with a release candidate ("=rc=") extension, 3.3.1604.rc##, it means this is a pre-production version, which will be deployed after the merge and point the crab_pre.sh and crab_pre.csh to the 3.3.1604.rc## client.

  • Wait for the automatic build to run and the corresponding comment to appear on the PR. If no error occurred, the message will show a "+1", the architecture version and the link to the build log.

  • Add a comment with the "merge" text to merge the PR.

Edit | Attach | Watch | Print version | History: r36 < r35 < r34 < r33 < r32 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r36 - 2020-05-22 - StefanoBelforte
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback