Notes about CRAB3 build and release management

Instructions below are mostly tailored for patching a given release once master branch in GIT has progressed ahead. For the more normal case of creating a new release from current master branch, the tag can be simply created in github, see:

Staying aligned with WMCore

Cross service dependencies

Releasing a CRABClient release candidate / patch


Building a new release rpm's (same instructions for CRAB TaskWorker, CRAB Server and CRAB Cache)

the procedure indicate below should be used also to make new releases for crabserver and crabcache so that they will be built and updated for deploying in CMSWEB.

Important Whenever you want deploy services that are part of cmsweb (in our case crabserver and crabcache), you should update .spec files in GIthub cmsdist repo ( and make a comment in 'HGXXX Release Request' gitlab issue (e.g. Every month CMSWEB operators shares a link to the new gitlab issue) to request these changes to be deployed. Once you make a comment, your pull request will be merged by CMSWEB operator (in other words, do not merge it yourself). It is important to know, that once PR is merged, those RPM's will be built and deployed within the next CMSWEB release.

Notes about the .spec files

See these from Shahzad

So the pkgtools machinery processes our .spec files removing the # something comments and expanding the ## RPM and similar metacomments to prepare a new spec file located in the build work directory. e.g.: /build/belforte/w/SPECS/cms/crabtaskworker/3.200531.01/spec

This file is then fed to rpm building tools (see e.g. )

And in the that file you can see how each %{...} variable in the spec file is expanded to. E.g.

belforte@vocms055/belforte> cat w/SPECS/cms/crabtaskworker/3.200531.01/spec |cut -c 1-80|grep define|head
%define pkgname        crabtaskworker
%define pkgversion     3.200531.01
%define pkgcategory    cms
%define cmsroot        /build/belforte/w
%define instroot       /build/belforte/w/tmp/BUILDROOT/4d94cea5841f13ff179ab4ab3
%define realversion    3.200531.01
%define gccver         6.3.0
%define compilerRealVersion 6.3.0
%define pkgrevision    1
%define pkgreqs        external/gcc/6.3.0 external/zlib/1.2.8 external/openssl/1

note also the two shortcuts used in our .spec files :

%{n} is the same as %{pkgname}
${v} is the same as %{pkgversion}

Important note about variables and RPM version numbers

  • %{realversion} is always set to the version string defined in ### RPM line. %{v} or %{pkgversion} is the actual version ( or directory where it will install your package) which is generated by cmsBuild. It will be set to %{realversion} for first build. But for any rebuild (e.g. you changed the spec without updating the realversion or if one of the dependencies were changed) %v will be set to %{realversion}-somestring (in your case somestring will be comp[0-9]* ). %{n} or %{pkgname} is always set to name of the package defined in ### RPM line.

Releasing a CRAB TaskWorker release candidate / patch

*1) Create a new release in GitHub

*2) If the new code requires changes to WMCore, make a Pull Request for this and coordinate with WMCore developers in order to have a WMCore release with the changes in. You can not create a WMCore release yourself.

3) Modifying the specfile and building a package

Quick tutorial about building RPMs with cmsdist scripts

Building and uploading a package:

For the *slc6 distribution use vocms022 as building machine, gcc493 as <gcc-version> and slc6_amd64_gcc493 as <architecture>.

  • For the slc7 distribution use vocms055 as building machine, gcc630 as <gcc-version> and slc7_amd64_gcc630 as <architecture>.

# prepare a build area  (only do once):
mkdir -p /build/$USER
cd /build/$USER
(git clone -b V00-32-XX
(git clone && cd cmsdist && git checkout comp_<gcc-version>)
(cd ..)

# no need to touch pkgtools unless someone of the maintainers of that tells you to
# but keep your cmsdisk local repostory up to date with upstream one in github

# Do needed changes to the specfile
# usually we only modify the first line "### RPM cms crabserver 3.3.1603.rc1" to e.g. have the version 3.3.2002
# if a new WMCore version is also neede, chane the line : %define wmcver 1.2.9
vim cmsdist/crabtaskworker.spec

You can now use git to commit and make a PR in the `cms-sw` repository, this wil trigger an automatic check, once your PR receive the "+1" comment from the DMWMbot you can merge it by adding a "merge" comment if you have the authorization, or ask CMSWEB operator(s) or cmssw people to do so, via a proper comment. List of authorized people is maintained in . People can be added/removed by making a PullRequest which CMSSW people (i.e. Shahzad) will approve (or not).

An automatic script will build the package that will be available in ~1h at the RPM repository listed below

You can also build the package yourself and upload it to$USER: However, before uploading it, make sure that you are subscribed to cms-comp-devs e group which will give you required permissions to do that.

# building the new package (works only in the machine mentioned above)
pkgtools/cmsBuild -c cmsdist --repository comp -a <architecture> --builders 8 -j 5 --work-dir w build crabtaskworker | tee logBuild

# uploading it to comp.$USER (works only in the machine mentioned above)
pkgtools/cmsBuild -c cmsdist --repository comp -a <architecture> --builders 8 -j 5 --work-dir w upload crabtaskworker --upload-user=$USER | tee logUpload

The user repository in comp.$USER only holds one release, latest one will override previous versions. If you want to upload more packages (e.g. crabtaskworker and crabserver) you need to indicate both in the same upload command like

pkgtools/cmsBuild -c cmsdist --repository comp -a <architecture> --builders 8 -j 5 --work-dir w upload crabserver crabtaskworker --upload-user=$USER
In case of strange errors that can't be explained by logs, try to reset your environment by deleting the work directory (/build/$USER/w/). Also make sure that you're checked out to the correct commit/branch of the git repositories!

4) Deploying and testing the TaskWorker on your VM

Follow the regular deployment instructions on the twiki:

5) After testing the TaskWorker on your VM

If all looks good, install the new TaskWorker release on a test machine, configure it to access preprod instance of CRAB database andmake sure that it is thourougly tested before releasing it into production. Afterwards, make a pull request to the cmsdist repository with the new crabtaskworker.spec file.

Notes on RPM repositories

Automatic Docker image build for CRABServer repository

Every time new Release is created in CRABServer repository, Jenkins jobs which builds RPMs and docker image for crabcache, crabserver, crabtaskworker is triggered: . Built RPMs are stored in comp.crab3 repository every time overwriting old RPMs files.

Since crabtaskworker deployment is done by us, docker image for crabtaskworker is stored right away in cmssw/crabtaskworker DockerHub repository and is ready for deployment to pre-production. On the other hand, crabcache, crabserver are are stored in ddaina/crabcache and ddaina/crabserver repository and are ready to be deployed to Kubernetes test2 cluster for our validation and testing.

In case there are some infrastructure problems and new release didn't trigger Jenkins jobs, that can be done manually going to the Jenkins job page and pressing Build with parameters button. In pop-up window enter RELEASE_TAG which is tag from CRABServer Github repository and WMCORE_TAG which is WMCore tag that should be used to deploy service.

Notes about cross service dependencies

WMCore tag is maintained in the Jenkins job configuration, i.e. every time we want to use new tag we have to update an environment variable WMCORE_TAG in

Edit | Attach | Watch | Print version | History: r50 < r49 < r48 < r47 < r46 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r50 - 2021-03-25 - StefanoBelforte
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback