Deployment of CRAB REST Interface

Complete: 5 Go to SWGuideCrab


This twiki explains how to deploy the CRAB server frontend (a.k.a. the CRAB REST Interface). It will guide you through the steps required to:

  1. Ask for an Oracle account and install the CRAB database schema from scratch.
  2. Get a virtual machine (VM) with the right architecture from the CERN OpenStack Cloud Infrastructure.
  3. Install the required software on the machine.
  4. Configure the machine.

Note: Legend of colors for the examples:

Commands to execute
Output sample of the executed commands
Configuration files
Other files

Note: We will sometimes use the short cut "REST Interface" or just "REST" to refer to "CRAB REST Interface".

Get a CERN Oracle database account

Go to and subscribe to the "oracle-general-purpose-users" e-group.

Note: You will need to wait for approval before you are actually added to the e-group. You will get an e-mail when that happens. After you have subscribed, if you click on "oracle-general-purpose-users" from the subscription page above, you will see a button with the label "Remove me from members". But this doesnít mean that you have been approved. To see your status, you can click again on "oracle-general-purpose-users" and go to the "Members" tab. You will only be able to see your name, and until you have been approved in the far-right column it will say "Waiting for approval".

Once you have been approved, go to and fill out a request to get a CERN Oracle database account:

Login: crab3_<username>
Database: devdb11 
Description: For private CRAB3 server instance.
Beware we will switch to devdb19 in early 2021

If you want to be able to see/edit your CERN Oracle database, install Oracle SQL developer (requires to create an Oracle account). Download this file and put it in /etc/ (at least for linux, for other OSes the locations in which SQL Developer looks for the tnsnames.ora file may be different). Better yet, install the CERN's oracle-instantclient rpm which will keep /eetc/tnsnames.ora always up to date, see below CMSCrabRESTInterface#Install_tnsnames_ora

When creating the connection to the CERN Oracle DB in SQL developer, input an arbitrary connection name, username (same as the "Login" you chose for the account, something like crab3_username) and your chosen password. In the "Connection type" dropdown, select "TNS". Finally, in the "Network Alias" dropdown, select the database instance (devdb11 in our case). Connection should be now set up, so click the "Test" button to see if it works, and the "Save" button to store the connection details for later use. Beware we will switch to devdb19 in early 2021

Get and install a virtual machine

The CRAB REST Interface uses the code and the infrastructure provided by the HTTP group. The production and pre-production CRAB REST Interfaces are deployed on the CMSWEB cluster. The official tutorial of the HTTP group for installing your own version of the CRAB REST Interface on a VM can be found in, in particular section "1.1.1. CERN virtual machine". That's the tutorial we suggest to follow, up to point "6. Set up authentication" inclusively, to prepare your VM. The only suggested change is to use the following parameters when requesting the VM:

Availability Zone: Any Availability Zone
Instance Name: <any name you want>
Flavor: m1.large
Instance Count: 1
Instance Boot Source: Boot from image
Image Name: SLC6 CERN Server - x86_64 (the latest)

Note: For the interested reader, there is a CERN OpenStack Private Cloud Guide at

Machine preparation

If, as suggested in section Get and install a virtual machine, you followed the HTTP group tutorial up to point "6. Set up authentication" inclusively (the recommended way!), then you don't need to perform the following steps and you can go directly to section REST Interface installation and configuration. But check first Install voms clients.

1) Install git, clone the dmwm/deployment repository and use the Deploy script for preparing the VM.

Note from the HTTP group tutorial: The Deploy command below will also check if it is possible to resize the current root partition on the fly, asking for your confirmation for doing so or skip. This is recommended, because the default SLC5/6 installations do not size partitions to use the entire allocated disk. It is also useful if you later resize the VM to a different flavor (i.e. with more disk space). Note, however, the partition resizing procedure is not very reliable, so do never, ever, run it on a production server VM, nor expect it to work for VM images other than the SLC5/6 ones.

sudo yum -y install git.x86_64
mkdir -p /tmp/foo
cd /tmp/foo
git clone git:// cfg
sudo -l
cfg/Deploy -t dummy -s post $PWD system/devvm

INFO: can resize '/' to increase in 75162 MB. Proceed? (y/n)


The Deploy script will request a Grid host certificate for your VM and put it in the right directory with the right permissions. This is done in an ad-hoc way (the script does exactly the same as what you would do manually; there is no API). If the Deploy script fails when running careq-cern-ch, then you can request the Grid host certificate manually as described below and then re-run cfg/Deploy -t dummy -s post $PWD system/devvm.

Requesting a Grid host certificate for your VM

a) Using a browser that has your Grid user certificate imported on it, go to

  • Click on "New Grid Host Certificate".
  • Click on "Request certificate using OpenSSL (recommended for Linux machines)".
  • Choose the VM host name from the pull-down menu under "Certificate Subject".
  • Click "Select".

b) Now you must have got the instructions on how to prepare the Grid host certificate request.

  • Log in to your VM.
  • Make sure you have the ~/.globus directory with your valid Grid user certificate (files usercert.pem and userkey.pem).
  • If your Grid user certificate was issued by a certification authority other than CERN CA, associate your CERN primary account to the certificate. You can do that in
  • Source the UI (source /afs/ if not done by default.
  • Run the following command (for example in the private directory of your home area):
cd ~/private
openssl req -new -subj "/CN=`hostname -f`" -out newcsr.csr -nodes -sha512 -newkey rsa:2048
  • Two files should have been created, privkey.pem and newcsr.csr. The first contains the private key and the second the certificate request that you should send to CERN CA. Open the file newcsr.csr, copy all its content and paste it in the webpage in the field "Certificate Request".
  • Click "Submit".

c) Download the certificate by clicking on "Base 64 encoded" under "Download Certificate". (The default name of the downloaded file is host.cert.)

d) Copy the host.cert file to your VM (to the same directory where you have the privkey.pem file, for example the private directory of your home area).

scp host.cert <username>

e) Optional: Create a certificate in pkcs12 format.

In your VM run the following command:

openssl pkcs12 -export -inkey privkey.pem -in host.cert -out myCertificate.p12

f) Move the certificate and private key to the directory /etc/grid-security/, change the ownership of the files to root:root and protect the private key so that only root can read it. Remove the file newcsr.csr.

cd ~/private/
sudo mkdir /etc/grid-security
sudo mv host.cert /etc/grid-security/hostcert.pem
sudo mv privkey.pem /etc/grid-security/hostkey.pem
sudo chmod 400 /etc/grid-security/hostkey.pem
sudo chown root:root /etc/grid-security/host{cert,key}.pem
rm newcsr.csr


less /tmp/foo/.deploy/* # if you want to check what happened
cd ~
rm -rf /tmp/foo

2) Request proxy renewal rights for your VM.

Send an e-mail to with Cc to cms-service-webtools(AT)

E-mail subject:

myproxy registration request for <hostname>

E-mail body:

Could you please add the following host certificate to trusted retrievers, authorized retrievers, authorized renewers policy?
This is a development server for CM web services and requires use of grid proxy certificates.


<Your Name>

where <hostname> is the full host name as shown by the command hostname -f. Run openssl x509 -in /etc/grid-security/hostcert.pem -noout -subject and verify that the host certificate is correct in the message body.

3) Log out.

Note from the HTTP group tutorial: It is also recommended to reboot the server, specially if you have resized partitions on the fly, but it isnít really necessary. If you use SSH persistent connections, be sure to terminate the master connection before logging in again. The commands in step 1) modify users and groups, and itís important they apply when you execute the next steps. This requires completely new log-in, and using persistent SSH connections may cause you to use cached credentials.

4) Log in.

SSH into your VM again and run the following to double check that you got the multiple service accounts:

id # should print out large number local _foo groups now

5) Make sure that you have both 'voms' and 'voms-clients' packages installed:

rpm -qa |grep voms  

Install voms clients

If voms-proxy-init is not available in the VM, install the voms clients. Check which version should be installed by running yum provides on lxplus (in the example below it tells to install voms-clients-2.0.12-1.el6.x86_64).

sudo yum provides */voms-proxy-init

As of July 2019, it is best to use the C++ version, not the Java one, i.e. voms-clients-2 not voms-clients-3

sudo yum install -y voms-clients-cpp

You also need the VOMS configuration files for CMS. Via

sudo yum install -y wlcg-voms-cms

As last resort if that fail you may copy the directories /etc/vomses and /etc/grid-security/vomsdir/cms from lxplus.

sudo scp -r <username> /etc/vomses
sudo mkdir /etc/grid-security/vomsdir
sudo scp -r <username> /etc/grid-security/vomsdir/cms

Install tnsnames.ora

This file is needed for Oracle client to know how to find the DB. It used to be part of CMSW but now we use instead a CERN rpm which keeps it up to date in /etc/tnsnames.ora

this is done via

sudo yum install oracle-instantclient-tnsnames.ora --enablerepo={cernonly,cernonly-testing}


Disable default httpd

Sometimes the operating system installtion ends up in starting a default httpd daemon at boot time which will create obscure APACHE errors when you try to start the frontend later, as it runs its own httpd server from RPM's built in COMP repository. The system httpd can be stopped and disabled by:
sudo systemctl stop httpd
sudo systemctl disable httpd
You may want to check that it is not running after any reboot, together with checking IP tables

REST Interface installation and configuration


The following instructions are basically point "7. Software installation" of the HTTP group tutorial, where we updated the versions of the software being installed (the latest HG tags can be found in

Get the configuration and deploy the frontend, crabserver and crabcache services (the following command will ask you to type your Grid user certificate passphrase twice):

for slc6:

(cd /data; git clone git:// cfg && cd cfg && git reset --hard $HGVER)
(REPO="-r comp=comp" A=/data/cfg/admin;
 cd /data;
 $A/InstallDev -R comp@$HGVER -A slc6_amd64_gcc493 -s image -v $HGVER $REPO -p "admin/devtools frontend crabserver crabcache")

for cc7:

(cd /data; git clone git:// cfg && cd cfg && git reset --hard $HGVER)
(REPO="-r comp=comp" A=/data/cfg/admin;
 cd /data;
 $A/InstallDev -R comp@$HGVER -A slc7_amd64_gcc630 -s image -v $HGVER $REPO -p "admin/devtools frontend crabserver crabcache")

Note: crabserver is the CRAB REST Interface, crabcache is the CRAB User File Cache and frontend is the service receiving the http requests and redirecting them to the corresponding (backend) service (it runs an HTTP daemon).

Installing a custom build

If you want to install a custom build of one service (see NotesAboutReleaseManagement ) instead of the build prepared by HTTP group and inserted in HGxxxx version, the following instructions from Valentin are the way to go. See

it can be done using private comp repository. Basically you create a full set of
RPMs and upload them to your repository, I do that all the time.

Then you can change your /data/cfg/das/deploy (I use das as an example)
and change its deploy_das_sw step as following where I specify concrete
tag from I want to deploy (and it should be present in your comp repository):

deploy_pkg comp.valya cms+das 04.01.09-comp

Doing this way you can modify ANY package you deploy, for those where you don't
specify repository and tag it will be taken from the HGxxxxx tag, while where
you specify it explicitly it will be taken from your repository.

and also these more suggestions from Alan, see

in addition to that. In case you're building comp instead of only your service RPM, something like:
pkgtools/cmsBuild -c cmsdist --repository comp \
   -a slc7_amd64_gcc630 --builders 8 -j 4 --work-dir w \
   --upload-user=$USER upload comp

you can then deploy the new comp tag (and all other service RPMs built with that, I assume you
will have a new crabserver version here) with  a command line like:
(VER=HG1806m-comp1 REPO="-r comp=comp.amaltaro" A=/data/cfg/admin; ARCH=slc7_amd64_gcc630;
 cd /data;
 $A/InstallDev -R comp@$VER -A $ARCH -s image -v $VER -a $PWD/auth $REPO -p "admin frontend couchdb reqmgr2 workqueue reqmon acdcserver")

you just need to check the new comp RPM name (see VER above) and replace my repo
by yours (REPO), well, and of course the services you'd like to deploy.

You might have to manually update your service config, since we usually disable services
in nodes "unknown".


External (dynamic) configuration

Prepare the external configuration file for the REST Interface.

The external REST configuration file should be in JSON format. It may contain more than one configuration. Each configuration is a dictionary identified by an (arbitrary) key.

Configuration example for the production REST Interface in CMSWEB (maintained in

    "cmsweb-prod": {
        "delegate-dn": [
        "backend-urls" : {
            "cacheSSL" : "",
            "baseURL" : "",
            "htcondorSchedds" : {
                "" : {
                    "proxiedurl": ""
                "" : {
                    "proxiedurl": ""
                "" : {
                    "proxiedurl": ""
                "" : {
                    "proxiedurl": ""
            "htcondorPool" : ",",
            "asoConfig" : [
                {"couchURL" : "" , "couchDBName" : "filetransfers"}
        "compatible-version" : ["3.3.14", "3.3.15", "3.3.16"],
        "banned-out-destinations" : ["T1_*"]

Configuration example for the pre-production REST Interface in CMSWEB-testbed (maintained in

    "cmsweb-preprod": {
        "delegate-dn": [
        "backend-urls" : {
            "cacheSSL" : "",
            "baseURL" : "",
            "htcondorSchedds" : {
                "" : {
                    "proxiedurl" : ""
            "htcondorPool" : ",",
            "asoConfig" : [
                {"couchURL" : "" , "couchDBName" : "filetransfers"}
        "compatible-version" : ["3.3.8", "3.3.9.rc2", "3.3.9", "3.3.10"],
        "banned-out-destinations" : ["T1_*"]

Configuration example for a private REST Interface (maintained in

    "cmsweb-dev": {
        "delegate-dn": [
        "backend-urls" : {
            "cacheSSL" : "",
            "baseURL" : "",
            "htcondorSchedds" : {
                "" : {
                    "proxiedurl" : ""
            "htcondorPool" : ",",
            "asoConfig" : [
                {"couchURL" : "" , "couchDBName" : "filetransfers"}
        "compatible-version" : ["3.3.[0-9a-zA-Z\.]+"],
        "banned-out-destinations" : ["T1_*"]

Here is an explanation of the meaning of each of these parameters:

delegate-dn List of DNs that are allowed to retrieve the user's proxy from myproxy server. CRAB services use Grid host certificates to contact myproxy server. So delegate-dn should include the Grid host certificate DNs of the hosts where the corresponding instances (production, pre-production or private) of the TaskWorker and AsyncStageOut are running ("corresponding" instances means the TaskWorker instance that will work on the tasks submitted to this REST, and the AsyncStageOut instance(s) that these tasks will use).
cacheSSL, baseURL URL of the crabcache instance to be used by this REST. If you want to use your private crabcache, change these parameters to point to your machine.
htcondorSchedds Dictionary of schedds on the pool. The keys of the dictionary are the schedd names as shown by condor_status -schedd. The values for the schedds are a dictionaries that contains the configuration of that schedd. For each schedd it is possible to specify a parameter called proxiedurl that indicates the URL used for the dashboard logfiles and for the crab status --short command
htcondorPool As from Nov 2014, everything is moved to the global pool and this parameter must be, crabserver will split by ','.
asoConfig REST URL and DB name for the (default) DB instance to be used by the CRAB server for ASO-related businesses, "filetransfers" DB means Oracle (the user can overwrite this config in the task CRAB configuration).
compatible-version List of CRAB client versions that are compatible with this REST Interface instance. Can use python regular expressions. (If the client is not compatible, a warning message will be printed on every crab command. To avoid the warning message in private tests, you may simply allow any version.)
banned-out-destinations List of sites where stage out is forbidden (e.g. it is not allowed to stage out to any T1).

Note: Under any doubt about what value should you assign to a parameter, please contact the CRAB3 team.

You can put this configuration file on gist ( This is what all CRAB3 developers do and is assumed you will do as well.

Internal (static) configuration

The internal REST configuration file is /data/srv/current/config/crabserver/

The internal REST configuration contains a link to the external REST configuration file. So edit the internal REST configuration file and add the link to your external REST configuration file (specify also the key of the configuration that should be used).

sudo vi /data/srv/current/config/crabserver/

Example for the external REST configuration file in used by the production and pre-production REST Interfaces:

data.extconfigurl = ';a=blob_plain;f=cmsweb-rest-config.json'
data.mode = 'cmsweb-prod' # or 'cmsweb-preprod'

Example for an external REST configuration file corresponding to a private REST Interface:

data.extconfigurl = ''
data.mode = 'cmsweb-dev' # or whatever is the key of your configuration

WARNING: The above link contains a revision number (the field between raw and the file name). This implies that every time the file in gist is changed, one needs to get the new link and change the URL in the static REST configuration file. To avoid this, one can use a link that points always to the latest version of the file on gist by removing the revision number:

data.extconfigurl = ''
data.mode = 'cmsweb-dev' # or whatever is the key of your configuration


Authentication with CERN Oracle database

The REST authentication file for accessing the Oracle database is /data/srv/current/auth/crabserver/

For private installations, modify the file to have the required information (don`t forget to change the oracle credentials (*********)).

sudo vi /data/srv/current/auth/crabserver/

import cx_Oracle as DB
import socket
fqdn = socket.getfqdn().lower()
dbconfig = {'dev': {'.title': 'Pre-production',
                    '.order': 1,
                    '*': {'clientid': 'cmsweb-dev@%s' % (fqdn),
                          'dsn': 'devdb11',    # or devdb19
                          'liveness': 'select sysdate from dual',
                          'password': '***************',
                          'schema': '**********',
                          'timeout': 300,
                          'trace': True,
                          'type': DB,
                          'user': '*********'}

Parameters explanation:

title, order Any string will do.
clientid, dsn, liveness, timeout, trace, type Must be like indicated.
user Your CERN Oracle account username.
password Your CERN Oracle account password.
schema Put your CERN Oracle account username here as well.

Make sure the file is owned by _sw _config:

ls -l /data/srv/current/auth/crabserver/

-r--r-----. 1 _sw _config 622 Jun 15 09:30 /data/srv/current/auth/crabserver/

If not, do

sudo chown _sw:_config /data/srv/current/auth/crabserver/

You will have to update the file every year with the new oracle password.

Authentication with frontend

ONLY for PRIVATE installations.

1) Check that your user DN is present in /data/srv/state/frontend/etc/authmap.json. If you want other users to be able to access your REST Interface, add their DNs to this file.

2) As part of the frontend installation, a cron job is created (which runs every 4 hours) that executes /data/srv/current/config/frontend/mkauthmap -q -c sitedbread.db -o /data/srv/state/frontend/etc/authmap.json. This cron job updates the /data/srv/state/frontend/etc/authmap.json file with the DNs from all users as obtained from querying the sitedb service. If you are not running the sitedb service in your VM, the cron job will not fail, but update the /data/srv/state/frontend/etc/authmap.json file with only your user DN. On the other hand, if you are running the sitedb service in your VM, the updated file will contain the DNs from all users. Thus, for private installations, we recommend to comment out this "automatic role setting" in the crontab:

crontab -e

#*/4 * * * * . /data/srv/current/apps/frontend/etc/profile.d/ && PYTHONPATH=/data/srv/current/auth/frontend:$PYTHONPATH /data/srv/current/config/frontend/mkauthmap -q -c sitedbread.db -o /data/srv/state/frontend/etc/authmap.json

NOTE: for both PRIVATE and PROD|PREPRD|DEV installations, the certificate of the TaskWorker needs to be granted the role of (CRAB3) Operator and to be added as a member of the group crab3.

  • For PRIVATE: manually in the authmap.json file - example:
{"DN": "/DC=ch/DC=cern/OU=computers/CN=tw/", "ID": "361", "LOGIN": "service@vocms0118", "NAME": "service@vocms0118", "PASSWD": "1234567890", "ROLES": {"operator": ["group:crab3", "site:t3-nx-foobar"]}},
  • For PROD|PREPRD|DEV: Just need to be sure that the proper attributes have been set to the entry: vocms0118. And in case it is not set properly contact the cmsweb Operator to fix the issue.

Certificate for interactions with CMSWEB

Access to CMSWEB is restricted to CMS users and services by requesting authentication with certificates registered in VO CMS and SiteDB. The static REST configuration file has two parameters to point to a certificate and private key that the REST Interface should use for interactions with CMSWEB:

data.serverhostcert = "%s/auth/crabserver/dmwm-service-cert.pem" % __file__.rsplit('/', 3)[0]
data.serverhostkey = "%s/auth/crabserver/dmwm-service-key.pem" % __file__.rsplit('/', 3)[0]

where __file__ = /data/srv/current/config/crabserver/ and therefore __file__.rsplit('/', 3)[0] = /data/srv/current/.

Some of the CMS services use Grid service certificates for interactions with CMSWEB, but the majority, and in particular the production and pre-production CRAB services, use the operator's proxy. The reasons are both for convenience and security. For private installations you are the operator, so you should use your own user proxy:

voms-proxy-init --voms cms --valid 192:00
sudo cp /tmp/x509up_u$UID /data/srv/current/auth/crabserver/dmwm-service-cert.pem
sudo cp /tmp/x509up_u$UID /data/srv/current/auth/crabserver/dmwm-service-key.pem

The proxy is created for 8 days (192 hours), because this is the maximum allowed duration of the VO CMS extension. Thus, the proxy has to be renewed every 7 days (at least). You can do it manually (executing the last three commands) or you can set up an automatic renewal procedure like is being done in production and pre-production.

Make sure the certificate and private key are owned by _sw _config (or _crabserver _crabserver for newer HG tags):

ls -l /data/srv/current/auth/crabserver/dmwm-service-*

-r--r-----. 1 _sw _config 5233 Jun  1 19:15 /data/srv/current/auth/crabserver/dmwm-service-cert.pem
-r--r-----. 1 _sw _config 5233 Jun  1 19:15 /data/srv/current/auth/crabserver/dmwm-service-key.pem

If not, do

sudo chown _sw:_config /data/srv/current/auth/crabserver/dmwm-service-{cert,key}.pem

Note: Grid host certificates can not be used as Grid service certificates if they are not registered in VO CMS and SiteBD.

Using your own CRABServer (and WMCore) repository

If you are a developer, most probably you want to use your own repositories.

1) Clone the repositories.

Fork the dmwm/CRABServer and dmwm/WMCore repositories on github to have them under your github username (you need to have a github account) and clone them.

Note: The instructions below suggest to put the cloned repositories into the /data/user/ directory of your host. But if you will install the REST Interface and the TaskWorker on different hosts, then you should better use another place that is accessible from both hosts, e.g. AFS.

cd /data/user/
git clone<your-github-username>/CRABServer
cd CRABServer
git remote -v
origin<your-github-username>/CRABServer (fetch)
origin<your-github-username>/CRABServer (push)
git remote add upstream
git remote -v
origin<your-github-username>/CRABServer (fetch)
origin<your-github-username>/CRABServer (push)
upstream (fetch)
upstream (push)

cd /data/user/
git clone<your-github-username>/WMCore
cd WMCore
git remote -v
origin<your-github-username>/WMCore (fetch)
origin<your-github-username>/WMCore (push)
git remote add upstream
git remote -v
origin<your-github-username>/WMCore (fetch)
origin<your-github-username>/WMCore (push)
upstream (fetch)
upstream (push)

Each crabserver release uses a given version of WMCore as specified in Unless you will use an old tag of CRABServer, you should use the given WMCore tag:

git checkout <tag> # e.g. 1.3.6.crab5

2) Configure crabserver to use your repositories.

In the crabserver init script, fix the setting of the PYTHONPATH to point to your cloned CRABServer and WMCore repositories.

Edit the file /data/srv/current/sw.pre/*/cms/crabserver/*/etc/profile.d/, comment out the lines that are changing the PYTHONPATH (should be the last two lines) and add a new line at the end of the script adding your repositories to the PYTHONPATH (here we assume the repositories are under /data/user/):

#[ ! -d /data/srv/HG1505c/sw.pre/slc6_amd64_gcc481/cms/crabserver/3.3.16.rc2/${PYTHON_LIB_SITE_PACKAGES} ] || export PYTHONPATH="/data/srv/HG1505c/sw.pre/slc6_amd64_gcc481/cms/crabserver/3.3.16.rc2/${PYTHON_LIB_SITE_PACKAGES}${PYTHONPATH:+:$PYTHONPATH}";
#[ ! -d /data/srv/HG1505c/sw.pre/slc6_amd64_gcc481/cms/crabserver/3.3.16.rc2/x${PYTHON_LIB_SITE_PACKAGES} ] || export PYTHONPATH="/data/srv/HG1505c/sw.pre/slc6_amd64_gcc481/cms/crabserver/3.3.16.rc2/x${PYTHON_LIB_SITE_PACKAGES}${PYTHONPATH:+:$PYTHONPATH}";
export PYTHONPATH=/data/user/CRABServer/src/python/:/data/user/WMCore/src/python/:$PYTHONPATH

Install/Scratch/Update the Oracle database tables

1) Create a configuration for connecting to the Oracle database, needed to install the database tables.

vi /data/

Copy/paste the following content (put your oracle_username and oracle_password):

from WMCore.Configuration import Configuration
config = Configuration()
config.CoreDatabase.connectUrl = 'oracle://<oracle_username>:<oracle_password>@devdb11'
Beware we will switch to devdb19 in early 2021

2) Set the environment. see

export ORACLE_CERN=/afs/
. $ORACLE_CERN/script/
# for devdb11 use this
setoraenv -s prod
# for devdb19 use this
setoraenv -s 19090
source /data/srv/current/sw*/*/cms/crabserver/*/etc/profile.d/ 
# if you use devdb19 MUST now repeat the command
setoraenv -s 19090

3) Install the required database tables (you have to clone WMCore from github so you have the wmcore-db-init command).

/data/user/WMCore/bin/wmcore-db-init --config /data/ --create --modules=Databases.TaskDB,Databases.FileMetaDataDB,Databases.FileTransfersDB

4) To scratch the database you will need to run the following commands:

export PYTHONPATH=/data/user/WMCore/src/python/:$PYTHONPATH #we need additional WMCore modules
/data/user/WMCore/bin/wmcore-db-init --config /data/ --destroy --modules=Databases.TaskDB,Databases.FileMetaDataDB,Databases.FileTransfersDB

N.B.: the --destroy command will be available in WMCore once this pull is merged:

5) To update the database when new code requires new columns in Task data base, the DB table needs to be modified by hand. The SQL commands to execute are added to updateOracle.sql by the developer who introduces the DB changes. Those command can be e.g. copied and pasted to the Oralcle SQL prompt on lxplus, :

#  see
export ORACLE_CERN=/afs/
. $ORACLE_CERN/script/
setoraenv -s prod
sqlplus <username>@devdb11
sqlplus <username>/<password>@devdb11


belforte@lxplus0067/ISTRUZ> sqlplus crab3_belforte/c3s-1705@devdb11

SQL*Plus: Release Production on Fri Jul 24 11:59:04 2015

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Enter password: 

Connected to:
Oracle Database 11g Enterprise Edition Release - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

SP2-0310: unable to open file "?/sqlplus/admin/glogin.sql"
SQL> select table_name from user_tables;


SQL> alter table tasks add (tm_user_files CLOB DEFAULT '[]');
alter table tasks add (tm_user_files CLOB DEFAULT '[]')
ERROR at line 1:
ORA-01430: column being added already exists in table

SQL> alter table tasks add (tm_publish_groupname VARCHAR(1) DEFAULT 'F');

Table altered.

SQL> alter table tasks add constraint check_tm_publish_groupname check (tm_publish_groupname IN ('T', 'F')) ENABLE;

Table altered.

SQL> alter table tasks add (tm_nonvalid_input_dataset VARCHAR(1) DEFAULT 'T');
alter table tasks add constraint ck_tm_nonvalid_input_dataset check (tm_nonvalid_input_dataset IN ('T', 'F')) ENABLE;

Table altered.

Table altered.

SQL> quit
Disconnected from Oracle Database 11g Enterprise Edition Release - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

Start/stop the REST Interface

Create a file named with the following content:

if [ $numargs -eq 0 ]; then
  echo "Please specify the action (start, stop or status) to take on local CRAB server."
elif [ $numargs -eq 1 ]; then
  echo "Sourcing environment from /data/srv"
  source /data/srv/current/sw.pre/slc6_amd64_gcc481/cms/crabserver/3.3.16.rc2/etc/profile.d/
  echo "Executing cd /data; /data/cfg/admin/InstallDev -d /data/srv -s $action; cd $cwd; stty sane"
  cd /data; /data/cfg/admin/InstallDev -d /data/srv -s $action; cd $cwd; stty sane
  echo "This script takes only 1 argument ($numargs arguments were given)."

Now you can source the script to perform one of the following actions on the crabserver, crabcache and frontend services:

  • Start:

source start

Check that you can access the frontend using the URL https://<host-name>/ You will be able to access the services you installed. In particular, https://<host-name>/crabserver/dev/workflow, etc.

  • Get the status:

source status

  • Stop:

source stop


Somehow on SL7 ipable permissions defined during installation gets lost and/or overridded, you need to do

sudo iptables -F

Special settings of the Oracle production database

In the production database for security reason SELECT and UPDATE/INSERT are performed by different users with appropriate readonly and writeonly privileges, namely cms_analysis_reqmgr_r and cms_analysis_reqmgr_w. Every request that is done using a GET will use the reader account, every POST or PUT will use the writer account.

If you add a table you need to add the following privileges as well (not tested):


You also need to create synonyms on the reader and witer accounts for the added table (TODO document).

Log files

Supposing the setup has been done under the /data/srv directory, the log files of the REST interface are located under /data/srv/logs/crabserver/.

The log file shows all HTTP that receives and shows the resources called, the returned status code, the calling IP and DN. Of course, also the date is associated for every log entry. As example:

[12/Jul/2013:15:14:15] mattia-dev01 "GET /crabserver/dev/workflowdb?getstatus=HOLDING&limit=6& HTTP/1.1" 200 [data: 5763 in 576 out 34844 us ] [auth: OK "/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=mcinquil/CN=660800/CN=Mattia Cinquilli" "" ] [ref: "" "CRABClient/0.0.1pre13" ]

The Apache frontend logs can instead be found under /data/srv/logs/frontend/.

Pre-Installed CRABSERVER

A development version of crabserver is available thanks to Todor and Emilis. which is an alias for

This can be used for quick checks w/o need to install a personal one. Check with other people before changing things !

  • to submit to it: config.General.instance = '' (can't use alias here)
  • to operate on it, ssh and then sudo su crab3
  • repository: crab server code comes from /user/data which is cloned from Emilis' GIT
  • start/stop/check server with : /etc/init.d/crabserver start/stop/status
  • connected to dev TW: which is an alias for

Tips and tricks

Client fails to contact server with No such instance error

In the server log you can usually find:
[11/Jul/2013:10:07:03]  SERVER REST ERROR WMCore.REST.Error.NoSuchInstance 30cacdcf7522dbb72b1310e875070251 (No such instance)
[11/Jul/2013:10:07:03]    Traceback (most recent call last):
[11/Jul/2013:10:07:03]      File "/data/hg1307b/sw.pre.spiga/slc5_amd64_gcc461/cms/crabserver/3.2.0pre15/lib/python2.6/site-packages/WMCore/REST/", line 693, in default
[11/Jul/2013:10:07:03]        return self._call(RESTArgs(list(args), kwargs))
[11/Jul/2013:10:07:03]      File "/data/hg1307b/sw.pre.spiga/slc5_amd64_gcc461/cms/crabserver/3.2.0pre15/lib/python2.6/site-packages/WMCore/REST/", line 731, in _call
[11/Jul/2013:10:07:03]        self._precall(param)
[11/Jul/2013:10:07:03]      File "/data/hg1307b/sw.pre.spiga/slc5_amd64_gcc461/cms/crabserver/3.2.0pre15/lib/python2.6/site-packages/WMCore/REST/", line 1836, in _precall
[11/Jul/2013:10:07:03]        raise NoSuchInstance()
[11/Jul/2013:10:07:03]    NoSuchInstance
[11/Jul/2013:10:07:03] c3p1 "GET /crabserver/dev/info?subresource=delegatedn HTTP/1.1" 404 [data: 5479 in 1962 out 4438 us ] [auth: OK "/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=mmascher/CN=720897/CN=Marco Mascheroni" "" ] [ref: "" "CRABClient/3.2.0" ]

If the client is correctly performing calls to the server, this can be a symptom of misconfiguration in the database configuration file /data/srv/current/auth/crabserver/

Client fails to contact server with You are not allowed to access this resource error

In the server log you can usually find:

[11/Jul/2013:10:09:44]  SERVER HTTP ERROR cherrypy._cperror.HTTPError f0aa48dfd2f9d215c2214434494543d7 ((403, 'You are not allowed to access this resource.'))
[11/Jul/2013:10:09:44]    Traceback (most recent call last):
[11/Jul/2013:10:09:44]      File "/data/hg1307b/sw.pre.spiga/slc5_amd64_gcc461/cms/crabserver/3.2.0pre15/lib/python2.6/site-packages/WMCore/REST/", line 693, in default
[11/Jul/2013:10:09:44]        return self._call(RESTArgs(list(args), kwargs))
[11/Jul/2013:10:09:44]      File "/data/hg1307b/sw.pre.spiga/slc5_amd64_gcc461/cms/crabserver/3.2.0pre15/lib/python2.6/site-packages/WMCore/REST/", line 764, in _call
[11/Jul/2013:10:09:44]        v(apiobj, request.method, api, param, safe)
[11/Jul/2013:10:09:44]      File "/data/hg1307b/sw.pre.spiga/slc5_amd64_gcc461/cms/crabserver/3.2.0pre15/lib/python2.6/site-packages/CRABInterface/", line 20, in validate
[11/Jul/2013:10:09:44]        authz_login_valid()
[11/Jul/2013:10:09:44]      File "/data/hg1307b/sw.pre.spiga/slc5_amd64_gcc461/cms/crabserver/3.2.0pre15/lib/python2.6/site-packages/CRABInterface/", line 50, in authz_login_valid
[11/Jul/2013:10:09:44]        raise cherrypy.HTTPError(403, "You are not allowed to access this resource.")
[11/Jul/2013:10:09:44]    HTTPError: (403, 'You are not allowed to access this resource.')
[11/Jul/2013:10:09:44] c3p1 "GET /crabserver/dev/info?subresource=delegatedn HTTP/1.1" 403 [data: 5479 in 2074 out 8193 us ] [auth: OK "/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=mmascher/CN=720897/CN=Marco Mascheroni" "" ] [ref: "" "CRABClient/3.2.0" ]

This is usually a symptom of misconfiguration of a private deployment. In a private deployment it is needed to set up the users that are allowed to access the frontend in the /data/srv/state/frontend/etc/authmap.json file. If you do not disable the crontab entry as indicated in Authentication with frontend, then the authmap.json file will be rewritten with only your DN.

Note: This applies exclusively to private deployments and not to the CMSWEB environment.

Client fails to contact server with Impossible to retrieve proxy from for /DN/... error

In the server log you can usually find:
[11/Jul/2013:10:14:14]  SERVER REST ERROR WMCore.REST.Error.InvalidParameter 6ab2195b14aa1cc3d1929e75a4ad17a2 (Invalid input parameter)
[11/Jul/2013:10:14:14]    Traceback (most recent call last):
[11/Jul/2013:10:14:14]      File "/data/hg1307b/sw.pre.spiga/slc5_amd64_gcc461/cms/crabserver/3.2.0pre15/lib/python2.6/site-packages/WMCore/REST/", line 693, in default
[11/Jul/2013:10:14:14]        return self._call(RESTArgs(list(args), kwargs))
[11/Jul/2013:10:14:14]      File "/data/hg1307b/sw.pre.spiga/slc5_amd64_gcc461/cms/crabserver/3.2.0pre15/lib/python2.6/site-packages/WMCore/REST/", line 768, in _call
[11/Jul/2013:10:14:14]        obj = apiobj['call'](*safe.args, **safe.kwargs)
[11/Jul/2013:10:14:14]      File "/data/hg1307b/sw.pre.spiga/slc5_amd64_gcc461/cms/crabserver/3.2.0pre15/lib/python2.6/site-packages/WMCore/REST/", line 1713, in dbapi_wrapper
[11/Jul/2013:10:14:14]        self._dberror(e, format_exc(), False)
[11/Jul/2013:10:14:14]      File "/data/hg1307b/sw.pre.spiga/slc5_amd64_gcc461/cms/crabserver/3.2.0pre15/lib/python2.6/site-packages/WMCore/REST/", line 1711, in dbapi_wrapper
[11/Jul/2013:10:14:14]        return handler(*xargs, **xkwargs)
[11/Jul/2013:10:14:14]      File "/data/hg1307b/sw.pre.spiga/slc5_amd64_gcc461/cms/crabserver/3.2.0pre15/lib/python2.6/site-packages/CRABInterface/", line 183, in put
[11/Jul/2013:10:14:14]        edmoutfiles=edmoutfiles, runs=runs, lumis=lumis, totalunits=totalunits)
[11/Jul/2013:10:14:14]      File "/data/hg1307b/sw.pre.spiga/slc5_amd64_gcc461/cms/crabserver/3.2.0pre15/lib/python2.6/site-packages/CRABInterface/", line 111, in submit
[11/Jul/2013:10:14:14]        return self.workflow.submit(*args, **kwargs)
[11/Jul/2013:10:14:14]      File "/data/hg1307b/sw.pre.spiga/slc5_amd64_gcc461/cms/crabserver/3.2.0pre15/lib/python2.6/site-packages/CRABInterface/", line 107, in wrapped_func
[11/Jul/2013:10:14:14]        raise invalidp
[11/Jul/2013:10:14:14]    InvalidParameter
[11/Jul/2013:10:14:14] c3p1 "PUT /crabserver/dev/workflow HTTP/1.1" 400 [data: 6351 in 2312 out 368852 us ] [auth: OK "/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=mmascher/CN=720897/CN=Marco Mascheroni" "" ] [ref: "" "CRABClient/3.2.0" ]

As said in the previous sections, the CRAB REST interface needs to access MyProxy, and in this case it is failing, because it cannot retrieve the user proxy from MyProxy. The source of this failure is not unique:

Please check the value of the data.serverhostcert and data.serverhostkey parameters: by default they point to dmwm-service-key.pem, which is what cmsweb expects but does not work for private installations. Please set these two parameters to:

data.serverhostcert = '/data/srv/current/auth/crabserver/hostcert.pem'
data.serverhostkey = '/data/srv/current/auth/crabserver/hostkey.pem'

Also, another possible solution for "Impossible to retrieve proxy" error is using your own proxy instead of the hostcert/hostkey. You can do that by modifying the external configuration and adding your user DN to the delegate-dn field so that it looks something like this:

delegate-dn": [
            "/DC=ch/DC=cern/OU=computers/|/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=erupeika/CN=775659/CN=Emilis Antanas Rupeika"
  • The proxy of the user has not been delegated to MyProxy; there is an utility to help the debugging of such issues which can be used as following:
    [15:27:04][mcinquil@mattia-dev01] ~> source /data/srv/current/apps/crabserver/etc/profile.d/ 
    [15:28:03][mcinquil@mattia-dev01] ~> wget --no-check-certificate
    --2013-07-12 15:28:10--
    Connecting to||:443... connected.
    WARNING: cannot verify's certificate, issued by `/C=US/O=DigiCert Inc/ High Assurance CA-3':
      Unable to locally verify the issuer's authority.
    HTTP request sent, awaiting response... 200 OK
    Length: 1000 [text/plain]
    Saving to: `'
    100%[==================================================================================================================================================================>] 1,000       --.-K/s   in 0s      
    2013-07-12 15:28:10 (19.1 MB/s) - `' saved [1000/1000]
    [15:28:10][mcinquil@mattia-dev01] ~> python 
    Usage example: 
    python "/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=mcinquil/CN=660800/CN=Mattia Cinquilli" /data/certs/hostkey.pem /data/certs/hostcert.pem "/DC=ch/DC=cern/OU=computers/"

Task worker fails to connect to Frontend with HHTP code 403 and "rejecting non-vo certificate" on the frontend

on the Taskworker:

2019-01-15 11:13:29,264:ERROR:MasterWorker:HTTP Error during _lockWork: url=, code=403, reason=Forbidden, headers={'Date': 'Tue, 15 Jan 2019 10:13:29 GMT', 'Transfer-Encoding': 'chunked', 'Content-Type': 'text/html', 'CMS-Server-Time': 'D=27681 t=1547547209236680', 'Server': 'Apache'}
HTTP Headers are {'Date': 'Tue, 15 Jan 2019 10:13:29 GMT', 'Transfer-Encoding': 'chunked', 'Content-Type': 'text/html', 'CMS-Server-Time': 'D=27681 t=1547547209236680', 'Server': 'Apache'}: 

on the Frontend error log:

[Tue Jan 15 10:42:20.183765 2019] [perl:warn] [pid 16811] [client] [cookie-auth/16811] all authentication method failed; forbidding /crabserver/dev/workflowdb
[Tue Jan 15 10:42:50.416934 2019] [perl:notice] [pid 16705] [client] [cookie-auth/16705] rejecting non-vo certificate /DC=ch/DC=cern/OU=computers/CN=tw/, verify SUCCESS, vstart Jan 11 11:50:08 2019 GMT, vend Feb 15 11:50:08 2020 GMT, vremai
n 397

The reason is the lack of voms-gridmap file (not the authmap.json file). This should be created by a cronjob running under the deployment user (either the operator's username or crab3) and the cronjobs itself should have been created by /data/cfg/frontend/deploy and should look llike:

13 */3 * * * /data/srv/current/config/frontend/mkvomsmap --key /data/certs/hostkey.pem --cert /data/certs/hostcert.pem -c /data/srv/current/config/frontend/mkgridmap.conf -o /data/srv/state/frontend/etc/voms-gridmap.txt --vo cms

This cronjob usually vanishes in puppet managed machines because puppet tends to be smarter than one could accept and deletes all user's cronjobs that are not maintained by puppet. This could be prevented if in the puppet profile is added:

site_cron_purge: false

What to do if your Oracle database is growing too much

This is the simple procedure and list of commands you need to execute to clean everything older than 1 month from the filemetadata and filetransfers tables.

  • SQL commands
select * from user_ts_quotas;
delete from filemetadata where fmd_creation_time < ADD_MONTHS (SYSDATE, -1);
alter table filemetadata enable row movement;
alter table filemetadata shrink space;
select * from user_ts_quotas;
delete from filetransfersdb where tm_creation_time  < ADD_MONTHS (SYSDATE, -1);
alter table filetransfersdb enable row movement;
alter table filetransfersdb shrink space;
select * from user_ts_quotas;

If you want to cleanup everything older than 1 week, use

delete from filemetadata where fmd_creation_time < (SYSDATE - 7);
  • Example
/afs/ > sqlplus

SQL*Plus: Release Production on Fri May 9 01:34:37 2014

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Enter password:

Connected to:
Oracle Database 11g Enterprise Edition Release - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SP2-0310: unable to open file "?/sqlplus/admin/glogin.sql"

SQL> select * from user_ts_quotas;

------------------------------ ---------- ---------- ---------- ---------- ---
DATA01                 51642368   52428800       6304       6400 NO

SQL> delete from filemetadata where fmd_creation_time < ADD_MONTHS
(SYSDATE, -1);

30034 rows deleted.


Recyclebin purged.

SQL> alter table filemetadata enable row movement;

Table altered.

SQL> alter table filemetadata shrink space;

Table altered.

SQL> select * from user_ts_quotas;

------------------------------ ---------- ---------- ---------- ---------- ---
DATA01                 30539776   52428800       3728       6400 NO

useful SQL commands

/* list tables */
select table_name from user_tables;
/* list columsn in a table */
describe filemetadata;
/* count rows in a tale */
select count(*) from tasks; 
/* exit sql */
See also
Edit | Attach | Watch | Print version | History: r85 < r84 < r83 < r82 < r81 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r85 - 2021-02-15 - StefanoBelforte

    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback