CRAB and local submission for site with a CE


This twiki is for site administrators who would like to give more priority and reserve some resources for their local users (thus the term local resources). It assumes that you have basic knowledge of the GlideInWMS system used by CRAB to access Grid resources (you know what a schedd, a frontend, a factory, and a pilot is). The original plan is outlined here

It goes w/o saying that local control can only be enforced on over-the-pledge resources, while prioritization of resources pledged to CMS stays with central CMS coordination.


Access to local resources in CRAB can be done through the GlideInWMS Global Pool deployed at CERN (which is also used for regular pledged resources).

In a nutshell, what normally happens is that "user jobs" are sent by CRAB to one or more schedd with their DESIRED_SITES list; GlideInWMS frontend looks at these user jobs, and then tells the GlideInWMS factory to send "pilot jobs" to the sites; then matching between available pilots and user jobs is done, and the CRAB job wrapper is downloaded for execution by the pilot. Normally, when pledged resources are requested, these pilots have VOMS FQAN /cms/Role=pilot (i.e. pilot role in the main cms group).

In the case of "local users" there is the possibility to tell the the factory to send special pilots with Role=pilot in the local group (i.e. VOMS FQAN /cms/local/Role=pilot) which will be matched only to local users' jobs, giving site administrators the possibility to manage the priority of local users by setting an appropriate priority for the /cms/local pilots on their end.

Each site indicates which user they consider "local" (i.e. can have access to local extra resources) using groups. One such group has the same name as the site and it is meant to allow sites to list single users as they see fit, but we envisions supporting also other kind of groups, starting with VOMS Groups and CERN e-groups.


Currently (March 2016) two ways to make this work are supported, they can both be used at same time by the site

  1. using the local group
  2. using one ore more VOMS groups

At the Site

local group

This group will be named like the site (e.g. T2_US_Nebraska ) and is populated with CERN computing user IDs. A site administrator will put the list of local users in GlideinConfig/local-users.txt file in SITECONF in CVMFS (see example [1] below). This file needs to be created in the site local configuration git repository. From there it will be automatically propagated to the worker nodes when CVMFS is updated. For sites not using CVMFS, they will still have to create this file in the git repository otherwise one of the SAM tests (org.cms.WN-basic) will start failing. In this case the site admin also has to manually make sure that the file is present on the worker nodes. Please contact the site support team at for more details.

The user name must be the same used in CRAB tasks and stageout i.e. the username listed in SiteDB. CRAB will look for this file to set a classAd (named "CMSGroups") in the job submission JDL which specifies the list of sites this user is "local" to. The frontend will then look at this list, and request the factory to send pilots using the /cms/local/Role=pilot credential to the particular site.


> less /cvmfs/

VOMS groups

A site administrator can also choose to define VOMS groups as local to the site, thus avoiding the need to list down individual names. Similar to the previous approach, a site administrator will put a list of VOMS group(s) in GlideinConfig/local-groups.txt file in SITECONF in CVMFS (see example [2] below). Note that the proper syntax to identify a VOMS group is /cms/[/<name]. When a user will submit a CRAB task, CRAB will look up all the groups associated with the user's certificate and add an extra classAd (named "CMSGroups") into the JDL (for example, see [3]). The frontend will then look at this CMSGroups classAd, and request the factory to send pilots using the /cms/local/Role=pilot credential to the particular site.


# ls -al /cvmfs/
total 2
drwxr-xr-x. 2 cvmfs cvmfs   3 Nov 21 04:21 .
drwxrwxr-x. 5 cvmfs cvmfs   3 Apr 27  2015 ..
-rw-r--r--. 1 cvmfs cvmfs  21 Dec  7 20:21 local-groups.txt
-rw-r--r--. 1 cvmfs cvmfs 304 May 11  2015 local-users.txt

# cat /cvmfs/ 
# voms-proxy-info -fqan

CRAB will add:

In the central submission infrastructure

CRAB adds and populates a classAd named "CMSGroups" for the local user feature to work. CRAB first looks up the various VOMS groups the user certificate belongs to, and adds them to CMSGroups. Note that VOMS proxies always list all the VOMS groups, differently from VOMS Roles where the user has to explicitely pick one. Next CRAB looks up,using CMSGroupMapper class, all the local-users.txt files provided by sites in CVMFS and if the user is "local" to any site, it also appends the CMS site name to the classAd. Basically the site name is treated like another group whose membership is defined via the local-user.txt file. If a user has some VOMS role associated with their certificate and are part of a local-users.txt file, the CMSGroups classAd for their jobs will, for example, look like this:
# condor_q cms1446 -af:h CMSGroups | uniq

If a user is not listed in any 'local-users.txt' CRAB simply adds the VOMS group details. For example:

# condor_q cms488 -af:h CMSGroups | uniq

In the glideinWMS frontend, there is a separate frontend group named "local_users" for sending local pilots configured with VOMS FQAN /cms/local/Role=pilot. In the match expression of the frontend group, glideinWMS makes use of "CMSGroupMapper" class and compares 'CMSGroups' defined by CRAB with what the site has in their CVMFS local files. If a match is found, frontend requests the glideinWMS factory to send "local" pilots to the site. The frontend adds an extra configuration parameter to these glideins ("CMSIsLocal") and sets its value to true.

When a pilot starts running, it runs a list of validation scripts on a worker node. While running '' (link here), the pilot checks if "CMSIsLocal" is set to true. Since this is being set for the "local" pilots, the script then reads local-groups.txt from the SITECONF area on the worker node and appends an extra requirement to the START expression of the startd. For example, in case of Nebraska:


This extra expression then prevents the pilot from being treated as a normal pilot and allows only the users with appropriate "CMSGroups" classAd to use it.

Responsible: JamesLetts
(Added per request of documentation services' team)
Edit | Attach | Watch | Print version | History: r12 < r11 < r10 < r9 < r8 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r12 - 2017-08-16 - KenyiPaoloHurtadoAnampa
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback