HTCondor CE token configuration tips

ALERT! OSG sites should follow their own OSG HTCondor-CE documentation and related upgrade notes for OSG 3.5 upcoming or OSG 3.6 release.


Please refer to the HTCondor download documentation.

While your service needs to support job submissions with X509 / VOMS proxies for some VO(s), please use the Long Term Support (LTS) Channel (a.k.a. stable) which currently denotes v9.0.x.

The EGI UMD will remain on the LTS until its EOL, planned for Feb 2023; v9.0.12 has been requested and should soon appear there.

If all the VOs supported by your service can submit their jobs using tokens (e.g. SciTokens a.k.a. WLCG tokens), then the Feature Channel can be used, e.g. v9.8.x and higher.

For the HTCondor CE component, please use v5 and refer to its documentation.

Configuring HTCondor-CE

The following examples assume that the "new syntax" is being used. OSG has created a web page that contains the default mappings that come with the osg-scitokens-mapping rpm. osg-scitokens-mapfile.conf

Routes for ATLAS and CMS are set to work with pilots having GSI proxy or SciToken or both. Currently (Feb. 2022) pilots have both, but these routes are expected to also match when the x509 proxy is missing.


For the time being, the mappings are done based on the token issuer + subject (cf. proxy DN). In the future we expect to be able to map tokens also based on the groups they contain (cf. VOMS attributes) and possibly even on the scopes (capabilities) they possess.


We assume that the VO declares to the HTCondor-CE administrator which JWT issuer + subject is going to submit which kind of jobs (i.e. production, analysis, SAM tests). Currently, up to three distinct JWT subjects per issuer are expected to be declared. This information is needed to configure the htcondor-ce mapfile. In the case of ATLAS and CMS, for example:

The mapfile

Assuming the following content for ATLAS and CMS:

# issuer and subject can be found here:
# TIP: we map to a unix username which reminds the CE hostname (ce07 <--> 007)
[root@ce07-htc ~]# cat /etc/condor-ce/mapfiles.d/10-scitokens.conf | egrep 'cms|atlas'
SCITOKENS /^https:\/\/atlas-auth\.web\.cern\.ch\/,7dee38a3-6ab8-4fe2-9e4c-58039c21d817$/ atlasprd007
SCITOKENS /^https:\/\/atlas-auth\.web\.cern\.ch\/,750e9609-485a-4ed4-bf16-d5cc46c71024$/ pilatlas007
SCITOKENS /^https:\/\/atlas-auth\.web\.cern\.ch\/,5c5d2a4d-9177-3efa-912f-1b4e5c9fb660$/ atlassgm007

SCITOKENS /^https:\/\/cms-auth\.web\.cern\.ch\/,08ca855e-d715-410e-a6ff-ad77306e1763$/ cmssgm007
SCITOKENS /^https:\/\/cms-auth\.web\.cern\.ch\/,490a9a36-0268-4070-8813-65af031be5a3$/ pilcms007
SCITOKENS /^https:\/\/cms-auth\.web\.cern\.ch\/,bad55f4e-602c-4e8d-a5c5-bd8ffb762113$/ cmsprd007

We set the following routes:

Routes for ATLAS (optional)

These examples shows how to use job routing REQUIREMENTS for matching jobs submitted with tokens. Currently all ATLAS production job gets submitted with delegated X.509 VOMS proxy, HTCondor-CE job classAd still contains all x509* attributes and that makes AuthTokenIssuer + AuthTokenSubject configuration optional. Job routing with token classAds become mandatory once ATLAS starts to submit jobs that completely rely on tokens also for job payload, but this is not going to happen in a near future (it is also not clear if "sub" claim is the best option and there might be some changes once we get more token experience).

[root@ce07-htc config.d]# cat 90-jobrouternew.conf
JOB_ROUTER_ROUTE_atlas_sam @=jrt
  REQUIREMENTS (x509UserProxyVoName =?= "atlas" && x509UserProxyFirstFQAN =?= "/atlas/Role=lcgadmin/Capability=NULL") ||\
 (AuthTokenIssuer =?= "" &&\
 AuthTokenSubject =?= "c3a22a4c-8257-1abf-562c-4c15e8fac421")
  SET Requirements (TARGET.t1_allow_sam =?= true) && (!StringListMember("gpfs_atlas",t1_GPFS_CHECK ?: "",":"))
  SET MaxJobs 35
  SET MaxIdleJobs 12

JOB_ROUTER_ROUTE_atlas @=jrt
  REQUIREMENTS (((x509UserProxyVoName =?= "atlas") ||\
 (AuthTokenIssuer =?= "" &&\
 StringListMember(AuthTokenSubject ?: "",\
 "7dee38a3-6ab8-4fe2-9e4c-123abc82f732:642c5236-548b-2ad1-3b42-6fab13b21f03", ":"))) && (queue == "atlas"))
  SET Requirements (!StringListMember("gpfs_atlas",t1_GPFS_CHECK ?: "",":")) && (t1_allow_sam =!= true)
  SET MaxJobs 3500
  SET MaxIdleJobs 1280

Notes on the above:

  • Only the REQUIREMENTS boolean expression is involved here
  • Simply matching the job Owner in the REQUIREMENTS (i.e. atlasprd007 ) would require different configurations for each CE since the username identifies the CE.

AuthToken* attributes in the routed job (optional)

While the x509UserProxy* attributes are set in the routed job, the AuthToken* are not. Both the X.509 and Token attribute names are protected, and a JobRouter rule has insufficient privileges to set values for these names. It is however easy to do by simply altering the names. The following example prepend a capital R (as in Routed) to the AuthToken* attributes


  REQUIREMENTS MY.AuthTokenSubject =!= undefined
  EVALSET RAuthTokenId AuthTokenId
  EVALSET RAuthTokenIssuer AuthTokenIssuer
  EVALSET RAuthTokenSubject AuthTokenSubject
  EVALSET RAuthTokenIssuer AuthTokenIssuer

Experimental configurations

Use case: Hierarchical fairshare (optional)

Consider a VO (wlcg in this example) having several different subgroups (i.e. FQAN=/wlcg/Role=role1 ... FQAN=/wlcg/Role=roleN) in GSI terms. and assume that hierarchical fairshare should be granted to these subgroups. That is, in htcondor terms:

GROUP_QUOTA_DYNAMIC_wlcg.role1 = 0.1
GROUP_QUOTA_DYNAMIC_wlcg.role2 = 0.2
GROUP_QUOTA_DYNAMIC_wlcg.role3 = 0.3
GROUP_QUOTA_DYNAMIC_wlcg.role4 = 0.4

The following example assumes that the jwt brings the information about the job role exactly in the second element of the groups list. For example, given the following "groups" for the "wlcg" VO, the subgroup name would be "role3":

 "iss": "",  
 "wlcg.groups": [

We want to extract the subgroup name and use it to assign the job to the correct condor AcctGroup. The wlcg.groups value is copied by condor-ce into the AuthTokenGroups classad. In this case, using a jobrouter entry like this:

 JOB_ROUTER_ROUTE_wlcg @=jrt
   REQUIREMENTS  AuthTokenIssuer =?= ""
   COPY AuthTokenGroups MyGroup
   EVALSET TokenGroup strcat(Owner,split(MyGroup,",")[1])
   EVALSET AcctGroup UserMap("AssignAccountingGroup",TokenGroup)

The UserMap function above takes the file defined by

CLASSAD_USER_MAPFILE_AssignAccountingGroup = /some/path/TokenGroups.txt

whose content is:

[root@ce01t-htc ~]# cat /some/path/TokenGroups.txt
  * wlcg001/wlcg/role1  wlcg.role1
  * wlcg001/wlcg/role2  wlcg.role2
  * wlcg001/wlcg/role3  wlcg.role3

WLCG sites status

Edit | Attach | Watch | Print version | History: r19 < r18 < r17 < r16 < r15 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r19 - 2022-06-21 - PetrVokac
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback