Note on site-launched startd's

For a trusted resource, the site should create a pilot certificate from a CA recognized at CERN and send the DN to the Submission Infrastructure group ( who will enter the certificate DN in the condor_mapfile of the central manager, etc. In special circumstances, the Submission Infrastructure group can provide a CMS pilot certificate.

Creating Pilot and Service Certificates

Every grid job, be it a glideinWMS pilot (glidein) or a user job, needs a grid proxy in order to authenticate at sites. A proxy is created on the frontend from a grid certificate and a key. This page describes how to obtain new pilot certificates from the CERN Certificate Authority.

Creating a New Certificate

On the CERN Certificate Authority website, go to "New Grid Host Certificate" and choose "Request certificate using OpenSSL (for Linux machines)" here. Create a certificate request with a subject, for example:

You need to be an owner of the machine at CERN in the certificate request name. You should use a certificate-friendly browser like Firefox if you want to not make your life difficult.

You will next be asked to generate a certificate-key pair. You can do this on lxplus:

openssl req -new -subj "/CN=cmspilot01\/" -out newcsr.csr -nodes -sha512 -newkey rsa:2048
or follow the updated instructions on the CERN CA webpage if they have changed. Once the certificate is generated, download the base64 certificate to a file, which we will call host.cert.

Next we will export the certificate from this file to a p12 file:

openssl pkcs12 -export -inkey privkey.pem -in host.cert -out ${dir}.p12

Create cert file and key file from the p12 file and set permissions correctly:

openssl pkcs12 -clcerts -nokeys -in ${dir}.p12 -out ${dir}cert.pem
openssl pkcs12 -nocerts -in ${dir}.p12 -out ${dir}key-enc.pem
openssl rsa -in ${dir}key-enc.pem -out ${dir}key.pem
chmod 400 ${dir}key*
chmod 600 ${dir}cert.pem

Next verify the cert and key have the same hash:

openssl x509 -noout -modulus -in ${dir}cert.pem | openssl md5 
openssl rsa -noout -modulus -in ${dir}key.pem |openssl md5 | uniq
This should always be the case unless you did something really incorrect like mix the files from two different certificates.

Lastly, get some information about the certificate like its Distinguished Name (DN) subject, and its validation period:

openssl x509 -in ${dir}cert.pem -noout -subject -startdate -enddate

Register the Certificate with CMS

For service certificates (i.e. for the frontend), you can skip this step.

"Lasciate ogne speranza, voi ch'intrate" - Dante Alighieri

In order to use CMS resources, every pilot certificate must be registered with the CMS VOMS group as a member of CMS, as well as get special roles like the pilot role. This distinguishes pilot jobs from other types of jobs like user jobs. Follow the procedure to "Add an additional certificate" on the CMS VOMS administration website ( direct link may work, if it does not, use the previous link and scroll down about a page to find the button on the right side) VOMS administrators are Stefano, Andreas Pfeiffer and Tony, if you need speedy approval. Note also that as of Summer 2015 the Global Pool certificate needs also the production role since fair-share at the Tier-1 sites depends on having it.

Collector Authorization

As user condor, authorize the new certificate on the collectors:

glidecondor_addDN -daemon "cms pilot cert DN" "/DC=ch/DC=cern/OU=computers/CN=cmspilot12/" pilot12
Each glidein needs to communicate with the collector, and cannot do so without authorization in the condor_mapfile. Never update the condor_mapfile by hand! Use the script glidecondor_addDN.

Certificate Repository

There isn't one.

Creating a Proxy

You should store the certificate and key files on the glideinWMS frontend. Next you will have to create proxies. The proxies are short-lived versions of the certificate information that can be used more safely on the grid. In the Global Pool, these are found in, and the certificate and key files in the certs directory below that. The fundamental procedure boils down to:

Create the base proxy:

voms-proxy-init -cert ${certdir}/cmspilot${idstr}cert.pem -key ${certdir}/cmspilot${idstr}key.pem -hours 72

Add the appropriate VOMS roles to the base proxy, for example.

export X509_USER_PROXY=${here}/x509_pilot${idstr}_cms.proxy.tmp
voms-proxy-init -cert ${certdir}/cmspilot${idstr}cert.pem -key ${certdir}/cmspilot${idstr}key.pem -voms cms -hours 72 -valid 72:0
See the script for the full details. This script is run as a cron job. The resulting proxy files can be used in the glideinWMS frontend configuration xml file.

List of Certificates in Production

Next expiration date: April 28, 2020

subject= /DC=ch/DC=cern/OU=computers/CN=cmspilot01/
notAfter=Apr 28 22:53:58 2021 GMT
subject= /DC=ch/DC=cern/OU=computers/CN=cmspilot02/
notAfter=Apr 28 22:59:02 2021 GMT
subject= /DC=ch/DC=cern/OU=computers/CN=cmspilot03/
notAfter=Apr 28 23:02:03 2021 GMT
subject= /DC=ch/DC=cern/OU=computers/CN=cmspilot04/
notAfter=Apr 28 23:06:18 2021 GMT
subject= /DC=ch/DC=cern/OU=computers/CN=cmspilot05/
notAfter=Apr 28 23:07:59 2021 GMT
subject= /DC=ch/DC=cern/OU=computers/CN=frontend01/
notAfter=Apr 28 23:10:58 2021 GMT
subject= /DC=ch/DC=cern/OU=computers/CN=frontend02/
notAfter=Apr 28 23:12:42 2021 GMT
subject= /DC=ch/DC=cern/OU=computers/CN=frontend03/
notAfter=Apr 28 23:14:31 2021 GMT
subject= /DC=ch/DC=cern/OU=computers/CN=frontend04/
notAfter=Apr 28 23:17:35 2021 GMT
subject= /DC=ch/DC=cern/OU=computers/CN=frontend05/
notAfter=Apr 28 23:19:14 2021 GMT

List of Active certificates

some of the nodes have been configured to use puppet managed certificate renewal. It use 'certmgr' puppet module provided by CERN IT. Once the certificate is in the proper place it is expected to be renewed automatically. Following is the list of all active pilot and frontend certificates in Submission Infrastructure

Node Service subject Certificate management/Next expiration
vocms080 Global pool FE service /DC=ch/DC=cern/OU=computers/CN=frontend02/ Manual renewal: expiry 28-April,2021
vocms080 Global pool pilot /DC=ch/DC=cern/OU=computers/CN=cmspilot02/ Manual renewal: expiry 28-April,2021
vocms0819 CERN pool FE service /DC=ch/DC=cern/OU=computers/CN=frontend05/ Manual renewal: expiry 28-April,2021
vocms0819 CERN pool pilot /DC=ch/DC=cern/OU=computers/CN=cmspilot05/ Manual renewal: expiry 28-April,2021
vocms0801 ITB-dev pool FE service /DC=ch/DC=cern/OU=computers/CN=frontend03/ puppet
vocms0801 ITB-dev pool pilot /DC=ch/DC=cern/OU=computers/CN=cmspilot03/ puppet
vocms0802 ITB pool FE service /DC=ch/DC=cern/OU=computers/CN=frontend04/ puppet
vocms0802 ITB pool pilot /DC=ch/DC=cern/OU=computers/CN=cmspilot04/ puppet
vocms0840 Volunteer pool FE service /DC=ch/DC=cern/OU=computers/CN=frontend04/ puppet
vocms0840 Volunteer pool pilot /DC=ch/DC=cern/OU=computers/CN=cmspilot06/ puppet

Script to simplify certificate creation (July 2016, last rev. March 2020) for all certificates at once.

# 24-Mar-2020 Some urls changed since last year
HOST=`basename $TOP`

if [ $HOST == "vocms080" ] ; then
  DIRS="cmspilot01 cmspilot02 cmspilot03 cmspilot04 cmspilot05 frontend01 frontend02 frontend03 frontend04 frontend05"
elif [ $HOST == "vocms052" ] ; then
  exit 1

for dir in $DIRS ; do
  if [ ! -d $TOP/$dir ] ; then
    mkdir $TOP/$dir
  cd $TOP/$dir
  if [ ! -f host.cert ] ; then
    cp /etc/pki/tls/openssl.cnf .
cat >>openssl.cnf <<END
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

    echo "DNS.1 = ${HOST}" >> openssl.cnf

    openssl req -new -subj "/CN=${dir}\/${HOST}" \
      -out newcsr.csr -nodes -sha512 -newkey rsa:2048 \
      -config ${TOP}/${dir}/openssl.cnf
    echo "Go to the following url and request the certificate with the"
    echo "information in newcsr.csr for certificate $dir/${HOST}"
    #echo ""
    echo "${dir}%2f${HOST}"
    echo "Download the base-64 certificate and then copy host.cert from"
    echo "your desktop to this area:"
    echo "*** Enter when prompted:"
    openssl pkcs12 -export -inkey privkey.pem -in host.cert -out ${dir}.p12
    openssl pkcs12 -clcerts -nokeys -in ${dir}.p12 -out ${dir}cert.pem
    openssl pkcs12 -nocerts -in ${dir}.p12 -out ${dir}key-enc.pem
    openssl rsa -in ${dir}key-enc.pem -out ${dir}key.pem
    chmod 400 ${dir}key*
    chmod 600 ${dir}cert.pem
    openssl x509 -noout -modulus -in ${dir}cert.pem | openssl md5 
    openssl rsa  -noout -modulus -in ${dir}key.pem | openssl md5 | uniq

exit 0

JamesLetts - 2020-03-24

Edit | Attach | Watch | Print version | History: r37 < r36 < r35 < r34 < r33 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r35 - 2021-03-04 - SaqibHaleem
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback