Journey to CMS Research:

Square One --> Grid Access

"The journey of a thousand miles begins beneath one's feet."
~ Lao-tzu, The Way of Lao-tzu Chinese philosopher (604 BC - 531 BC)


This walkthrough was only made possible by the selfless work of our colleague, on ATLAS, AustinBasye. NOTE: Everything that is covered here is also covered in the CMS Offline WorkBook. This is supposed to serve as a stremalined walkthrough.

So, you want to work at CMS? Unfortunately, unless you have worked at CERN or with CMS before, there is a bureaucratic-quagmire of hoops between you and the data that you are, no doubt, eager to analyze. This TWiki will guide you successfully through the hoops, point out pitfalls along the way, and hopefully get you through this hazing process safely, but more importantly, expeditiously.

Unfortunately, these hoops are not stationary. What is unnecessary today may be essential tomorrow; such is the nature of the beast. Thus, keep your eyes open for changes to the process, then, kindly update this TWiki with the correct information. (This will help familiarize you with the TWiki editing rituals as well as keep this document relevant)

Now, so as to make this guide more specific, I have made some assumptions (again, feel free to amend this document to remove any of these assumptions if you so wish):

  • You are an American Citizen with access to your Passport (or Driver's License?).
  • You are familiar with the process of downloading and installing applications on your computer.
  • You are currently working with one of the UIUC HEP groups.

  • You are NOT required to be familiar with a UNIX computing environment (Yet).
  • You are NOT required to know anything about the CMS computing structure (Yet).

First Things First - (Gather the Appropriate Documents/Info/Tools)

Before we get started, it would be a good idea to get a few things in order and readily available. However, these items are not essential for every step. Therefore, feel free to go on even if you may not have some of the following at the moment.

Needed Documents/Info

  1. Your passport!
  2. Your personal information.
  3. Your PI/Advisor's consent and contact information (fax, phone number, email).
  4. Various 0th order Tutorial Pages and CMS Administrations Pages:

Needed Tools

  1. An acceptable Browser
  2. A Terminal, or acceptable Terminal Emulator. (no T-shell)
    • Linux Distribution
      • If you are using a Linux Distribution... you know what I am talking about.
    • Mac OSX
      • Command+Space>"Terminal"
    • Windows
      • Putty - Easier of the two to get going, not very good for local use, (i.e. can't use your computer's resources easily).
      • Cygwin - Crazy to set (last I remember) up but good for local use, (i.e. can use your computer's resources, can forward X11 windows).
  3. ROOT - Primary Analysis/Visualization tool for HEP datasets
    • Not essential but very useful.
    • Might be a pain to install, depending on your computing environment.

The All Important Red Tape

Working at any Government Funded lab, or Government Funded anything for that matter, will require you to jump through a few hoops. Compared to the hoops we will encounter later, these will be relatively painless. Essentially, following these next steps, you will be registered as a user for both the UVa computing facilities, as well as, the CERN web based resources. (CERN Computing resources come later, see Obtaining a GRID Certificate)

Register with UVa Senior Scientist Mike Arenton

Approximate Time to Complete: [~15-30 Minutes]

  • This will allow you to login to the available Windows and lx Machines
  • Mike is located in room 219 at the HEP labs

Register as a CMS User with CERN

Approximate Time to Complete: [~3-7 Business Days]

  • Once you have completed this step, you will be notified of your new username/password via e-mail.

  • Navigate to the CERN account web-page and click on your preferred language version of Flyer for newcomers. Follow the instructions there.
    • You will then be asked to provide the username/password information supplied in the previous step.
    • Warning, important Next, you will have 3 days to pass the mandatory Cyber Security Training course, or your account will be blocked.
    • If your supplied password does not work right out of the box, (this happened to me) you may need to reset it. In which case you should contact the CERN IT Helpdesk at


The TWiki(s)

Our TWikis provide us with a way to organize and keep-relevant tons of diverse information. Everything from presentation files and small data samples to HOW-TO's and Interesting links can be stored, displayed and updated here. As such, your input is encouraged if not required in some cases.

There are 2 primary TWikis that we regularly interface with, that I am aware of: The CERN TWiki*, and the UIUC TWiki. Both of these require your registration to edit. The following steps will walk you through it. We will start by geting you registered with the UIUC TWiki as your group's meetings will probably host the presentation files here.

* This will require a CERN username and password.

UIUC TWiki Registration

Approximate Time to Complete: [~1-3 Business Days]

As stated here (check there for possible updated instructions), all you need to do is send an email to:, in the following format.

CERN TWiki Registration

Approximate Time to Complete: [~20-25 Minutes]

After logging in to your CERN account (login at upper right hand corner), go to the CERN TWiki Registration Page.

  • At this point, you should see a form which has been partially filled out with an email address ending in This is your new CERN-provided email address. Write this down as we will need it in a while.
  • Next, fill out your first and last name, set the country value appropriately, and write: University of Virginia, CMS in the CERN Affiliation box.
  • Make sure the information you supplied is correct, then hit the submit button.
  • This will send an Authorization email to your mail address.

Setting Up Your Cern Email

Approximate Time to Complete: [~5-10 Minutes]

We now need to get access to your mail!

  • If you are still logged on to your CERN account, go to the Mail Services page and click on the Outlook Web Access (Webmail) Link.
    • This will provide you with an online Inbox where you can manage your emails.
  • If you you would prefer a client handle your emails the following links will show you how to set them up correctly.
    • In the left-hand pannel of the Mail Services page, you will find numerous documents devoted to helping you set up your favorite mail client. As of right now I have yet to learn how to permanently forward these emails to another web-based service like Gmail, if this is even possible. (input?)
    • Here are the directions for:
  • Once you have set up your CERN email, open the inbox and look for a message from: TWiki Administrator <>.
  • This message should contain a statement to the fact that following a link will activate your account.
  • Follow this link and subsequently change your password to a new, secure one that you can remember. And you're done!

Computing Square One

This section covers very basic navigation to, and in the various computing environments. This is most likely too basic for most users, however, I feel that it is prudent to include as this TWiki is targeted at the most inexperienced. Feel free to skip this section if your are familiar with Secure Shells and aware of the various Hostnames for lxplus and the UIUX lx machines.

Logging on to CERN's Computing Environment (lxplus)

"The PLUS service (Public Login User Service) is the interactive logon service to Linux for all CERN users. The cluster LXPLUS consists of public machines provided by the IT Department for interactive work. "
~Public Interactive Logon Service

You will usually use a terminal to access data and applications on CERN's lxplus. Thus, now would be a good time to familiarize yourself with a few basic UNIX commands. If you know nothing about UNIX commands, I will entrust you with only these three for now:

  • cd directory -- This command Changes the current Directory to directory.
  • ls -l -- This command lists all the files in the current directory.
  • man command -- This command tells you all you need to know about most all usable commands.

If you are comfortable with using UNIX commands, here is a UNIX Cheat Sheet for you anyway. If you don't like this one, there are plenty more where it came from. Don't forget that Google is your friend!

Now, lets log on:

  • Now open your terminal and type ssh [username]
  • You should then see:
*                    The LXPLUS Public Login Unix Service                     *
* : Govern the use of CERN computing facilities *
[username]'s password: 
  • After inputting your password and hitting return you should see:
*                                                                                *
* If you cannot access:                                                          * 
*                     - the Indico protected agendas                             *
*                     - the TWiki  protected pages                               *
*                     - the CDS protected collections                            *
*                                                                                *
* Look at the CMS information protection web page:                             *
*   *
*                                                                                *
* If you fulfill the conditions and cannot access the protected pages:           *
*        send a mail to:                        * 
*                                                                                *
*                                                                                *  
[[username]@lxplus###]~% _
  • You are now in your home directory.
  • Type pwd to see it's exact location in the grand scheme of things.
  • Finally, type logout to return to your local terminal session.

Logging on to UVa's Computing Environment (lx Machines)

UVa's lx Machines constitute our local computing facility. Stored on these machines are various kits used as a part of the overall CMS analysis framework. It is also our tiny part of the GRID [cue ominous da da da], an offline distributed computing network. Logging on to these machines require registering with the UVa HEP techs mentioned above.

After registering with the techs:

  • Open your terminal and type ssh [username]
  • Don't forget to change your password using the passwd command after logging on the first time.

Obtaining a GRID Certificate

"The Worldwide LHC Computing Grid (WLCG) is a global collaboration of more than 170 computing centres in 34 countries. The mission of the WLCG project is to build and maintain a data storage and analysis infrastructure for the entire high energy physics community that will use the Large Hadron Collider at CERN."
~ Worldwide LHC Computing Grid

Requesting a Grid Certificate

Approximate Time to Complete: [~1-3 Business Days]

In order to use the Grid you need a Digital Certificate (sometimes called a PKI or X509 certificate) that acts as a passport and says who you are (know as Authentication). You obtain a Digital Certificate from your National Certification Authority (CA). ~Atlas WorkBook

The DOEGrids CA stopped issuing new certificates as of May 2013. The new CA is managed by the Open Science Grid (OSG). The OSG strongly recommends that the users should apply to CERN CA for a certificate. The following steps for obtaining certificates from the CERN CA are alos explained at PersonalCertificate.

  1. Navigate your modern browser (e.g., Firefox) to the CERN Certification Authority.
  2. Click on "New User Certificate".
  3. Sign in with your CERN account.
  4. Complete the identity verification by entering your account password and birth date.
  5. Verify your browser has the CERN Root Certificate installed.
    • Open the Install CERN Root Certificate page found on the CERN Certification Authority home page.
    • Follow the instructions there.
    • Verify CERN root CA certificate installation. In Mozilla go to Firefox menu and click on Preferences (called Options on PC). Then from the Advanced tab, select the Certificates tab and click View Certificates.
    • Select the Authorities tab and verify that the CERN Trusted CA and CERN Root CA certificates are present.
  6. If the CERN Root Certificate is already installed, choose the default Key Strength (High Grade), then click on "Submit".
  7. If successful, on the next page click on "Download this certificate".
  8. A small alert window will appear saying "Your personal certificate has been installed. You should keep a backup copy of this certificate." Click "OK".
  9. Verify that your new CERN personal certificate is installed in your browser.
    • Return in Firefox to the Certificates tab and click View Certificates. Select the Your Certificates tab. You should see the under the "Certificate Name" column a certificate called "CERN Trusted Certification Authority".
  10. Follow the instructions for how to use your certificate with grid-proxy-init. These points may help with the procedure, so please read through them first:
    • If you have existing personal certificates ~/.globus/usercert.pem and ~/.globus/userkey.pem , then you may want to rename those files first.
    • To get info about your *.pem files , try a command like this (for example my old DOEGrids cert):
      openssl x509 -in ~/.globus/usercert.pem -subject -issuer -dates -noout
      subject= /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=feickert/CN=696794/CN=Matthew Feickert
      issuer= /DC=ch/DC=cern/CN=CERN Trusted Certification Authority
      notBefore=Jun 10 16:13:54 2013 GMT
      notAfter=Jun 10 16:13:54 2014 GMT
    • When you use the openssl command to create the ~/.globus/usercert.pem file, it will ask for the "Import Password" ; this is the "Backup password" you chose in the previous step. When you use the openssl command to create the ~/.globus/userkey.pem file, it will ask for the "Import Password" again, and it will also ask you to choose a "PEM pass phrase". The "PEM pass phrase" will need to be typed in every time you issue the grid-proxy-init or voms-proxy-init commands.


  1. First you need to begin using the new OSG CA through the OSG PKI Certificate Request & Management System.
  2. Select "Request New" from the menu on the left of the page.
  3. Fill out the information.
    • Select "CMS" from the Virtual Organization pull down menu.
    • Select "Anthony Tiradani " from the Sponsor pull down menu.

After a few days (typically the same day), you should receive an email confirming approval of your application, which will contain a link for retrieving your certificate.

    • Please follow the provided instructions for downloading and backing up your certificate.

Joining the CMS VO (Virtual Organization)

CMS User Documentation for Specific VOs and Sites

Access to CMS grid resources is granted by joining the CMS VO. Registration consists of two phases. In phase I, a visitor completes the registration form labeled "Registration (Phase I)". Next, the visitor becomes a "candidate" for VO membership. After an initial confirmation of identity, the candidate moves to phase II. In phase II, the candidate completes the registration form labeled "Registration (Phase II)" and confirms intent to comply with the Grid and VO AUPs of the LCG Grid . At this point the candidate becomes an "applicant". Once the applicant is approved, he or she becomes a "member". The VO administrator may grant you administrator rights, as appropriate. Before proceeding:

  • Be sure to use the same browser into which you've imported your certificate, since the registration page will require your certificate in order to authenticate you. *You'll not be able to proceed with these steps until you've received your grid certificate and installed it into your browser.
  • Read the LCG Usage Rules.
  • Install the CERN Trusted Certificate Authority certificate into your browser.

Phase I, Becoming a Candidate

Approximate Time to Complete: [~5-10 Minutes]

To become a candidate for the CMS VO, you must perform the Phase I registration:

  1. Browse to the CMS VOMRS (Virtual Organization Management Registration Service) server.
    • If you haven't installed the CERN Trusted Certificate Authority certificate into your browser, you may need to add an exception for the site to your browser's security settings in order to trust the site's certificate.
    • Note that your personal certificate DN and Certificate Authority are noted in red at the lower left of the window. If they are not, then you will need to install your certificate into your browser before proceeding. (If you followed the directions in the email you received notifying you of your new grid certificate, you should have this.)
  2. On the left-hand side, click Phase I Registration.
  3. Fill out the requested information.
    • You must use your CERN email address.
    • Under Grid job submission rights, select the default value of full.
    • As a US CMS candidate, you should select the following as your VO Representative:
      • For "Select representative" select Anthony Tiradani
    • For "grid job submission rights" select full
  4. Submit the form.

You will receive an e-mail with subject "Automatic Notification from cms VOMRS: You have to confirm e-mail address and finish registration" when your identity has been confirmed.

Phase II, Becoming an Applicant

Approximate Time to Complete: [~1-5 Business Days]

Once your basic identity has been established, you are confirmed as a candidate. To become an applicant, you need to perform Phase II registration. In this phase, you'll select groups and/or roles for which you wish to be authorized within the VO, and formally agree to be bound by the LCG Usage Rules.

  1. Return to the CMS VOMRS (Virtual Organization Management Registration Service) server. You should now see Phase II Registration on the left side bar.
  2. Click Phase II Registration.
  3. Fill out the requested information.
    • Only check the two boxes on the right hand side which correspond to "Group Role = cmsuser" for "Group = /cms" and "Group = /cms/uscms"
    • //You should select /atlas/usatlas and /atlas/lcg1 as groups to join.
    • //*Do not* select any Roles. Only ATLAS/US and ATLAS software managers and system administrators need (or will be permitted) to be authorized for the production and software roles and groups. Those applicants will definitely know who they are. Your application may be delayed if you request groups or roles for which you shouldn't be authorized.
    • //Read and Agree to the LCG Usage Rules.

You will receive an e-mail when your application has been approved.

First order of business as a Member

Within a few days, your application will be reviewed by your Representative and (presumably) approved. You will receive an e-mail confirmation that you are officially a member of the CMS VO. Once you are approved as a member of the CMS VO, you can install your grid certificate and begin using grid software. Note that there may be up to 24 hours of lag between your final approval and the point at which all CMS sites worldwide will recognize your credentials.

Installing Your Grid Certificate

Approximate Time to Complete: [~10-25 Minutes]

This will walk you through the process of installing the Grid Certificate you previously downloaded into your web browser.

Exporting Your Certificate

Before you can use your certificate to perform grid tasks, you'll need to export it from your browser to your home directory.

  1. Select your certificate:
    • In Firefox v3 and v2 (Linux): Click Edit -> Preferences, click Advanced, click the Encryption tab, and then click View Certificates.
    • In Firefox v3 (Mac): Click Firefox -> Preferences, click Advanced, click the Encryption tab, and then click View Certificates.
    • In Firefox v2 (Windows): Click Tools -> Options, click Advanced, click Encryption, and then click View Certificates.
    • In Firefox v1.5 (Linux): Click Edit -> Preferences, click Advanced, click Security, and then click View Certificates.
    • In Firefox v1.5 (Windows): Click Tools -> Options, click Advanced, click Security, and then click View Certificates.
    • In Mozilla v1.7 or Seamonkey v1 (Linux): Click Edit -> Preferences, expand Privacy & Security, click Certificates, and then click Manage Certificates.
    • In Internet Explorer v7: Click Tools -> Internet Options, click the Content tab, and under Certificates, click the Certificates button.
    • In Internet Explorer v6: Click Tools -> Internet Options, click the Content tab, and under Certificates, click Personal.
  2. Select your new certificate, click Backup (or Export), and save this file to a safe location on your computer or in your directory.

If the computer you are using is shared with other users, be sure to remove this copy of the file when you're finished. Apply a good password to the backup when you are given the option, and be sure to remember this password. If your browser supports the option to use a password mechanism for its certificate storage, be sure to use that option.

Converting Your Certificate

To convert the new certificate for use in grid jobs:

1. Use the openssl pkcs12 command to convert the certificate and its key:

openssl pkcs12 -in [your-cert-file] -clcerts -nokeys -out [path]/usercert.pem
openssl pkcs12 -in [your-cert-file] -nocerts -out [path]/userkey.pem

where [your-cert-file] is the name and path of your exported certificate file, and [path]/[filename].pem is the name and path for the certificate and key file to be generated in an existing directory (US ATLAS users will want to use the path $HOME/.globus/[filename].pem).

  • In response to each command, you will be prompted for two passwords:
    • Enter Import Password: This is the password you created when you exported your certificate from your browser.
    • Enter PEM pass phrase: This is the optional Challenge Phrase Password you created when you first requested your certificate from DOEgrids.

2. Change permissions to protect the converted key file:

  • In Linux/UNIX/Mac:
    chmod 600 userkey.pem
  • In Windows:
    1. Right-click the file userkey.pem, and choose Properties
    2. Change the Permissions settings so that you have Read and Write permissions, and that no permissions at all are selected for Group and World.

Setup Your Shiny New Grid Environment (Setup Check)

As a US CMS user, you will likely use your grid certificate to copy data files or to submit jobs over the grid. We will now check to see if you have successfully followed the instructions for obtaining and installing your Certificate! We shall use the lxplus machines and an automated script to set the proper Environment Variables.

To source the script from CERN, from a session on, enter:

source /afs/ [sh,csh]

* where the .sh extension should be used in Bourne and BASH shell environments, and the .csh extension should be used in C shell and Tcsh environments.

To verify the environment:

  • After sourcing the script, verify your certificate installation with the command:

voms-proxy-init -voms atlas

  • This should return with a message similar to the following

Your identity: /DC=org/DC=doegrids/OU=People/CN=John Smith 123456
Creating temporary proxy ..................................................................... Done
Contacting [/DC=ch/DC=cern/OU=computers/] "atlas" Done
Creating proxy .................................. Done=
Your proxy is valid until <date>

You will be prompted to enter your pass phrase for this certificate. Once you enter the pass phrase, a grid proxy will be created for you.

Additional Guides for the Grid can be found here and here

Comments & Update Suggestions

Thank you for taking time to provide feedback!
  • Reminder that the VPN address has changed!!! Ref. the Email -- AustinBasye - 10 Sep 2010


  1. First you need to go to our Certification Authority which just so happens to be the RCF facility at BNL.
  2. Once you have loaded your browser with a DOEgrid CA Certificate Chain. you must actually request a Grid Certificate.
    • Using the same computer and browser you used to load the chain, go to the DOEgrids Subscriber Enrollment site.
      • Warning, important If your browser reports a security warning and asks you if you want to trust the site, you have not correctly obtained the DOEgrid CA Certificate Chain. In which case you must repeat step 1.
    • On the left, under Subscriber, click New User.
    • Complete the form using the following information:
      For US CMS users:
      Affiliation: OSG
      VO Name: CMS
      Sponsor Information: Include the name, e-mail, address, and phone number of your PI/Advisor
    • Although the Challenge Phrase Password field is optional, it is strongly recommended that you create and remember a challenge phrase for your certificate in order to protect your grid identity.
    • Under Public/Private Key Information: For *Firefox and Safari: it is recommended that choose the highest Key Length value available for your browser, again to help protect your certificate and your grid identity.
      • For Internet Explorer: for Cryptographic Provider, choose Microsoft Enhanced Cryptographic Provider v1.0.
    • Click Submit.

-- MatthewFeickert - 01-Jul-2013
Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r5 - 2013-07-11 - MatthewFeickert
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback