1.4 CMS Computing Concepts: Programming, GitHub, etc

Complete:
Detailed Review status

Goals of this page

You will learn what programming language(s) you are expected to know, what software packages you should be familiar with, and how to access software from the code repository.

Note that in 2013, the CMS software repository was moved from CVS to git. CMS no longer uses CVS. The new git software repository is hosted at a website called GitHub, which provides extra features beyond git's file version control.

Contents

• Operating Systems
• Programming Languages
• Analysis Packages
• GitHub
• Git Documentation
• Git Tutorials and Info
• CMS Releases
• Review status

Operating Systems

Familiarity with Linux is necessary. Current versions of the CMS Offline software, CMSSW, run on Scientific Linux CERN 6 (SLC6) (which is built on Scientific Linux 6 (SL6), which is in turn built on Red Hat Enterprise Linux 6 (RHEL6)). Run I (2010-2012) was based on SL*5. The upgrading process, from SL*5 to SL*6, started in 2014 with the 7_0_0 release. Currently for Run II analyses the release being used is CMSSW_9_4_2.

Programming Languages

CMSSW code is written in C++. You will need to be able to write compatible code in C++. There are many good text books available. See Walter Brown's (of FNAL) book recommendations for C++ Programmers .

Analysis Packages

You will need to know how to use ROOT. It is an object-oriented data analysis framework.

GitHub

CMS software, including CMSSW, is now maintained in a GitHub software repository. You'll need to use GitHub to check out code and configuration files to your area so that you can modify and use them. There are lots of working examples in GitHub for you to try; various workbook tutorials point you to specific directories. Since CMSSW is such a large project we use the sparse check-out features of git. Because of this it is highly recommended that you look at the CMS git tutorials to understand how this is done.

Here we give instructions on checking out code from GitHub. When you are working in the CMS environment, you can run any of the CMS code in the release without checking it out. You only need to "check out" code that you need to modify, code that has been modified by someone else, or code that depends on modified code. Then that code can be rebuilt.

A "release" is a set of software which has a version number and has been thoroughly tested to all work together. The CMS code evolves with stable, working releases numbered and advertised from time to time as a recommended set of code to use.

Git Documentation

The primary documentation for usage of git in CMS is at the following locations:

• http://cms-sw.github.io/cmssw/index.html
  • http://cms-sw.github.io/cmssw/tutorial.html
  • http://cms-sw.github.io/cmssw/faq.html
  • http://cms-sw.github.io/cmssw/advanced-usage.html

The information located there is much more detailed and complete as compared to the information listed below.

(Warning: Early in the introduction of git into CMS, CMS created a thing called the Topic Collector which some documentation still references. The Topic Collector is no longer used for any purpose. GitHub directly offers features that replace that functionality and now we use them. If you see references or links to the Topic Collector, you should understand you are looking at obsolete documentation that needs to be updated.)

Obtaining a personal GitHub account:

If you want to modify code and then have your modifications merged into the official repository of CMSSW code, you will need a personal GitHub account. You would first edit the code and commit it to a git repository on the machine where you work (maybe the machine on your desk or lxplus). Then you would push the changes to the repository in your personal GitHub account. Then you would issue a pull request to have the modifications pulled from your personal GitHub repository to the official CMSSW GitHub repository by a release manager.

A personal GitHub account has other uses. It is also a useful way to send your code modifications to other people or groups. It can also be used as a way to backup code modifications that you make and maintain version control of your own personal code.

But a personal GitHub account is only mandatory if you want to merge code into the official CMSSW code repository. You can download code, build it, and run it locally without a personal GitHub account.

First you have to obtain a GitHub account. Go to github.com and follow the instructions to open a new account if you do not already have one.

Then fork the official CMSSW repository. That creates a copy of it in your personal GitHub account. Fork by going to https://github.com/cms-sw/cmssw and click on the fork button and follow the instructions. After the forking is done you see you have a new repository cmssw, https://github.com/<your-username>/cmssw.

Setup personal information

• Login on lxplus at CERN or wherever you plan to do your code development work.
• Run the following commands:
  • git config --global user.name "First name Last name"
  • git config --global user.email <Your-Email-Address>
  • git config --global user.github <Your-Just-Created-GitHub-Account-Username>

Easy access of GitHub (optional)

• If you have no ssh key: Follow generating-ssh-keys on GitHub to create a ssh key.
• If you have already an ssh key: Login to GitHub -> settings -> SSH keys (https://github.com/settings/ssh) and add your key. I.e. copy paste the content of yourKey.pub
• To use your key you have to execute following commands after each login:
  • eval $(ssh-agent); ssh-add /path/to/your/key
• Test your configuration: ssh -T git@github.com
  • You should see: Hi XYZ! You've successfully authenticated, but GitHub does not provide shell access.

Checkout a CMSSW package from Git

• Setup a CMSSW environment

cmsrel CMSSW_9_4_2
cd CMSSW_9_4_2/src
cmsenv

• Checkout a package, e.g. PhysicsTools/PatExamples

git cms-addpkg PhysicsTools/PatExamples

No release tags specified, using default CMSSW_9_4_2.
...
You are on branch from-CMSSW_9_4_2
Checking out packages
PhysicsTools/PatExamples

• Create your own development branch e.g. mydev

git checkout -b mydev

Switched to a new branch 'mydev'

• Check which branches are available

git branch

  CMSSW_9_4_X
  from-CMSSW_9_4_2
* mydev

• Apply modifications

echo '#modfication' >> PhysicsTools/PatExamples/BuildFile.xml #this adds a line containing #modification at the end of BuildFile.xml

• Check if modification is recognized

git status

# On branch mydev
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   PhysicsTools/PatExamples/BuildFile.xml
#
no changes added to commit (use "git add" and/or "git commit -a")

• Add and commit your changes

git add PhysicsTools/PatExamples/BuildFile.xml
git commit -m "test message"

[mydev 6812826] test message
 1 files changed, 1 insertions(+), 0 deletions(-)

Your modifications were commited to your local git repository. You can push these changes to your personal GitHub repository with the following command.

git push my-cmssw mydev

Before you do this you will have to have already created your personal GitHub account and forked the cmssw repository. (Creating and forking are one time actions. They do not need to be done with every push, see above for more information).

If obtain this message:

ERROR: Repository not found.
fatal: The remote end hung up unexpectedly

git wasn't able to find the my-cmssw repository in you GitHub account. Check if you forked the https://github.com/cms-sw/cmssw repository. If the push is successful you see:

Counting objects: 9, done.
Delta compression using up to 16 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 436 bytes, done.
Total 5 (delta 4), reused 0 (delta 0)
To git@github.com:fhoehle/cmssw.git
 * [new branch]      mydev -> mydev

You visit your account at GitHub and click on the branches drop-down menu you see your new branch mydev. Inspect the PhysicsTools folder and you see that PatExamples were changed recently with a message test message.

It showed you are checkout done of specific packages and how you are able to save your local changes at your private GitHub account. Git has a lot more feature which help and support you in developing and keeping track of your code. Please visit GitHub.com for more detailed information on branches, merging them, tags, etc.

Git Tutorials and Info

Tutorials

• https://indico.cern.ch/conferenceDisplay.py?confId=254365
• https://indico.cern.ch/conferenceDisplay.py?confId=254369
• https://indico.cern.ch/conferenceDisplay.py?confId=286055
• https://indico.cern.ch/conferenceDisplay.py?confId=286056
• https://indico.cern.ch/event/917661/ (May 2020)

CMSSW releases

To find out which CMSSW releases are available, do:

scram list CMSSW

You'll see a list with names like CMSSW_x_y_z, some with other suffixes. The names point to directories, e.g.,

CMSSW CMSSW_9_4_2 -->  /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_2

For each listed directory, subsystems are under its src directory, i.e., under /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_2/src . As an example, let's list the subsystems under this directory:

ls /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_2/src

Alignment            DQMOffline                  HLTriggerOffline     OnlineDB           RecoLuminosity          SimG4Core
AnalysisAlgos        DQMServices                 HeavyFlavorAnalysis  PerfTools          RecoMET                 SimGeneral
AnalysisDataFormats  DataFormats                 HeavyIonsAnalysis    PhysicsTools       RecoMuon                SimMuon
BigProducts          DetectorDescription         HiggsAnalysis        QCDAnalysis        RecoParticleFlow        SimRomanPot
CalibCalorimetry     DiffractiveForwardAnalysis  IOMC                 README             RecoPixelVertexing      SimTracker
CalibFormats         Documentation               IOPool               RecoBTag           RecoRomanPot            SimTransport
CalibMuon            EgammaAnalysis              IORawData            RecoBTau           RecoTBCalo              TBDataFormats
CalibTracker         ElectroWeakAnalysis         IgTools              RecoCTPPS          RecoTauTag              TauAnalysis
Calibration          EventFilter                 JetMETAnalysis       RecoCaloTools      RecoTracker             TopQuarkAnalysis
CaloOnlineTools      FWCore                      JetMETCorrections    RecoEcal           RecoVZero               TrackPropagation
CommonTools          FastSimDataFormats          L1Trigger            RecoEgamma         RecoVertex              TrackingTools
CondCore             FastSimulation              L1TriggerConfig      RecoHI             SLHCUpgradeSimulations  Utilities
CondFormats          Fireworks                   L1TriggerOffline     RecoJets           SUSYBSMAnalysis         Validation
CondTools            GeneratorInterface          MagneticField        RecoLocalCalo      SimCalorimetry
Configuration        Geometry                    Mixing               RecoLocalFastTime  SimDataFormats
DPGAnalysis          GeometryReaders             MuonAnalysis         RecoLocalMuon      SimFastTiming
DQM                  HLTrigger                   NOTICE               RecoLocalTracker   SimG4CMS

Each subsystem contains several packages, for example

ls /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_2/src/Alignment/

APEEstimation     CocoaFit         CommonAlignmentAlgorithm        HIPAlignmentAlgorithm        MuonAlignment            SurveyAnalysis
CTPPS             CocoaModel       CommonAlignmentMonitor          LaserAlignment               MuonAlignmentAlgorithms  TrackerAlignment
CocoaApplication  CocoaToDDL       CommonAlignmentParametrization  LaserAlignmentSimulation     OfflineValidation        TwoBodyDecay
CocoaDDLObjects   CocoaUtilities   CommonAlignmentProducer         LaserDQM                     RPData
CocoaDaq          CommonAlignment  Geners                          MillePedeAlignmentAlgorithm  ReferenceTrajectories

with directories such as interface (containing the header files), src (containing the *.cc files), plugins (defining the modules to be used), python (containing the configuration files), doc (containing files needed for reference manual documenation), test (containing test setup) and data (obsolete).

Review status

Reviewer/Editor and Date (copy from screen)	Comments
NitishDhingra - 21 Jun 2018	Updated to CMSSW_9_4_2
AndreasPfeiffer - 07 Dec 2015	updated OS version to SLC6 and CMSSW_7_4_15
AndreasPfeiffer - 05 Apr 2011	updated OS version to SLC5
KatiLassilaPerini - 08 Jul 2008	added addpkg
AndreasPfeiffer - 12 Feb 2008	reviewed page, added info on how CVSROOT will change in 2008
LucasTaylor - 27 Nov 2006	reviewed page and made modest changes
KatiLassilaPerini - 18 Oct 2006	changes modules -> subsystems, packages, directories

Responsible: SudhirMalik