1.3 Your Computing Environment

Complete: 5
Detailed Review status

Goals of this page:

After following the instructions on this page you should have your working environment ready to run tutorials and examples.

Contents

Introduction

The tutorials and examples in the workbook may be done on the CERN UNIX cluster LXPLUS or on any other machine or cluster that has CMSSW installed. The instructions on this page assume that you already have a CERN account; if not, see WorkBookGetAccount. Grid computing comes a little later; see WorkBookStartingGrid for information on that.

CMSSW Software Installed at a Generic Site

CMSSW does not need to be installed locally at every site. Users can access it via the LXPLUS cluster at CERN or at some of the T1 centers. For sites that wish to install it locally, installation instructions are beyond the scope of this workbook, and can be found on this page Installing software using apt.

If you are using a local installation, you may need to use slight variations of the standard instructions in this workbook for some tasks. Any variations should be documented in the workbook page Specific Information for Remote Sites and Institutions. If not, please contact your system admin and ask that the necessary information be added or links provided.

Log in to the CERN UNIX Cluster

We'll assume you want to run CMSSW on the the LXPLUS cluster at CERN. AFS is installed on this cluster, so your home area is your AFS home area in cern.ch. Instructions for other remote systems may be found at Specific Information for Remote Sites and Institutions.

If you are connecting remotely, you'll need an X connection to view output.

To log on, use ssh:

ssh <username>@lxplus.cern.ch

You should automatically obtain your CERN kerberos credentials; if not, use the AFS kinit program (in /usr/sue/bin) with your CERN AFS password to authenticate to the CERN Kerberos 5 server.

Log in at remote sites

Instructions for other login platforms can be found in Specific Information for Remote Sites and Institutions.

Create a work area and build the application

We will be using CMSSW (currently at version CMSSW_9_2_3) as the application and SCRAMv1 as the build tool. If you are not familiar with SCRAMv1, you might want to look at WorkBookScramV1Intro.

Build the work area

First execute the following commands to access CMSSW repository:

setenv VO_CMS_SW_DIR /cvmfs/cms.cern.ch

source $VO_CMS_SW_DIR/cmsset_default.csh

Then verify that CMSSW release CMSSW_9_2_3 is installed on your system.
scram list -a | grep CMSSW_9_2_3

If installed, it will return a pointer of the form /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_9_2_3 or /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_2_3 depending upon whether the SCRAM_ARCH is set to slc6_amd64_gcc530 or slc6_amd64_gcc630.

Some common errors:

  • If you get the error message scram: command not found on lxplus, it is likely that you are not registered as a CMS user. Follow the instructions on how to get a CMS account so that some settings are done automatically when you login.

To build the work area, go to your work directory (often ~/scratch1) and type

cmsrel CMSSW_9_2_3

This is equivalent to the full command scramv1 project CMSSW CMSSW_9_2_3

You will now have your developer area under a new CMSSW_9_2_3 directory. For instance, at CERN, the developer areas look like the following:

/afs/cern.ch/user/n/nitish/scratch1/CMSSW_9_2_3

Setup your runtime environment

You must setup your runtime environment every time you start work in your project area.

cd CMSSW_9_2_3/src

To setup the runtime variable do

cmsenv

This is equivalent to the longer command, for a csh varient shell (csh, tcsh)

eval `scramv1 runtime -csh`

or if your shell is a sh varient (bash, zsh)

eval `scramv1 runtime -sh`

Set up GitHub

CMSSW code that you will need to access is maintained in a GitHub repository. We describe how to use this in GitHub.

Set up ROOT

To learn about ROOT, see ROOT Basics or the ROOT homepage. ROOT automatically loads all the necessary CMS-specific libraries (shell prompt shown for clarity):

$ root -l              #the -l, lower case L, is optional, and if used, omits the logo

SSH, X11 forwarding and opening windows to remote machines: Windows

Often you'll need to open xwindows applications from a PC - e.g. to run the CMS event display, or more regularly to run a ROOT session. Both these tasks tend to require you to connect from a Windows or Mac PC to a linux machine (generally using ssh), and then to have windows from the ssh session pop up to display graphics.

The general process for this: start an ssh connection from your windows/mac machine into a linux system(e.g. CERN lxplus) and then start the xapplication. Note that you have to enable the X11 connection (in PuTTY, select Connection -> SSH -> X11 in the left bar and check the corresponding box). To enable the graphics windows to appear, you'll need to have an X11 server running on your PC, and also to have an ssh client which supports X11 forwarding. It's not that frightening...

To do windows X11 forwarding, you need an X Server for Windows (e.g. Cygwin/X, xwin32, Xmanager, etc.) and an SSH client that supports X11 forwarding.

Some good SSH clients are:

  • PuTTY is an SSH client and WinSCP is a companion SCP program for secure file transfer in Windows. Both are available from CERN (the links will point you to the correct folders) where you will log in using your CERN NICE account.
  • Secure Shell (SSH) is a simple, easy-to-use SSH client that comes with SFTP so you don't have to download a separate program for secure file transfer in Windows.
There are various choices of X Servers for Windows:
  • Xming server. Small, fast and easy. If you do not want to install the Font Package, you can run XLaunch (part of Xming) to set the font server: xtsoft1.cern.ch
  • Cygwin/X11 server has a good integration with the Windows desktop.
  • Hummingbird Exceed is a commercial product used by CERN, and it's available for installation on NICE computers.
  • Starnet xwin32 is another commercial product used by many US university groups. While it can be used for 30 days on a free trial, more prolonged use requires a license.
  • Similarly, Xmanager has a free 30 days trial period, and then costs about US$100 to keep using.
For a simple, free and fast setup, all you need are an SSH client and Xming to log into a Linux machine from a Windows desktop. Remember to set SSH to allow X11 tunnelling and you can work straightaway.

VNC

Often the linux machine you want to run applications such as ROOT on is far from the PC you are using - e.g working in the US using a CERN lxplus machine. Then X11 is often a bad choice for graphics, as X11 was not designed to work over high latency connections (hence can get very slow responses). Using VNC over ssh rather than X11 can avoid the problem of remote access. However, the default Windows Manager twm that you get with VNC is difficult to use. If you start up Gnome or KDE when you log in through VNC, then the connection becomes slow, which defeats the purpose of using VNC in the first place. If you are only running programs on a remote computer that require little user interaction, then VNC is fine. If you are doing a lot of interactive work, then VNC is not suitable. Note that the VNC protocol is not encrypted, so you must set up a secured connection through SSH. Also, please read this security message by CERN IT.

Remote Desktop

Perhaps the best solution for a user across the ocean to work on a CERN lxplus machine is to connect to cernts (CERN Windows Terminal Services) either through Remote Desktop Connection (for Windows and Mac OS), or through rdesktop (for Linux OS). From the web site, register to use the service and follow the instructions to connect to cernts. Once connected, go to the Start menu and look for the program Hummingbird Connectivity 2007 → CERN Predefined Connections for TS → lxplus (SLC5). Running emacs will be almost as fast as running it from a local machine.

Macintosh

Sierra (10.12) and High Sierra (10.13) are stricter about X forwarding compared to older Mac systems. Use:

ssh -Y <username>@lxplus.cern.ch

This option enables "trusted" X11 forwarding and will allow you to open ROOT and the TBrowser without any problem.

Get CERN Kerberos ticket

If you are onsite at CERN, you should get your Kerberos ticket normally. When you login on LXPLUS you get a ticket which is valid for 25 hours, if you stay logged in for a longer time, you can renew it with command

kinit

and typing your LXPLUS (afs) password.

If you are running jobs from an offsite location, but accessing files or resources at CERN, you will need to obtain CERN Kerberos credentials. To get valid CERN Kerberos credentials, run:

kinit @CERN.CH

It will ask for your LXPLUS login username and password. You can safely transmit them over the network.

Working with large files

The storage of the output data is explained in page WorkBookUsingT2Resources#StoreDataOutput.

Locally at CERN, if your personal working area on CERN's LXPLUS cluster isn't big enough to handle your output files, you'll need to write them to a larger-capacity area. To ask for "AFS workspace" (up to 100 GB, backed up), login to the Cern account web page and go to "Applications and Resources", take the "Manage"-link next to "Linux and AFS". There you can ask for "workspace" in AFS, as well as extend the quota for your backed-up $home (up to 10 GB). Please note the different AFS path to your workspace: /afs/cern.ch/work/initial/username where initial is the first letter of your username, i.e. the workspace is not hanging from your $home. For other sites, see WorkBookRemoteSiteSpecifics.

Working with several large files

The area AFS "workspace" is usually not enough for running or storing output of several jobs at the same time. In such a case CERNBox/EOS area prove to be extremely useful. The conventional use of CASTOR is now discouraged.

Using CERNBox/EOS

CERNBox provides cloud data storage to all CERN users. You can store your data, share it and synchronise it across devices - smartphones, tablets, laptops, desktops, the lot! One has to register to CERNBox by connecting to website: https://cernbox.cern.ch/. The data can be accessed from any Web browser or file explorer, and you decide which data you want to share with other individuals or groups of collaborators. Data can be synchronised to your devices (Windows, Mac, Linux, iOS and Android) with the CERNBox application.

Accessing your CERNBox files stored in EOS on lxplus

CERNBox files are stored in the EOSUSER instance. To access your files from lxplus you can use standard unix commands: e.g. ls /eos/user/initial/username

In order to transfer files, you should set up your environment:

Set up the variable EOS_MGM_URL:

export EOS_MGM_URL=root://eosuser.cern.ch (for bash) setenv EOS_MGM_URL root://eosuser.cern.ch (for tcsh)

The XRootD/EOS entry point (instance server) for your files in CERNbox is root://eosuser.cern.ch and your user directories are under /eos/user/initial/username.

NOTE: if you are using EOS for both CERNBox and your experimental/analysis work, be sure to verify which instance of EOS you are querying when running EOS commands. One way to force an EOS command to query a specific instance is to specify the server for that instance:

eos (instance server) (command)

E.g. to list xyz's files in CERNBox: eos root://eosuser.cern.ch ls /eos/user/x/xyz and to query EOSCMS: eos root://eoscms.cern.ch (command).

More details can be found at: CERN EOS Twiki.

Using Castor (Not recommended)

Another method (which is now discouraged) is to store on CASTOR. The CASTOR Project stands for CERN Advanced STORage Manager, and its goal is to handle LHC data in a fully distributed environment.

Every user with CERN account has a directory in CASTOR. To list the files in your directory, type

rfdir /castor/cern.ch/user/initial/username/

where initial is the first letter of your user name. You can create directories with

rfmkdir /castor/cern.ch/user/initial/username/dir

and you can copy files to a castor directory with

rfcp filename /castor/cern.ch/user/initial/username/dir

The output file can be saved to CASTOR by using the following script called runMyJob.(c)sh

#!/bin/(c)sh
cd /afs/cern.ch/user/initial/username/scratch1/CMSSW_9_2_3/src/.../.../test
eval `scramv1 runtime -(c)sh`
cd -
cmsRun /afs/cern.ch/user/initial/username/scratch1/CMSSW_9_2_3/src/../../test/aConfigFile.cfg
rfcp outputFile.root /castor/cern.ch/user/initial/username/largeOutputFile.root

The batch job starts to run in a directory called /pool/lsf/username/jobnumber, and this directory has a large scratch space that is available for the duration of the job. You then cd to scratch1/CMSSW_9_2_3/src/.../.../test to set the environment variables. The command "cd -" takes you back to /pool/lsf/username/jobnumber so that the large scratch space is available. aConfigFile.cfg is run from that directory, and the last line copies the output root file from there directly to your space in CASTOR. Note that while writing directly to a file in CASTOR is possible, it is not recommended, as it is very error-prone. The recommended practice is the one indicated above, i.e. writing to a local area and copying the file after cmsRun has finished.

You submit the job in the same way as that for smaller jobs.

bsub -q 1nd -J job1 < runMyjob.(c)sh

It is also possible to request a minimum amount of space in /pool/lsf/username/jobnumber using the -R option, for example to request a minimum of 30GB type

bsub -R "pool>30000" -q 1nd -J job1 < runMyjob.(c)sh

Opening files from CASTOR (Not recommended)

To open a file in an interactive root session, type

TFile *f  = TFile::Open("rfio:/castor/cern.ch/user/initial/username/myFile.root")

If you want to use a file saved in CASTOR as input to a CMSSW job, add the following to the .cfg file

source = PoolSource{ untracked vstring fileNames = {'rfio:/castor/cern.ch/user/initial/username/inputFile.root'} }

See the lxbatch introduction pages: http://information-technology.web.cern.ch/services/batch and the CASTOR web page for further information. Some advice in case of trouble with data access is given in the Troubleshooting guide.

UserCode area in GitHub

With the move from cvs to git, there is no longer any central UserCode area. Instead, users are encouraged to create their own github repository to store and share their own personal code. You can put any code you want in your own repository.

If instead you want to put code in the central cmssw repository, the process is more complicated. For instructions on developing official cmssw code, see the tutorial.

SUMMARY OF STEPS FOR EVERY LOGIN SESSION

Set up CMS Environment:
If you're using the LXPLUS cluster at CERN, the CMS environment is set up by default at login time.

For remote systems you'll need to set up the CMS environment explicitly. This is site-specific. To find out how to setup the CMS environment other remote sites; see Specific Information for Remote Sites and Institutions or your sysadmin.

ONLY if not done previously, create your work area (do this only once per CMSSW version, not every login session):

cmsrel CMSSW_9_2_3

Set your runtime environment:
cd CMSSW_9_2_3/src
cmsenv

Review status

Reviewer/Editor and Date (copy from screen) Comments
NitishDhingra - 12 Nov. 2017 Added information on CERNBox/EOS.
NitishDhingra - 29 Oct. 2017 Updated to CMSSW_9_2_3, removed outdated info, fixed broken links etc.
XuanChen - 30 May 2014 added command of setting an environmental variable, replaced the link of SSH, changed the part of usercode and replaced CVS by github
JohnStupak - 4 June 2013 revised and updated to CMSSW_5_3_5
AndreasPfeiffer - 15 Dec 2009 update login part (kerberos 5 now) and Mac part (all OS versions above 10.4)
KatiLassilaPerini - 08 Jul 2008 replaced scram commands with cmsrel and cmsenv, removed project CMSSW
AndreasPfeiffer - 11 Feb 2008 update statment on (obsolete) slc3
AndreaBocci - 10 Jan 2008 reflect change in Xming licensing terms
AndreaBocci - 26 Jun 2007 different definitions of free
MarkoDragicevic - 11 Jan 2007 added Xming to suggested X servers
KatiLassilaPerini - 29 Sep 2006 added more CASTOR from Nick Grant
KatiLassilaPerini - 10 Oct 2006 added UserCode in CVS
JennyWilliams - 29 Nov 2006 added notes about X11 forwarding

I went through chapter 1 section 3, the information is relevant and clear, this page has several links out off date.

Several links require just to update the link (email from morelos to Kati).

For the four links at the bottom of this comment, I could not relate a link.

best regards, Antonio Morelos


I would suggest to update the link at " Installing software using apt "

I would suggest to update the link at " PuTTY "

I would suggest to update the link at " WinSCP "

I would suggest to update the link at " Xmanager "

I would suggest to update the link at " secured connection through SSH "

I would suggest to update the link at " security message "

I would suggest to update the link at " CERN Windows Terminal Services "


The link at " Secure Shell (SSH) " It is sent to a "Not Found" page. I could not find a clear option on what to suggest concerning this link. I guess the intention is to click-download the exe file ?

The link at " SnowLeopard " The link goes to the twiki editor. I could not find a clear option on what to suggest concerning this link. Is the intention that the explanation text is missing ? or is the intention to sent the user to the SnowLeopard MacIntosh Page ?

The link at " http://batch.web.cern.ch/batch/doc/quick-intro.html " Which confuses myself on what should I see on that page, related to " See the lxbatch introduction pages: "

The link at " UserCode/README " which confuses myself on what sould I see on that page, this page does not look like an instruction page.


Updated the straightforward links that A.Morelos suggested.

Responsible: SudhirMalik
Last reviewed by: AndreasPfeiffer - 11 Feb 2008

Topic attachments
I Attachment History Action Size Date Who Comment
Microsoft Executable fileexe SSHSecureShellClient-3.2.9.exe r1 manage 5388.0 K 2009-05-21 - 10:38 ChungKhimLae Secure Shell (SSH) program
Texttxt cmscvsroot.sh.txt r1 manage 0.5 K 2007-01-21 - 22:19 JennyWilliams Script for setting up cvsroot path
Texttxt projch.sh.txt r1 manage 0.6 K 2007-01-21 - 22:20 JennyWilliams "project" script for setting up cmssw environment
Edit | Attach | Watch | Print version | History: r138 < r137 < r136 < r135 < r134 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r138 - 2018-09-27 - DanielSunyouLi


ESSENTIALS

ADVANCED TOPICS


 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback