LHCb With Eclipse: Basics

This hands on tutorial should be followed by all newcomers to the LHCb software environment. It covers the following topics:

Prerequisites

Before doing these exercises, you should look at the slides of the "Overview of LHCb applications and software environment" [.ppt]. Please feel free to update these slides if you modify them for a future tutorial session.

The instructions for these exercises assume that they are being executed on the CERN Linux interactive cluster lxplus. However, except for some details of the interactive environment, the exercises should work on any machine where the LHCb software is installed.

To get an account on lxplus, please contact the LHCb secretariat. If you already have an lxplus account, but were previously working in another experiment, you should make sure that your account is registered as an LHCb account, in afs group z5 (see Exercise 1).

These instructions have last been checked against the DaVinci v29r0 (and Gaudi v22r4) environment. Please use this version of DaVinci or a more recent version.

Remote console on lxplus

For this tutorial, we suggest to use the remote console on lxplus as described in RemoteLxplusConsoleHowTo.

It is also possible to use plain ssh connections to lxplus. Open a connection using the command:

The -X option is needed to enable X11 connections when logging in to lxplus as a remote machine, to be able to open graphics windows and windows for the emacs editor. Alternatively, omit the -X option and include the following line in the ssh configuration file, ${HOME}/.ssh/config

:

Some sites require trusted X11 forwarding, in this case it is not enough to have -X option, the option -Y is needed or the following line in ${HOME}/.ssh/config:

Choice of shell

By default, all new accounts for on lxplus are configured to use bash shell. LHCb users are encouraged to change this to the tcsh shell (which was the lxplus default until 15th March 2011), an enhanced version of csh, however LHCb setup scripts are also available for the Bourne family of shells: sh, bash, ksh, zsh. There are subtle and not-so-subtle differences in the functionality of the different shells, and for scripting there are major differences between (t)csh and Bourne-type shells. A useful short summary can be found in:

• A.Taddei, Shell Choice: a shell comparison CERN report CN/DCI/162 (1994)

A user may redefine the default shell on lxplus by going to https://account.cern.ch/account/Management/MyAccounts.aspx, navigate to "Applications and Resources.." then [Manage] Linux services (you will need to login using your CERN username and password). In general, unless you have a strong preference, stick to tcsh as this is better tested.

In what follows, <script>.[c]sh is used as shorthand for:

• csh,tcsh: <script>.csh
• sh,bash,ksh,zsh: <script>.sh

LHCb software environment

The LHCb software environment is set up by sourcing the script LbLogin.[c]sh. For local installations the script resides in the directory defined by the environment variable $MYSITEROOT. For an installation using AFS , such as lxplus, it resides in

• /afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/scripts/LbLogin.[c]sh

LbLogin sets a number of environment variables relevant for working with LHCb software, including setting default PATH, LD_LIBRARY_PATH and PYTHONPATH variables. It also sources CMT.[c]sh, which performs the setup for the Configuration Management Tool (CMT), used in LHCb for building software and for environment setup.

For users logging on to lxplus with an account in the LHCb group ( z5), the LbLogin script is sourced automatically at startup by one of the group login scripts, which are all equivalent:

• /afs/cern.ch/group/z5/group_login.[c,z]sh
• /afs/cern.ch/group/z5/group_[ba,tc,z]rc

Which script is executed depends on the type of shell, and on how the shell was created (e.g. ssh or spawned from an existing shell).

LHCb Eclipse

The LHCb environment comes with a pre-installed, shared version of Eclipse. Since this IDE is pretty consuming on the display, you may experience some issues if you're running outside CERN. If you are having a slow connection, please follow the instructions at RemoteLxplusConsoleHowTo to set up the display through VNC.

You may also use your own private installation of Eclipse. To install the required plugins, you should follow this procedure.

Exercise 1

This is a simple exercise just to check your environment setup.

• Start the remote connection to lxplus (VNC or plain ssh)
• Check that X11 applications are working correctly, for example by starting the standard X11 test application:

  xlogo

• Check the group associated with your account by giving the command:

  echo $GROUP

  For an account in the LHCb group the result should be z5.
• Check which shell you are using, for example using:

  echo $0

  or

  echo $SHELL

• Check that the standard LHCb environment has been set. If it has, then you will have seen a message at login time about your CMT settings, and the result of the command:

  env

  should contain a number of variables with prefix LHCB ( LHCBHOME, LHCBRELEASES and so on), and with prefix CMT ( CMTROOT, CMTCONFIG and so on). If this isn't the case then you need to setup the environment yourself.
• Check that you can start eclipse:

  eclipse &

Organisation of LHCb software

LHCb software is based on a framework called Gaudi. This framework provides functionality useful in a wide range of contexts, such as file access, histogramming, message printing, and run-time configuration. The run-time configuration is based on job options.

Software in LHCb is developed in the context of CMT packages and CMT projects.

• A package is a set of files, organised according to some directory structure, which provides some well-defined, circumscribed functionality. It is the basic unit of software development, meaning that all the files in a given package are tagged with the same version number and are released together. Changing just one file in the package implies releasing a new version of the whole package. Most LHCb packages contain code that can be compiled into one or more libraries.
• Packages with related functionality are collected in groups (or are said to be placed under a hat). The purpose of package groups is simply to group together, in the directory tree, all packages that are related in some way (for example all event model packages under the /Event hat, or all Rich packages under the /Rich hat).
• A project is a set of packages that are grouped together according to some functionality, for example all Gaudi packages ( Gaudi project), most public LHCb header files and base class libraries ( LHCb project), all reconstruction packages ( Rec project), all packages specific to the simulation application ( Gauss project). All packages within a given project are released as a single unit: changing just one package in a project implies a releasing a new version of the whole project. Choosing a given version of a project automatically selects a single version of each of the packages in the project.

Projects and packages have an associated version number, of the form vXrY, with X and Y integers, and this is incremented when changes are made in package contents. The hierarchy of the software organisation is then: <PROJECT>/<PROJECT>_<version>/<package group>/<package>. For example

Sub-directories typically found in an LHCb package include:

• src: source code;
• python: python source files;
• <package> (Sub-directory with same name as package): header files;
• options: job options associated with the package;
• cmt: information for CMT processing;
• doc: package documentation.

Package development in LHCb is currently carried out using the Subversion code versioning system (SVN). This defines a code repository, provides dedicated commands for moving code in and out of the repository, and keeps a record of all changes made.

With the standard LHCb environment, a user can obtain a copy of any package inside a project from the Import menu in Eclipse, selecting the GetPack Wizard.

GetPack ran from a project

This import wizard is only available for existing projects in the workspace.

The appropriate repository is selected automatically by this wizard. All available versions will be displayed in a page, the default version for your project being selected automatically. The latest version of a package, (not necessarily a working version!) is always available as the package trunk. Access to the SVN repository requires ssh authentication, which is automatic on lxplus if you have a valid AFS token.

New releases of projects are placed in the release area after testing. The standard LHCb environment includes the variable $LHCBRELEASES pointing to the AFS release area.

Instructions for making package modifications and new packages available for inclusion in a release can be found in the documentation for Guidelines for SVN actions in LHCb. Web access is available to the SVN repository and to the release area.

As soon as a package update is committed to SVN, it will be picked up by one of the nightly builds the next night. Nightly builds are used to test new software prior to it being officially released. The nightly build status can be viewed at http://cern.ch/lhcb-nightlies/cgi-bin/nightlies.py. Advanced users wishing to pick up the latest updates can run their software against one of the nightlies, see instructions in the LHCb nightlies documentation

LHCb applications

The LHCb applications are projects that use the Gaudi framework to perform specific tasks. There is a project specific to, and with the same name as, each application. The main applications are:

• Gauss: event generation, using Pythia and EvtGen, and detector simulation based on Geant4;
• Boole: simulation of detector response (digitisation)
• Moore: high level trigger
• Brunel: event reconstruction
• DaVinci: event selection and data analysis
• Panoramix: graphical display of detector and event information
• Bender: python-based physics analysis

Additional projects contain software that is shared by several applications. The main ones are:

• LHCb: base classes and public include files
• Online: framework extensions and components for running in the online environment
• Lbcom: components common to all applications
• Hlt: components for online (HLT) reconstruction
• Rec: components for online (HLT) and offline reconstruction
• Phys: components for online (HLT) and offline analysis
• Analysis: components for offline analysis
• Stripping: components for offline event selection (stripping)

Exercise 2

This exercise is designed to give you some familiarity with the LHCb software organisation. It's fairly open-ended, in that you could spend a very long time exploring all of the LHCb packages. Until you're trying to find something - which is when knowing the organisation is a big help - it's probably not useful to spend more than 5-10 minutes looking around.

• Move to the LHCb release area: either

  cd ${LHCBRELEASES}

  or use the web interface to the release area: http://cern.ch/LHCb-release-area/
• Note the different project directories, and enter the project directory for any one of the projects.
• Note the different project versions available, and move to the directory for the latest.
• Note the different package groups, and have a look in two or more of these at what packages are available.
• Choose any two packages, and for each in turn look at which sub-directories are defined and have a look at the files that are in each.

Configuration Management Tool

CMT is used to specify the compilation and execution environment for packages within a project, the dependencies between projects and between packages, and how each package is built (for example, how to produce executables or libraries from source code). To allow this, a CMT package must have a cmt sub-directory containing a requirements file, where the relevant instructions are given. All LHCb packages developed in the Gaudi framework are CMT packages.

For a package with name myPackage, CMT itself defines an environment variable MYPACKAGEROOT that contains the package location.

The full set of commands that can be used in a requirements file is described in the CMT manual. Some of the most useful commands are shown below.

# Lines beginning with the symbol '#' are treated as comments. 
# Blank lines are ignored. 

#================================================================== 
# The following commands describe a package. 
# Package name. 
package myPackage 
# Package version, should correspond to SVN tag.
version v1r0
# Structure, i.e. directories to process. 
# A package must always have: 
#    a cmt directory (for the requirements file) 
#    a doc directory (for the release.notes file) 
# It may have: 
#    a src directory for source files (.h, .cpp)
#    a options directory for job options (.opts, .py) 
#    a python directory for python scripts and configurables 
#    a directory with the same name as the package, containing include files that should be visible outside the package (i.e. the public interface)
branches cmt doc src options myPackage
#==================================================================
# The following command specifies packages which the current package needs for compilation and linking. Put as many lines as necessary.
# The keyword is followed by package name and version and, where applicable, the package group in which the package is placed.
# The version can usually be v* (i.e. any version) because the exact version is usually determined by setting the environment in which a package is built
use DaVinciTools v* Phys
#================================================================== 
# The following commands tell CMT what to build and how 
# Build application with the specified file
application myApp ../src/myApp.cpp
# Build a library with the specified files
library myLib ../src/*.cpp
# Define link options and environment variables for loading component library
apply_pattern component_library library=myLib
# Define link options and environment variables for loading linker library
apply_pattern linker_library library=myLib
# Make the public include files visible to the whole project (i.e. copy them to the InstallArea)
apply_pattern install_more_includes more=myPackage
#==================================================================
# The following commands define symbols (variables and aliases).
# They consist of a keyword followed by a symbol name, and possibly by a value.
# Previously defined symbols, including environment variables, can be used in value assignments by placing brackets - () or {} - around them and a $ in front. 
# Treat symbol as a Make macro definition.This can be used to define a symbol as a local variable (i.e. one that won't be visible from the shell).
macro MYCMTDIR $(HOME)/cmtuser
# Treat symbol as an environment variable. The symbol will be visible from the shell after executing SetupProject
set LCGDIR /afs/cern.ch/sw/lcg
# Treat symbol as a path variable. The symbol will be visible from the shell after executing SetupProject
# When appending or prepending an item to a path variable not initialised in the requirements file, it's a good idea to first remove the item, to ensure it occurs only once.
# Set path.
path PYTHONPATH $(HOME)/python
# Append item to path.
path_append PYTHONPATH $(HOME)/myPythonPackage
# Remove item from path.
path_remove LD_LIBRARY_PATH $(HOME)/lib
# Prepend item to path.
path_prepend LD_LIBRARY_PATH $(HOME)/lib
# Create alias. 
# The alias will be visible from the shell after executing SetupProject
alias myApp $(MYPACKAGEROOT)/$(CMTCONFIG)/myApp.exe
#================================================================== 

Most CMT commands, but not all, operate on requirements files, and must be given from a package's cmt directory. If a requirements file includes lines indicating that other packages are used, then these packages will be searched for in locations specified by the project environment, and the requirements files of each used package is processed in turn.

Those are provided for your information. Eclipse LHCb installation comes with an automated build which will take care of most of theses commands.

The CMT manual should be consulted for the full set of CMT commands, but a useful subset is given below.

• Perform package configuration - that is, create setup files and Make files:

  cmt config

  Note that when a copy of a package is obtained using getpack the package is configured automatically.
• After configuration, Make commands can be used from the cmt directory:
  • Build the package binaries:

    cmt make

    The output of the build operation is placed in a package sub-directory. The name of this sub-directory is given by the value of the environment variable CMTCONFIG.
  • Build several files in parallel if possible:

    cmt make -j

  • Delete the package binaries:

    cmt make binclean

  • Delete all generated files, including copies made to the project InstallArea

    cmt make clean

• Show all packages used by the current package:

  cmt show uses

• Execute <shell command(s)> in cmt directory of current package and all used packages within the same project:

  cmt broadcast <shell command(s)>

  Multiple commands need to be separated by semicolons and enclosed in inverted commas. This can be useful, for example, to force rebuilding of all used packages in the user's area:

  cmt broadcast "cmt config ; cmt make binclean ; cmt make"

• Execute an executable within the environment of the current project

  cmt run gaudirun.py

Have a look at the options of gaudirun.py

Try

for help, or

for a verbose printout of all options.

To release a new package in the LHCb framework it must be imported into SVN so that getpack can find it. Follow the instructions here.

Exercise 3

This exercise introduces basic CMT functionality and demonstrates the use of the LHCb Eclipse Plugins to create different kinds of files.

• Look at the value of the CMTPROJECTPATH environment variable

  echo $CMTPROJECTPATH

The search path for CMT projects was set up when you logged in. It includes a User_release_area, the LHCb_release_area, the Gaudi_release_area and the LCG_release_area (for external software) When opening eclipse, you must provide a workspace. The Workspace is the root folder where all your projects and packages files will sit. The best option is to use the User_release_area, but you should be able to specify anyone. Using the SetupProject Wizard, you can now create a project in which to work. Right-click in the Project Explorer and select New > LHCb Project.

New LHCb Project

On the first page, you can set various options. Here, we will only select the DaVinci project in the drop-down list and hit Next.

DaVinci Project

The Wizard will then display a list of all available versions according to the options you set on the previous page. For the purpose of this tutorial, select the version v29r0 and click Finish.

Choose the project Version

Now Eclipse will create a project with all the basic files you need as DaVinci_v29r0. It appears in the Project Explorer view (on the left if you have the default Eclipse layout).

Done project

Please note that the version on the screenshot might be outdated

• Look at the cmt/project.cmt file that was created: in the Project Explorer View navigate to DaVinci_v29r0 > cmt > project.cmt. You can expand an item by double-clicking on it, or click on the little arrow at the left of the name.

The content of the file appears in the main window, with basic syntax highlighting. This file tells CMT that any CMT command issued from inside this directory tree should use the environment defined by the DAVINCI_v29r0 project, which is found somewhere on the CMTPROJECTPATH (look for it!).

If you want to work with a different application, or with a different version of DaVinci, simply lauch the LHCb Project Wizard again. To go further, information about CMT projects, including tips and tricks for working with them, can be found here

• Create a new package in your DaVinci_v29r0 project:

First you need to create a Hat. It's actually a simple folder. So right-click on the project, select New > Folder. Name it MyGroup.

Create a new Hat

Set Hat Name

Now right-click on the hat you have just created and select New > LHCb Package. A Wizard will be displayed, asking you to provide some informations about the package. Name it MyPackage. A version is set by default to v1r0 which corresponds to the initial version of a Package. The maintainer is set to your name. Check the C++ language, and hit Finish.

New Package

Package setup

Upon creation, the wizard will create the basic package directories and open the requirements file in the editor.

Package requirements

In the requirements file, add the line

Save the file (ctrl+S). The use statement that you added tells CMT that the files in this package will require to compile and link against files in the GaudiAlg package.

• In a terminal, you can navigate to your project directory and try some basic CMT commands:

  cmt show uses ls ls .. cmt config ls ls ..

• Try creating some different types of files with Eclipse. Each time, make a small modification to the file and save it. You can edit the release.notes (changelog), or create a new Component File : Right Click on the src Folder and select New > LHCb Component File.
• Use Eclipse to build the files by pressing Ctrl+B. Or right-click on the project and select Build Project or from Project in the top menubar.

Running LHCb Applications

When executing an application, the following environment variables are very important:

• PATH: search path for scripts and executables, e.g. gaudirun.py;
• LD_LIBRARY_PATH: search path for shared libraries.
• PYTHONPATH: search path for configurables

The environment in which you build and run an application is fully determined by choosing which CMT project you wish to work with, and the value of the CMTPROJECTPATH environment variable.

The SetupProject script removes the need to set these variables explicitly. It uses CMT to fully define the run time environment for a given version of an LHCb application. See also the SetupProject user guide and FAQ.

Once the environment is set, the application is executed by simply typing the executable name, giving the name of a job options file as argument. The job options control what the application actually does when it runs. Controlling what happens when the application is run means knowing how to add new code and understanding the job options.

In the remaining exercises of this tutorial we will use predefined job options files, and concentrate on different use cases for running applications in the LHCb environment.

Exercise 4

Unfortunately at the moment, there is no simple way to launch an LHCb application right from Eclipse. But you can configure an external tool, which will allow you to have the output directly in the Eclipse console.

In the toolbar, select the External Tools icon , and click External Tools Configurations....

External Tools

In the new window, double click on Program in the left list. In the main view, enter SetupProject and Run as the Name. The Location must be

This will execute the released DaVinci application, with default job options. The SetupProject command sets up all the necessary environment to run the application. Once you have chosen a version and set up the environment, the behavior of the application is determined by the job options file that you supply as the argument of the gaudirun.py command.

Hit Apply and Run (or Close).

External Tool Configuration

Note that you only have to do this once for all the projects in your workspace.

The project will then be built, and the commands provided above will be launched in a new console usually right below the editor.

Note: you can configure the external tool execution so that it skips the build using the build tab in the configuration window.

Notice also that you can keep any number of different job options files in your own area, each describing a different configuration of the application (e.g. different cuts), without having to rebuild the standard application. This is recommended if your analysis can be carried out simply by modifying job options (i.e. if you do not need to write your own C++ algorithms).

gaudirun.py is a generic Gaudi application written in python. It is used as main program for all LHCb applications. The specific behavior of the application is defined by the options file(s) provided as argument(s). Options files can be written either in the old Gaudi native syntax (.opts) or using python syntax. The python syntax is recommended.

Exercise 5

In this exercise we address the case where you also need to modify the application executable, typically by adding your own component library package.

In the project you already have, right click on the project and select Import > GetPack Wizard.

GetPack

We will use the Phys/DaVinci package. In the new dialog, enter a few letters of davinci (it's case insensitive) and select the one in the DaVinci project (blue icon) under the Phys hat (yellow icon).

GetPack

Then hit Finish to begin downloading the default version of the package. If you want to specify a different version, click Next and the wizard will display available versions for the selected packages.

• At this point you could modify the requirements file to add your own package to the list of packages to be accessed by the application. You could for example add the MyGroup/MyPackage that you created in Exercise 3. Note that the syntax is as follows:

  use <package name> <version> <package group>

• Build the application using the Build project entry in the Project menu or in the project context menu.
• Finally click on the green play button with the red toolbox in the toolbar (if you have followed the procedure in Exercise 4) to run the application.

Exercise 6

This exercise is independent from Eclipse and demonstrates an alternative method to opening a new window each time you wish to change the version of project you are working with. This method is recommended if you switch versions frequently, for example if you are comparing the results of a new release with a previous one.

• From one terminal window execute the following commands:

  SetupProject DaVinci v29r0 env | grep DAVINCI echo ${DAVINCIROOT}

• Start a new terminal window, and execute the following commands:

  setenvDaVinci v29r0 cd Phys/DaVinci/cmt cmt make which gaudirun.py echo ${DAVINCIROOT} cmt run 'gaudirun.py ${DAVINCIROOT}/options/DaVinci-Default.py $APPCONFIGOPTS/DaVinci/DataType-2010.py' env | grep DAVINCI

Notice that in the new window we didn't execute the SetupProject script, and that consequently the environment does not "remember" any of the environment variables required to run the application. The cmt run command executes the application inside its own shell which is configured according to the environment and requirements file of the directory from which you are issuing the command. In contrast to Exercises 4 and 5, here you have to execute the application from the directory that contains the requirements of the application.

The advantage of this method is that now you can switch to a different project, containing a different version of the application (or even a different application) and simply re-issue the cmt run command to execute the new application, without having to worry about opening a new window and setting up a new environment.

Exercise 7

Another use of SetupProject is to give access to software external to LHCb. In this exercise we see how to set up the environment for the Root application

• Start a new terminal window, and issue the following command:

  which root

The Root application is not accessible in the default lxplus login environment

• Now issue the commands:

  SetupProject Gaudi ROOT echo $ROOTSYS which root

You now have the environment to execute the Root application. The version that you will execute is the version compatible with the latest release of Gaudi

• Now issue the commands:

  SetupProject Gaudi v21r4 ROOT echo $ROOTSYS which root

This time the version of Root that has been selected is the version compatible with Gaudi v21r4

• Finally, issue the commands:

  SetupProject Gaudi v21r4 ROOT -v 5.24.00 echo $ROOTSYS which root

This time we have over-ridden the version of Root compatible with Gaudi v21r4, and selected explicitly a different version. This can be useful if you need to over-ride the version of an external library to use a different version from the one with which the application was released.

Exercise 8

CMT projects are very powerful tools for software development. You can find out more by reading the instructions for working with CMT install areas.

-- GregoireAstruc - 26-Jul-2011

-- MarcoClemencic - 13-Sep-2011