Difference: SoftwareEnvTools (1 vs. 17)

Revision 172018-03-22 - PaulSeyfert

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Tools for the LHCb Software Environment

Line: 68 to 68
 lb-run --runtime DaVinci v36r5 Moore/v23r5p2 gaudirun.py
Added:
>
>

Tunings

Using ccache

If you're going to compile identical files over and over again (when building full projects, but only manipulating a few files; whilst calling make clean in between, or changing git branches), it may pay off to enable ccache. As also described on the git4lhcb twiki page (https://twiki.cern.ch/twiki/bin/view/LHCb/Git4LHCb) this can be done with

# Use ccache
export CCACHE_DIR=/<somewhere>/.ccache   # local directory where the cache will be kept
ccache --max-size=10G                    # choose wisely based on the available space and the size of the projects

# tell CMake to use ccache
export CMAKEFLAGS="-DCMAKE_USE_CCACHE=ON"  # option 1: enable as environment variable
# or
cd Rec # project you want to build
echo 'set(CMAKE_USE_CCACHE ON CACHE BOOL "")' >> cache_preload.cmake   # option 2: use a cache_preload file
# cache_preload.cmake can hold some predefined variables, so you can skip 'CMAKEFLAGS'

colorized compiler warnings/errors

Modern compilers colorize their printouts (errors red, notes blue) to make them easier to read, especially when it's several pages of them. With ninja however compilers cannot determine if they print to a log file or to a terminal. Modern ninja versions take over that task, but require that compilers will print everything in color w/o auto detection on the compiler side. The setting to enable colors with ninja is (depending on the Gaudi version)

cd Rec # project to build
echo 'set(GAUDI_DIAGNOSTICS_COLOR ON CACHE BOOL "")' >> cache_preload.cmake       # the latest Gaudi versions, as cache_preload file
# or
echo 'set(GAUDI_DIAGNOTICS_COLOR ON CACHE BOOL "")' >> cache_preload.cmake        # some intermediate Gaudi versions, as cache_preload file
# or
export CMAKEFLAGS="-DGAUDI_DIAGNOSTICS_COLOR=ON" #  latest Gaudi, through environment variables
# or
export CMAKEFLAGS="-DGAUDI_DIAGNOTICS_COLOR=ON"  #  some intermediate Gaudi versions, through environment variables

(There are a few things to iron out at the ninja side, but it should work okay'ish in many cases https://github.com/ninja-build/ninja/issues/1214 https://github.com/ninja-build/ninja/pull/1382 https://github.com/ninja-build/ninja/pull/1268)

 

Runtime

The way we build and distribute our software require that we set several environment variables to run an application. This is taken care of by the script lb-run.
Line: 106 to 140
 lb-run Hlt/prod printenv LD_LIBRARY_PATH
Added:
>
>
For shell skripting, you may want to use an environment variable. For this it is important that the variable gets resolved in the lb-run environment and not in your user shell before lb-run gets called.
# do this:
lb-run Brunel/v54r0 echo '$BRUNELROOT' >> ${HOME}/foo.txt    # works with single quotes
lb-run Brunel/v54r0 echo \$BRUNELROOT >> ${HOME}/foo.txt    # works with escaped dollar
# don't do that
lb-run Brunel/v54r0 echo $BRUNELROOT >> ${HOME}/foo.txt    # doesn't work w/o quotes
lb-run Brunel/v54r0 echo "$BRUNELROOT" >> ${HOME}/foo.txt    # doesn't work w/ double quotes
 Note that to distinguish from the options for lb-run and the options to the command that lb-run will execute, it is mandatory to specify lb-run options before the name of the project. So in
lb-run Gaudi/prod myScript --nightly

Revision 162018-03-22 - PaulSeyfert

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Tools for the LHCb Software Environment

Line: 114 to 114
  Sometimes (e.g. for debugging purposes) it is useful to execute several commands in the modified environment, and wrapping all the calls with lb-run can be tedious and error prone. In that case it's easy to start a subshell with the modified environment:
Changed:
<
<
lb-run LHCb/prod bash -f
>
>
lb-run LHCb/prod bash --norc
  or, for the tcsh lovers,
Line: 156 to 156
 
lb-run LHCb/latest tcsh -f
Changed:
<
<
(or bash -f or zsh -f depending on your favourite shell)
>
>
(or bash --norc or zsh -f depending on your favourite shell)
  Other important differences are
  • SetupProject does not understand the metadata produced with CMake builds and lb-run does not understand project built with CMT, but it falls back on SetupProject if needed

Revision 152017-02-10 - PaulSeyfert

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Tools for the LHCb Software Environment

Line: 84 to 84
  It possible to tune in several ways the behaviour of lb-run, through the various command line options (see lb-run --help), for example to use a project from the nightly builds:
Changed:
<
<
lb-run --nightly lhcb-head DaVinci HEAD gaudirun.py MyOptions.py
>
>
lb-run --nightly lhcb-head DaVinci/HEAD gaudirun.py MyOptions.py
  or to fine tune the environment used to run the command:

Revision 142016-11-25 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Tools for the LHCb Software Environment

Line: 20 to 20
  The simplest way to call lb-dev is
Changed:
<
<
lb-dev
>
>
lb-dev /
  This will create a new subdirectory called Dev_ in the local directory and will create all the support files to be able to develop against .

Another very common use case is to build against the version of a project build in the LHCb Nightly Build System, for example:

Changed:
<
<
lb-dev --nightly lhcb-gaudi-head Gaudi HEAD
>
>
lb-dev --nightly lhcb-gaudi-head Gaudi/HEAD
  or
Changed:
<
<
lb-dev --nightly lhcb-gaudi-head Thu Gaudi HEAD
>
>
lb-dev --nightly lhcb-gaudi-head Thu Gaudi/HEAD
  They will both create GaudiDev_HEAD bound to the nightly slot lhcb-gaudi-head, but the first one to the Today build and the second on the build of Thursday (this connection can be changes editing the file build.conf in GaudiDev_HEAD).

Very useful options of lb-dev are

Changed:
<
<
  • --name to specify the name of the local project, like in lb-dev --name MyAnalysis DaVinci prod
>
>
  • --name to specify the name of the local project, like in lb-dev --name MyAnalysis DaVinci/prod
 
  • --dest-dir to specify the parent directory of the created local project (instead of '.')

More options can be explored with lb-dev --help.

Line: 60 to 60
 
  1. The name of the project will be different. For example Moore -> MooreDev
Example:
Changed:
<
<
lb-run --user-area $HOME MooreDev v23r5p2 gaudirun.py
>
>
lb-run --user-area $HOME MooreDev/v23r5p2 gaudirun.py
 

You can use the --runtime option to add projects, for example:

Changed:
<
<
lb-run --runtime DaVinci v36r5 Moore v23r5p2 gaudirun.py
>
>
lb-run --runtime DaVinci v36r5 Moore/v23r5p2 gaudirun.py
 

Runtime

Line: 75 to 75
  The simplest way to use it is
Changed:
<
<
lb-run
>
>
lb-run /
  for example:
Changed:
<
<
lb-run Brunel prod gaudirun.py MyBrunelOptions.py
>
>
lb-run Brunel/prod gaudirun.py MyBrunelOptions.py
 

It possible to tune in several ways the behaviour of lb-run, through the various command line options (see lb-run --help), for example to use a project from the nightly builds:

Line: 98 to 98
  It's also possible to change the platform to use on the fly without messing with the current environment:
Changed:
<
<
lb-run -c x86_65-slc5-gcc49-dbg Gaudi latest python
>
>
lb-run -c x86_65-slc5-gcc49-dbg Gaudi/latest python
 

As it happens with the Unix command env, if no command is passed to lb-run, it prints the modified environment. It's useful, though, to print only one variable, for example with something like

Changed:
<
<
lb-run Hlt prod printenv LD_LIBRARY_PATH
>
>
lb-run Hlt/prod printenv LD_LIBRARY_PATH
 

Note that to distinguish from the options for lb-run and the options to the command that lb-run will execute, it is mandatory to specify lb-run options before the name of the project. So in

Changed:
<
<
lb-run Gaudi prod myScript --nightly
>
>
lb-run Gaudi/prod myScript --nightly
  the option --nightly will be passed to the command myScript and not to lb-run.

Sometimes (e.g. for debugging purposes) it is useful to execute several commands in the modified environment, and wrapping all the calls with lb-run can be tedious and error prone. In that case it's easy to start a subshell with the modified environment:

Changed:
<
<
lb-run LHCb prod bash -f
>
>
lb-run LHCb/prod bash -f
  or, for the tcsh lovers,
Changed:
<
<
lb-run LHCb prod tcsh -f
>
>
lb-run LHCb/prod tcsh -f
  (The -f option avoids the invocation of /etc/bashrc or other /etc files, which would invoke the LbLogin scripts and manipulate the environment variables on top of lb-run) Once done, you can exit from the subshell and get back to the initial environment.
Line: 148 to 148
  is equivalent to
Changed:
<
<
lb-run LHCb latest CondDBBrowser
>
>
lb-run LHCb/latest CondDBBrowser
  but in the first case you have to exit from the shell to get back to a clean environment.

If you prefer the SetupProject behaviour, use:

Changed:
<
<
lb-run LHCb tcsh
>
>
lb-run LHCb/latest tcsh -f
  (or bash -f or zsh -f depending on your favourite shell)
Line: 162 to 162
 
  • SetupProject does not understand the metadata produced with CMake builds and lb-run does not understand project built with CMT, but it falls back on SetupProject if needed
  • lb-run does not support (yet) some of the less used features of SetupProject, mainly adding external libraries
Changed:
<
<
-- MarcoClemencic - 2015-01-15
>
>
-- MarcoClemencic - 2016-11-25
 \ No newline at end of file

Revision 132016-05-29 - PaulSeyfert

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Tools for the LHCb Software Environment

Line: 114 to 114
  Sometimes (e.g. for debugging purposes) it is useful to execute several commands in the modified environment, and wrapping all the calls with lb-run can be tedious and error prone. In that case it's easy to start a subshell with the modified environment:
Changed:
<
<
lb-run LHCb prod bash
>
>
lb-run LHCb prod bash -f
  or, for the tcsh lovers,
lb-run LHCb prod tcsh -f
Added:
>
>
(The -f option avoids the invocation of /etc/bashrc or other /etc files, which would invoke the LbLogin scripts and manipulate the environment variables on top of lb-run)
 Once done, you can exit from the subshell and get back to the initial environment. This is very handy to have nested environments that do not conflict with each other, for example to run in sequence Gauss, Bool and Brunel, each of them in the same LHCbDirac environment.
Line: 155 to 156
 
lb-run LHCb tcsh
Changed:
<
<
(or bash/zsh/... depending on your favourite shell)
>
>
(or bash -f or zsh -f depending on your favourite shell)
  Other important differences are
  • SetupProject does not understand the metadata produced with CMake builds and lb-run does not understand project built with CMT, but it falls back on SetupProject if needed

Revision 122016-05-03 - RosenMatev

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Tools for the LHCb Software Environment

Line: 92 to 92
  (in this case we prepend $HOME/python to the variable PYTHONPATH defined by the Bender runtime environment and then call the command python).
Added:
>
>
More exotic project versions can be accessed with the Project/version syntax, e.g.
lb-run --nightly-cvmfs --nightly lhcb-2016-patches Brunel/2016-patches gaudirun.py MyBrunelOptions.py
 It's also possible to change the platform to use on the fly without messing with the current environment:
lb-run -c x86_65-slc5-gcc49-dbg Gaudi latest python

Revision 112015-09-01 - MarcoCattaneo

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Tools for the LHCb Software Environment

Line: 60 to 60
 
  1. The name of the project will be different. For example Moore -> MooreDev
Example:
Changed:
<
<
b-run --user-area $HOME MooreDev v23r5p2 gaudirun.py
>
>
lb-run --user-area $HOME MooreDev v23r5p2 gaudirun.py
 

You can use the --runtime option to add projects, for example:

Revision 102015-06-25 - GiulioDujany

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Tools for the LHCb Software Environment

Line: 34 to 34
  They will both create GaudiDev_HEAD bound to the nightly slot lhcb-gaudi-head, but the first one to the Today build and the second on the build of Thursday (this connection can be changes editing the file build.conf in GaudiDev_HEAD).
Changed:
<
<
Very useful options of lb-def are
>
>
Very useful options of lb-dev are
 
  • --name to specify the name of the local project, like in lb-dev --name MyAnalysis DaVinci prod
  • --dest-dir to specify the parent directory of the created local project (instead of '.')

Revision 92015-06-18 - NikitaKazeev

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Tools for the LHCb Software Environment

Line: 53 to 53
 echo $TRACKKERNELROOT
Added:
>
>
If you want to use lb-run to run your modified projects created with lb-dev you need:
  1. Run make install in the project folder.
  2. Make sure the project folder is in the search path. If not, you can use --user-area
  3. Make sure you specify the version number.
  4. The name of the project will be different. For example Moore -> MooreDev
Example:
b-run --user-area $HOME MooreDev v23r5p2 gaudirun.py 

You can use the --runtime option to add projects, for example:

lb-run --runtime DaVinci v36r5 Moore v23r5p2 gaudirun.py
 

Runtime

The way we build and distribute our software require that we set several environment variables to run an application. This is taken care of by the script lb-run.

Revision 82015-03-06 - PaulSeyfert

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Tools for the LHCb Software Environment

Line: 46 to 46
 ./run gaudirun.py MyOptions.py
Added:
>
>
Similarly, if you want to do more than just run gaudirun.py once, you can start a shell with the runtime environment:
cd MyAnalysis
./run tcsh
echo $TRACKKERNELROOT
 

Runtime

The way we build and distribute our software require that we set several environment variables to run an application. This is taken care of by the script lb-run.

Revision 72015-02-12 - PaulSeyfert

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Tools for the LHCb Software Environment

Line: 125 to 125
  but in the first case you have to exit from the shell to get back to a clean environment.
Added:
>
>
If you prefer the SetupProject behaviour, use:
lb-run LHCb tcsh
(or bash/zsh/... depending on your favourite shell)
 Other important differences are
  • SetupProject does not understand the metadata produced with CMake builds and lb-run does not understand project built with CMT, but it falls back on SetupProject if needed
  • lb-run does not support (yet) some of the less used features of SetupProject, mainly adding external libraries

Revision 62015-01-26 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Tools for the LHCb Software Environment

Line: 97 to 97
 Once done, you can exit from the subshell and get back to the initial environment. This is very handy to have nested environments that do not conflict with each other, for example to run in sequence Gauss, Bool and Brunel, each of them in the same LHCbDirac environment.
Added:
>
>
 lb-run leverages on the metadata produced when building the projects with CMake, but it falls back on SetupProject if it cannot find them. In this case the command line options are passed t SetupProject, but some options are only available in lb-run (for example the prepend option -p), so the command line might need some tuning before it succeeds.

Comparison with SetupProject

Revision 52015-01-16 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Tools for the LHCb Software Environment

Line: 40 to 40
  More options can be explored with lb-dev --help.
Added:
>
>
In the local project directory you will also find a script called run. It is a shortcut to run a command in the environment of your local project, for example you can do something like:
cd MyAnalysis
./run gaudirun.py MyOptions.py
 

Runtime

The way we build and distribute our software require that we set several environment variables to run an application. This is taken care of by the script lb-run.

Revision 42015-01-16 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Tools for the LHCb Software Environment

Line: 103 to 103
 
  • by default SetupProject creates the local project in $User_release_area, while lb-dev creates it in the current (or specified) directory
  • local projects created with lb-dev contain details about the environment they require, so it is enough to enter the directory and build, while with SetupProject --build-env one has to call every time
  • lb-dev supports CMake-based LHCb software projects while SetupProject does not.
Added:
>
>
  • if the destination directory already exist SetupProject will silently do nothing (or prepare a possibly wrong build time environment), while lb-dev will print an error message (once created the directory, there is no need to call lb-dev again)
 

Runtime

Revision 32015-01-15 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Deleted:
<
<
 

Tools for the LHCb Software Environment

Line: 55 to 54
 lb-run Brunel prod gaudirun.py MyBrunelOptions.py
Changed:
<
<

Comparison with SetupProject

Development

The main differences between SetupProject --build-env and lb-dev are:

  • lb-dev does not modify the runtime environment and does not call cd to switch to the created directory
  • the default version of a project used by lb-dev is prod, instead of picking up the latest one
  • by default SetupProject creates the local project in $User_release_area, while lb-dev creates it in the current (or specified) directory

Basic Usage

The simplest use-case one can imagine, is the configuration of the runtime environment for a released version of a project, so that one can call directly the application passing the preferred options, without having to check out from CVS the top level package of the application.

To be able to use SetupProject, you need the minimal LHCb environment that you get when you log on lxplus.cern.ch or after having sourced the ExtCMT script in a local installation.

The minimal information that you need to pass to SetupProject is the name of the application (or project) you want to use. Obviously you may want to specify the version, but if you do not do it, SetupProject will use the latest available. An example is:

[lxplus] ~ > SetupProject DaVinci
Configuring DaVinci v20r0 from /afs/cern.ch/lhcb/software/releases/DAVINCI/DAVINCI_v20r0
Using CMTPROJECTPATH = '/afs/cern.ch/user/m/marcocle/cmtuser:/afs/cern.ch/lhcb/software/releases:/afs/cern.ch/sw/Gaudi/releases:/afs/cern.ch/sw/lcg/app/releases'
Environment for DaVinci v20r0 ready.
(taken from DaVinci v20r0 from /afs/cern.ch/lhcb/software/releases/DAVINCI/DAVINCI_v20r0)

As you can see, the output is pretty self explanatory. You have the version of the application, the CMTPROJECTPATH used to locate dependent projects and where the binaries are actually taken from.

To specify the version of the application that you need, just add it to the command line:

[lxplus] ~ > SetupProject DaVinci v19r14
Configuring DaVinci v19r14 from /afs/cern.ch/lhcb/software/releases/DAVINCI/DAVINCI_v19r14
Using CMTPROJECTPATH = '/afs/cern.ch/user/m/marcocle/cmtuser:/afs/cern.ch/lhcb/software/releases:/afs/cern.ch/sw/Gaudi/releases:/afs/cern.ch/sw/lcg/app/releases'
Environment for DaVinci v19r14 ready.
(taken from DaVinci v19r14 from /afs/cern.ch/lhcb/software/releases/DAVINCI/DAVINCI_v19r14)

If you do not remember or do not know which versions are available, you have two choices: you can tell SetupProject to list the versions it can use or let it propose the list from which to chose a version. The list is produced with the command line option --list-versions, that can be abbreviated with --list:

[lxplus] ~ > SetupProject DaVinci --list
v12r18 in /afs/cern.ch/lhcb/software/releases
v14r5 in /afs/cern.ch/lhcb/software/releases
v19r5 in /afs/cern.ch/lhcb/software/releases
v19r6 in /afs/cern.ch/lhcb/software/releases
v19r7 in /afs/cern.ch/lhcb/software/releases
v19r8 in /afs/cern.ch/lhcb/software/releases
v19r9 in /afs/cern.ch/lhcb/software/releases
v19r10 in /afs/cern.ch/user/m/marcocle/cmtuser
v19r11 in /afs/cern.ch/lhcb/software/releases
v19r12 in /afs/cern.ch/lhcb/software/releases
v19r13 in /afs/cern.ch/user/m/marcocle/cmtuser
v19r14 in /afs/cern.ch/lhcb/software/releases
v20r0 in /afs/cern.ch/lhcb/software/releases

(you can notice that most of the proposed versions come from the standard release area, while two come from my cmtuser directory, the reason of it is explained later in this page). If you prefer to chose the version from a list of possibilities, the command line option is --ask:

>
>
It possible to tune in several ways the behaviour of lb-run, through the various command line options (see lb-run --help), for example to use a project from the nightly builds:
 
Changed:
<
<
[lxplus] ~ > SetupProject DaVinci --ask Please enter your choice (v12r18 v14r5 v19r5 v19r6 v19r7 v19r8 v19r9 v19r10 v19r11 v19r12 v19r13 v19r14 v20r0 q[uit] [v20r0]): v19r8 Trying version 'v19r8' Configuring DaVinci v19r8 from /afs/cern.ch/lhcb/software/releases/DAVINCI/DAVINCI_v19r8 Using CMTPROJECTPATH = '/afs/cern.ch/user/m/marcocle/cmtuser:/afs/cern.ch/lhcb/software/releases:/afs/cern.ch/sw/Gaudi/releases:/afs/cern.ch/sw/lcg/app/releases' Environment for DaVinci v19r8 ready. (taken from DaVinci v19r8 from /afs/cern.ch/lhcb/software/releases/DAVINCI/DAVINCI_v19r8)

In this case, at the prompt, you can press enter without typing anything to accept the default (equivalent to a call to SetupProject without explicit version), type q to avoid changes to your environment or select a different version.

You may need to try versions that are not in production yet, like the build available in the special release are called LHCBDEV. Since SetupProject doesn't propose those version by default to avoid possible problems, you can enable LHCBDEV with the command line option --dev:

[lxplus] ~ > SetupProject Panoramix --list
v9r8 in /afs/cern.ch/lhcb/software/releases
v15r8 in /afs/cern.ch/lhcb/software/releases
v15r9 in /afs/cern.ch/lhcb/software/releases
v15r10 in /afs/cern.ch/lhcb/software/releases
v15r11 in /afs/cern.ch/lhcb/software/releases
v15r13 in /afs/cern.ch/lhcb/software/releases
v15r14 in /afs/cern.ch/lhcb/software/releases
v15r15 in /afs/cern.ch/lhcb/software/releases
[lxplus] ~ > SetupProject Panoramix --dev --list
v9r8 in /afs/cern.ch/lhcb/software/releases
v15r8 in /afs/cern.ch/lhcb/software/releases
v15r9 in /afs/cern.ch/lhcb/software/releases
v15r10 in /afs/cern.ch/lhcb/software/releases
v15r11 in /afs/cern.ch/lhcb/software/releases
v15r13 in /afs/cern.ch/lhcb/software/releases
v15r14 in /afs/cern.ch/lhcb/software/releases
v15r15 in /afs/cern.ch/lhcb/software/releases
v15r16 in /afs/cern.ch/lhcb/software/DEV
v16r0 in /afs/cern.ch/lhcb/software/DEV

Note that the version that would be taken by default is the last one of the list.

There is another way of using a DEV directory, --dev-dir, which allows you to specify something different from the standard LHCBDEV or more than one "dev" directory (not a very common use case, I agree). An example of a dev directory is the location of the LHCb nightly builds:

[lxplus] ~ > SetupProject Brunel --dev-dir /afs/cern.ch/lhcb/software/nightlies/lhcb2/Wed
Configuring Brunel HEAD from /afs/cern.ch/lhcb/software/nightlies/lhcb2/Wed/BRUNEL/BRUNEL_HEAD
Using CMTPROJECTPATH = '/afs/cern.ch/lhcb/software/nightlies/lhcb2/Wed:/afs/cern.ch/user/m/marcocle/cmtuser:/afs/cern.ch/lhcb/software/releases:/afs/cern.ch/sw/Gaudi/releases:/afs/cern.ch/sw/lcg/app/releases'
Environment for Brunel HEAD ready.
(taken from Brunel HEAD from /afs/cern.ch/lhcb/software/nightlies/lhcb2/Wed/BRUNEL/BRUNEL_HEAD)

User Local Projects

The supported way (within LHCb) of developing software in the context of an application/project is to create a "user project" in which you can check out packages from CVS or create your new ones. You do not need a deep knowledge of the internal structure of projects managed with CMT, SetupProject will do the dirty job of creating the project directory if you use the option --build-env. You have to specify the name of the project and the version you want to use. If the version is omitted, SetupProject will ask which one to use (this differs from the behavior described before to maintain the same behavior of a previous tool used for the same purpose). Example:

[lxplus] ~ > SetupProject LHCb --build-env v25r0
Build-time environment for LHCb v25r0 ready.
Created user project in /afs/cern.ch/user/m/marcocle/cmtuser/LHCb_v25r0
Current directory is '/afs/cern.ch/user/m/marcocle/cmtuser/LHCb_v25r0'.
Using CMTPROJECTPATH = '/afs/cern.ch/user/m/marcocle/cmtuser:/afs/cern.ch/lhcb/software/releases:/afs/cern.ch/sw/Gaudi/releases:/afs/cern.ch/sw/lcg/app/releases'
[lxplus] ~/cmtuser/LHCb_v25r0 > 

After the call to SetupProject, you will be in the directory of the local project, ready to call getpack or to create your package. If the project directory is already present, it will not be modified (you can tell the difference because SetupProject will not print the line Created user project in ...). The location in which the user project is created is defined by the variable User_release_area, if you want the project to be created somewhere else, just change its value (the default is ~/cmtuser).

It must be clear that when called with --build-env, SetupProject cannot prepare a runtime environment. The reason is simple: if you want to build some code, most probably the libraries you may want to use do not exist yet in your local project, so there is nothing you can use yet.

To prepare the runtime environment using your user project, you don't need to do anything special: SetupProject always look for projects in your User_release_area, as you can notice with --list:

[lxplus] ~ > SetupProject LHCb --list
v23r5 in /afs/cern.ch/lhcb/software/releases
v23r6 in /afs/cern.ch/lhcb/software/releases
v23r7 in /afs/cern.ch/lhcb/software/releases
v24r0 in /afs/cern.ch/lhcb/software/releases
v24r1 in /afs/cern.ch/lhcb/software/releases
v25r0 in /afs/cern.ch/user/m/marcocle/cmtuser

The fact that the user project is used, is also clear when setting up the runtime environment:

>
>
lb-run --nightly lhcb-head DaVinci HEAD gaudirun.py MyOptions.py or to fine tune the environment used to run the command:
 
Changed:
<
<
[lxplus] ~ > SetupProject LHCb v25r0 Configuring LHCb v25r0 from /afs/cern.ch/user/m/marcocle/cmtuser/LHCb_v25r0 Using CMTPROJECTPATH = '/afs/cern.ch/user/m/marcocle/cmtuser:/afs/cern.ch/lhcb/software/releases:/afs/cern.ch/sw/Gaudi/releases:/afs/cern.ch/sw/lcg/app/releases' Environment for LHCb v25r0 ready. (taken from LHCb v25r0 from /afs/cern.ch/user/m/marcocle/cmtuser/LHCb_v25r0)

If you do not want that your local project is picked up, you have to unset User_release_area (or to rename the user project directory).

Advanced Usage

Use Non-standard Packages

When SetupProject is asked to prepare the runtime environment for a project, it uses all the packages that belong to the requested project plus those that are used by them, directly or indirectly. This means that setting up Brunel will not give you access to all the packages in LHCb.

There is at least one case where the behavior could be annoying: the CondDBBrowser. The graphical user interface to the Conditions Database needs some libraries that are made available only if you use the package Tools/CondDBUI, which is part of LHCb. Of course, you can do SetupProject LHCb and get the browser, but you may be working with Panoramix and do not want to leave of modify (maybe corrupt) the environment to be able to start the browser, so SetupProject gives you the possibility of getting the environment specific to packages that are not in the project with the option --use:

>
>
lb-run -p PYTHONPATH=$HOME/python Bender python (in this case we prepend $HOME/python to the variable PYTHONPATH defined by the Bender runtime environment and then call the command python).
 
Added:
>
>
It's also possible to change the platform to use on the fly without messing with the current environment:
 
Changed:
<
<
[lxplus] ~ > SetupProject Panoramix --use Tools/CondDBUI Configuring Panoramix v15r15 from /afs/cern.ch/lhcb/software/releases/PANORAMIX/PANORAMIX_v15r15 Using CMTPROJECTPATH = '/afs/cern.ch/user/m/marcocle/cmtuser:/afs/cern.ch/lhcb/software/releases:/afs/cern.ch/sw/Gaudi/releases:/afs/cern.ch/sw/lcg/app/releases' Environment for Panoramix v15r15 ready. (taken from Panoramix v15r15 from /afs/cern.ch/lhcb/software/releases/PANORAMIX/PANORAMIX_v15r15) [lxplus] ~ > echo $CONDDBUIROOT /afs/cern.ch/lhcb/software/releases/LHCB/LHCB_v24r0/Tools/CondDBUI/v2r10

The usefulness of --use is not limited to add a package to the list of the used ones. With special projects like DBASE and PARAM, you have many versions of a package available at the same time, but only the latest one us used. If you need a special version of one of these packages, for example the version of DecFiles has to be fixed for productions of simulated data, you can add it to the use parameter:

>
>
lb-run -c x86_65-slc5-gcc49-dbg Gaudi latest python
 
Added:
>
>
As it happens with the Unix command env, if no command is passed to lb-run, it prints the modified environment. It's useful, though, to print only one variable, for example with something like
 
Changed:
<
<
[lxplus] ~ > SetupProject Gauss v33r2 --use "Gen/DecFiles v14r4" Configuring Gauss v33r2 from /afs/cern.ch/lhcb/software/releases/GAUSS/GAUSS_v33r2 Using CMTPROJECTPATH = '/afs/cern.ch/user/m/marcocle/cmtuser:/afs/cern.ch/lhcb/software/releases:/afs/cern.ch/sw/Gaudi/releases:/afs/cern.ch/sw/lcg/app/releases' Environment for Gauss v33r2 ready. (taken from Gauss v33r2 from /afs/cern.ch/lhcb/software/releases/GAUSS/GAUSS_v33r2) [lxplus] ~ > echo $DECFILESROOT /afs/cern.ch/lhcb/software/releases/DBASE/Gen/DecFiles/v14r4

Please, note the quotes around "Gen/DecFiles v14r4": they are needed.

More Advanced usage

Quick Reference

Set up the runtime environment for a released application
  • SetupProject DaVinci
  • SetupProject Brunel v33r1
  • SetupProject Panoramix --ask
Set up the runtime environment for an application in the nightly builds
  • SetupProject DaVinci --nightly lhcb2
  • SetupProject DaVinci --nightly lhcb2 Wed
Prepare a user local project
  • SetupProject --build-env LHCb v25r0
  • setenvLHCb v25r0
    (an alias to the previous command)

Custom Project Search Path (CMTPROJECTPATH)

Since beginning of October 2008, SetupProject does not use anymore the defined CMTPROJECTPATH (the environment variable used to declare where to find projects), but it always constructs it from User_release_area (by default ~/cmtuser), LHCBPROJECTPATH (the default basic project search path) and the command line.

The reason for the change of behavior is needed because the old behavior could lead to some inconsistencies across calls to SetupProject. Of course, something similar to the old behavior is still possible with the command line option --keep-CMTPROJECTPATH.

Here I describe how to have SetupProject constructing the CMTPROJECTPATH for you. The basic logic is that the minimal CMTPROJECTPATH is given by the environment variable LHCBPROJECTPATH, the options to modify the CMTPROJECTPATH are prepended to it and the User_release_area must always be the first one to be searched. The options that modify the CMTPROJECTPATH are:

  • --dev
  • --dev-dir <dir>
  • --nightly <slot> [<day>]
The basic call to SetupProject, without any special option will set CMTPROJECTPATH to ${User_release_area}:${LHCBPROJECTPATH}.

The option --dev prepends to the CMTPROJECTPATH the content of the environment variable LHCBDEV, so that a call

>
>
lb-run Hlt prod printenv LD_LIBRARY_PATH
 
Added:
>
>
Note that to distinguish from the options for lb-run and the options to the command that lb-run will execute, it is mandatory to specify lb-run options before the name of the project. So in
 
Changed:
<
<
SetupProject LHCb --dev

will set CMTPROJECTPATH to ${User_release_area}:${LHCBDEV}:${LHCBPROJECTPATH}.

--dev-dir is used to specify a different "dev" area, so --dev is actually equivalent to --dev-dir ${LHCBDEV}. The call

>
>
lb-run Gaudi prod myScript --nightly the option --nightly will be passed to the command myScript and not to lb-run.
 
Added:
>
>
Sometimes (e.g. for debugging purposes) it is useful to execute several commands in the modified environment, and wrapping all the calls with lb-run can be tedious and error prone. In that case it's easy to start a subshell with the modified environment:
 
Changed:
<
<
SetupProject LHCb --dev-dir /afs/cern.ch/user/m/marcocle/public

will result in the value ${User_release_area}:/afs/cern.ch/user/m/marcocle/public:${LHCBPROJECTPATH} for CMTPROJECTPATH.

The option --nightly allows to prepare the environment to use the standard LHCb nightly build directories. It accepts a mandatory argument defining the slot to use (e.g. lhcb1 or lhcb2) and an optional argument for the day (3 chars abbreviation), which is defaulted to the current day. With this option, SetupProject checks the existence of the configuration file of the nightly build and uses the information in it to prepare the CMTPROJECTPATH (needed for some special configuration of the nightly build slot). The call

>
>
lb-run LHCb prod bash or, for the tcsh lovers,
 
Changed:
<
<
SetupProject LHCb --nightly lhcb1

will set CMTPROJECTPATH to ${User_release_area}:/afs/cern.ch/lhcb/software/nightlies/lhcb1/Fri:...:${LHCBPROJECTPATH} (assuming today is Friday), where the ... will contain the special entries required by the slot.

The options described can be combined for a more complex CMTPROJECTPATH. The specified directories are added to the CMTPROJECTPATH in the order specified on the command line, after User_release_area and before LHCBPROJECTPATH. So from

>
>
lb-run LHCb prod tcsh -f Once done, you can exit from the subshell and get back to the initial environment. This is very handy to have nested environments that do not conflict with each other, for example to run in sequence Gauss, Bool and Brunel, each of them in the same LHCbDirac environment.
 
Changed:
<
<
SetupProject LHCb --dev-dir /afs/cern.ch/user/m/marcocle/public --nightly lhcb1 Mon --dev --dev-dir /afs/cern.ch/user/s/somebody/public
>
>
lb-run leverages on the metadata produced when building the projects with CMake, but it falls back on SetupProject if it cannot find them. In this case the command line options are passed t SetupProject, but some options are only available in lb-run (for example the prepend option -p), so the command line might need some tuning before it succeeds.
 
Changed:
<
<
you should expect CMTPROJECTPATH featuring, in this order,
  • ${User_release_area}
  • /afs/cern.ch/user/m/marcocle/public
  • /afs/cern.ch/lhcb/software/nightlies/lhcb1/Mon
  • ${LHCBDEV}
  • /afs/cern.ch/user/s/somebody/public
  • ${LHCBPROJECTPATH}.
>
>

Comparison with SetupProject

Development

 
Changed:
<
<

Caveats

>
>
The main differences between SetupProject --build-env and lb-dev are:
  • lb-dev does not modify the runtime environment and does not call cd to switch to the created directory
  • the default name of the local project created by lb-dev is different from that used by SetupProject; this is because the names used with SetupProject will confuse the CMake build system, and with the new names it will be more clear which is a released project and which a local one
  • the default version of a project used by lb-dev is prod, instead of picking up the latest one
  • by default SetupProject creates the local project in $User_release_area, while lb-dev creates it in the current (or specified) directory
  • local projects created with lb-dev contain details about the environment they require, so it is enough to enter the directory and build, while with SetupProject --build-env one has to call every time
  • lb-dev supports CMake-based LHCb software projects while SetupProject does not.
 
Changed:
<
<
There are 2 main usages for SetupProject: the fist one is to setup the full environment of a project and the second one is to prepare the environment for the software development. This second usage which is performed by using the --build-env option (or equivalently by using the alias setenvProject) doesn't setup the full environment. It only updates the environment variable LHCBPROJECTPATH.
>
>

Runtime

 
Changed:
<
<
This is all you need to build your software. For example, here is a normal usage of SetupProject for both development and run:
  1. Prepare the environment which the custom directory from the nightly builds:
                SetupProject --build-env --nightly lhcb2 DaVinci v24r0  (setenvProject --nightly lhcb2 DaVinci v24r0)
  2. checkout the package you would like to modifiy
                getpack MyPack
  3. Modify your package
  4. Build it
>
>
The biggest difference between SetupProject and lb-run is that the first modifies the environment of the shell, while the second prepare the environment to run a command. The sequence of commands
 
Changed:
<
<
cd MyPack/cmt cmt make
  1. Do the full setup to run it
                SetupProject --nightly lhcb2 DaVinci v24r0  

It is important that the build is done before the full setup is done. Otherwise you could end up with a package that doesn't build in the normal environment.

Bug reporting

Please report bugs to the SetupProject category of the lhcbscripts Savannah bug tracker

Appendix

SetupProject online help

This is the help message you can get calling SetupProject --help:

>
>
SetupProject LHCb CondDBBrowser is equivalent to
 
Changed:
<
<
Usage: SetupProject.py [options] [version|--ask] [options] [externals]

Options: --version show program's version number and exit -h, --help show this help message and exit --site SITE enable site specific defaults --ask ask for the version of the project to use (overrides the version specified) --disable-CASTOR remove CASTOR from the added dependencies --tag_add TAG_ADD specify extra CMT tags --use USE add a CMT use statement --verbose be a bit more verbose --debug output useful for debugging --ignore-missing do not fail if some externals are missing, just complain --ignore-context do not use CMTUSERCONTEXT even if it should be used --list-versions print available versions of the specified project and exit (all other options are ignored) --build-env sets only the build time environment for the project --external-only sets only the environment for the externals (the project is used only to select the version of LCG) --dev prepend $LHCBDEV to the search path. Note: the directories are searched in the order specified on the command line. --dev-dir DEVDIR prepend DEVDIR to the search path Note: the directories are searched in the order specified on the command line. -v VERSION must be used after the name of an external to specify a non default version for it --set-CMTPATH Set CMTPATH to the value used internally by CMT (DANGEROUS) --runtime-project PROJECT [VERSION] Add a project to the runtime environment --overriding-project PROJECT [VERSION] Add a project to override packages --no-auto-override Do not automatically prepend the projects [('ExtraPackages', [])] --use-grid Enable auto selection of LHCbGrid project -q, --silent Removes message printout during setup --keep-CMTPROJECTPATH Do not override the value of the environment variable CMTPROJECTPATH --nightly SLOT [DAY] Add the required slot of the LHCb nightly builds to the list of DEV dirs. DAY must be a 3 digit abbreviation of the weekday, by default the current day. Special settings of the CMTPROJECTPATH needed for the nightly build slot are taken into account.

How SetupProject works

>
>
lb-run LHCb latest CondDBBrowser but in the first case you have to exit from the shell to get back to a clean environment.
 
Changed:
<
<
>
>
Other important differences are
  • SetupProject does not understand the metadata produced with CMake builds and lb-run does not understand project built with CMT, but it falls back on SetupProject if needed
  • lb-run does not support (yet) some of the less used features of SetupProject, mainly adding external libraries
  -- MarcoClemencic - 2015-01-15

Revision 22015-01-15 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

Tools for the LHCb Software Environment

Line: 42 to 42
 More options can be explored with lb-dev --help.

Runtime

Added:
>
>
The way we build and distribute our software require that we set several environment variables to run an application. This is taken care of by the script lb-run.
 
Changed:
<
<
The way we build and distribute our software require that we set several environment variables
>
>
lb-run behavior is modeled after the standard Unix command env (see man 1 env) with the addition of concepts like prepend and append, and the knowledge of LHCb software projects.

The simplest way to use it is

lb-run <Project> <version> <command> <command args...>
for example:
lb-run Brunel prod gaudirun.py MyBrunelOptions.py
 

Comparison with SetupProject

Development

Revision 12015-01-15 - MarcoClemencic

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="LHCbComputing"

Tools for the LHCb Software Environment

Introduction

This page is a description of the tools used to prepare the local development environment and to run LHCb applications.

Note: Because of the migration of the LHCb build system from CMT to CMake, we had to update the tools (so far represented by SetupProject) and we decided to try to improve the developer/user experience.

Disclaimer: all the versions of the projects presented in this page are those available at the moment of writing and there is no implicit or explicit suggestion about which version to use or not to use. They are just examples.

Development

Software development in LHCb is based on the development of subsets of the software on top of an existing release of a software project (see LHCbSoftwareTutorials).

The script lb-dev helps to prepare the local working directories ("local projects" or "satellite projects") for the developers.

The simplest way to call lb-dev is

lb-dev <Project> <version>
This will create a new subdirectory called Dev_ in the local directory and will create all the support files to be able to develop against .

Another very common use case is to build against the version of a project build in the LHCb Nightly Build System, for example:

lb-dev --nightly lhcb-gaudi-head Gaudi HEAD
or
lb-dev --nightly lhcb-gaudi-head Thu Gaudi HEAD
They will both create GaudiDev_HEAD bound to the nightly slot lhcb-gaudi-head, but the first one to the Today build and the second on the build of Thursday (this connection can be changes editing the file build.conf in GaudiDev_HEAD).

Very useful options of lb-def are

  • --name to specify the name of the local project, like in lb-dev --name MyAnalysis DaVinci prod
  • --dest-dir to specify the parent directory of the created local project (instead of '.')

More options can be explored with lb-dev --help.

Runtime

The way we build and distribute our software require that we set several environment variables

Comparison with SetupProject

Development

The main differences between SetupProject --build-env and lb-dev are:

  • lb-dev does not modify the runtime environment and does not call cd to switch to the created directory
  • the default version of a project used by lb-dev is prod, instead of picking up the latest one
  • by default SetupProject creates the local project in $User_release_area, while lb-dev creates it in the current (or specified) directory

Basic Usage

The simplest use-case one can imagine, is the configuration of the runtime environment for a released version of a project, so that one can call directly the application passing the preferred options, without having to check out from CVS the top level package of the application.

To be able to use SetupProject, you need the minimal LHCb environment that you get when you log on lxplus.cern.ch or after having sourced the ExtCMT script in a local installation.

The minimal information that you need to pass to SetupProject is the name of the application (or project) you want to use. Obviously you may want to specify the version, but if you do not do it, SetupProject will use the latest available. An example is:

[lxplus] ~ > SetupProject DaVinci
Configuring DaVinci v20r0 from /afs/cern.ch/lhcb/software/releases/DAVINCI/DAVINCI_v20r0
Using CMTPROJECTPATH = '/afs/cern.ch/user/m/marcocle/cmtuser:/afs/cern.ch/lhcb/software/releases:/afs/cern.ch/sw/Gaudi/releases:/afs/cern.ch/sw/lcg/app/releases'
Environment for DaVinci v20r0 ready.
(taken from DaVinci v20r0 from /afs/cern.ch/lhcb/software/releases/DAVINCI/DAVINCI_v20r0)

As you can see, the output is pretty self explanatory. You have the version of the application, the CMTPROJECTPATH used to locate dependent projects and where the binaries are actually taken from.

To specify the version of the application that you need, just add it to the command line:

[lxplus] ~ > SetupProject DaVinci v19r14
Configuring DaVinci v19r14 from /afs/cern.ch/lhcb/software/releases/DAVINCI/DAVINCI_v19r14
Using CMTPROJECTPATH = '/afs/cern.ch/user/m/marcocle/cmtuser:/afs/cern.ch/lhcb/software/releases:/afs/cern.ch/sw/Gaudi/releases:/afs/cern.ch/sw/lcg/app/releases'
Environment for DaVinci v19r14 ready.
(taken from DaVinci v19r14 from /afs/cern.ch/lhcb/software/releases/DAVINCI/DAVINCI_v19r14)

If you do not remember or do not know which versions are available, you have two choices: you can tell SetupProject to list the versions it can use or let it propose the list from which to chose a version. The list is produced with the command line option --list-versions, that can be abbreviated with --list:

[lxplus] ~ > SetupProject DaVinci --list
v12r18 in /afs/cern.ch/lhcb/software/releases
v14r5 in /afs/cern.ch/lhcb/software/releases
v19r5 in /afs/cern.ch/lhcb/software/releases
v19r6 in /afs/cern.ch/lhcb/software/releases
v19r7 in /afs/cern.ch/lhcb/software/releases
v19r8 in /afs/cern.ch/lhcb/software/releases
v19r9 in /afs/cern.ch/lhcb/software/releases
v19r10 in /afs/cern.ch/user/m/marcocle/cmtuser
v19r11 in /afs/cern.ch/lhcb/software/releases
v19r12 in /afs/cern.ch/lhcb/software/releases
v19r13 in /afs/cern.ch/user/m/marcocle/cmtuser
v19r14 in /afs/cern.ch/lhcb/software/releases
v20r0 in /afs/cern.ch/lhcb/software/releases

(you can notice that most of the proposed versions come from the standard release area, while two come from my cmtuser directory, the reason of it is explained later in this page). If you prefer to chose the version from a list of possibilities, the command line option is --ask:

[lxplus] ~ > SetupProject DaVinci --ask
Please enter your choice (v12r18 v14r5 v19r5 v19r6 v19r7 v19r8 v19r9 v19r10 v19r11 v19r12 v19r13 v19r14 v20r0 q[uit] [v20r0]): v19r8
Trying version 'v19r8'
Configuring DaVinci v19r8 from /afs/cern.ch/lhcb/software/releases/DAVINCI/DAVINCI_v19r8
Using CMTPROJECTPATH = '/afs/cern.ch/user/m/marcocle/cmtuser:/afs/cern.ch/lhcb/software/releases:/afs/cern.ch/sw/Gaudi/releases:/afs/cern.ch/sw/lcg/app/releases'
Environment for DaVinci v19r8 ready.
(taken from DaVinci v19r8 from /afs/cern.ch/lhcb/software/releases/DAVINCI/DAVINCI_v19r8)

In this case, at the prompt, you can press enter without typing anything to accept the default (equivalent to a call to SetupProject without explicit version), type q to avoid changes to your environment or select a different version.

You may need to try versions that are not in production yet, like the build available in the special release are called LHCBDEV. Since SetupProject doesn't propose those version by default to avoid possible problems, you can enable LHCBDEV with the command line option --dev:

[lxplus] ~ > SetupProject Panoramix --list
v9r8 in /afs/cern.ch/lhcb/software/releases
v15r8 in /afs/cern.ch/lhcb/software/releases
v15r9 in /afs/cern.ch/lhcb/software/releases
v15r10 in /afs/cern.ch/lhcb/software/releases
v15r11 in /afs/cern.ch/lhcb/software/releases
v15r13 in /afs/cern.ch/lhcb/software/releases
v15r14 in /afs/cern.ch/lhcb/software/releases
v15r15 in /afs/cern.ch/lhcb/software/releases
[lxplus] ~ > SetupProject Panoramix --dev --list
v9r8 in /afs/cern.ch/lhcb/software/releases
v15r8 in /afs/cern.ch/lhcb/software/releases
v15r9 in /afs/cern.ch/lhcb/software/releases
v15r10 in /afs/cern.ch/lhcb/software/releases
v15r11 in /afs/cern.ch/lhcb/software/releases
v15r13 in /afs/cern.ch/lhcb/software/releases
v15r14 in /afs/cern.ch/lhcb/software/releases
v15r15 in /afs/cern.ch/lhcb/software/releases
v15r16 in /afs/cern.ch/lhcb/software/DEV
v16r0 in /afs/cern.ch/lhcb/software/DEV

Note that the version that would be taken by default is the last one of the list.

There is another way of using a DEV directory, --dev-dir, which allows you to specify something different from the standard LHCBDEV or more than one "dev" directory (not a very common use case, I agree). An example of a dev directory is the location of the LHCb nightly builds:

[lxplus] ~ > SetupProject Brunel --dev-dir /afs/cern.ch/lhcb/software/nightlies/lhcb2/Wed
Configuring Brunel HEAD from /afs/cern.ch/lhcb/software/nightlies/lhcb2/Wed/BRUNEL/BRUNEL_HEAD
Using CMTPROJECTPATH = '/afs/cern.ch/lhcb/software/nightlies/lhcb2/Wed:/afs/cern.ch/user/m/marcocle/cmtuser:/afs/cern.ch/lhcb/software/releases:/afs/cern.ch/sw/Gaudi/releases:/afs/cern.ch/sw/lcg/app/releases'
Environment for Brunel HEAD ready.
(taken from Brunel HEAD from /afs/cern.ch/lhcb/software/nightlies/lhcb2/Wed/BRUNEL/BRUNEL_HEAD)

User Local Projects

The supported way (within LHCb) of developing software in the context of an application/project is to create a "user project" in which you can check out packages from CVS or create your new ones. You do not need a deep knowledge of the internal structure of projects managed with CMT, SetupProject will do the dirty job of creating the project directory if you use the option --build-env. You have to specify the name of the project and the version you want to use. If the version is omitted, SetupProject will ask which one to use (this differs from the behavior described before to maintain the same behavior of a previous tool used for the same purpose). Example:

[lxplus] ~ > SetupProject LHCb --build-env v25r0
Build-time environment for LHCb v25r0 ready.
Created user project in /afs/cern.ch/user/m/marcocle/cmtuser/LHCb_v25r0
Current directory is '/afs/cern.ch/user/m/marcocle/cmtuser/LHCb_v25r0'.
Using CMTPROJECTPATH = '/afs/cern.ch/user/m/marcocle/cmtuser:/afs/cern.ch/lhcb/software/releases:/afs/cern.ch/sw/Gaudi/releases:/afs/cern.ch/sw/lcg/app/releases'
[lxplus] ~/cmtuser/LHCb_v25r0 > 

After the call to SetupProject, you will be in the directory of the local project, ready to call getpack or to create your package. If the project directory is already present, it will not be modified (you can tell the difference because SetupProject will not print the line Created user project in ...). The location in which the user project is created is defined by the variable User_release_area, if you want the project to be created somewhere else, just change its value (the default is ~/cmtuser).

It must be clear that when called with --build-env, SetupProject cannot prepare a runtime environment. The reason is simple: if you want to build some code, most probably the libraries you may want to use do not exist yet in your local project, so there is nothing you can use yet.

To prepare the runtime environment using your user project, you don't need to do anything special: SetupProject always look for projects in your User_release_area, as you can notice with --list:

[lxplus] ~ > SetupProject LHCb --list
v23r5 in /afs/cern.ch/lhcb/software/releases
v23r6 in /afs/cern.ch/lhcb/software/releases
v23r7 in /afs/cern.ch/lhcb/software/releases
v24r0 in /afs/cern.ch/lhcb/software/releases
v24r1 in /afs/cern.ch/lhcb/software/releases
v25r0 in /afs/cern.ch/user/m/marcocle/cmtuser

The fact that the user project is used, is also clear when setting up the runtime environment:

[lxplus] ~ > SetupProject LHCb v25r0
Configuring LHCb v25r0 from /afs/cern.ch/user/m/marcocle/cmtuser/LHCb_v25r0
Using CMTPROJECTPATH = '/afs/cern.ch/user/m/marcocle/cmtuser:/afs/cern.ch/lhcb/software/releases:/afs/cern.ch/sw/Gaudi/releases:/afs/cern.ch/sw/lcg/app/releases'
Environment for LHCb v25r0 ready.
(taken from LHCb v25r0 from /afs/cern.ch/user/m/marcocle/cmtuser/LHCb_v25r0)

If you do not want that your local project is picked up, you have to unset User_release_area (or to rename the user project directory).

Advanced Usage

Use Non-standard Packages

When SetupProject is asked to prepare the runtime environment for a project, it uses all the packages that belong to the requested project plus those that are used by them, directly or indirectly. This means that setting up Brunel will not give you access to all the packages in LHCb.

There is at least one case where the behavior could be annoying: the CondDBBrowser. The graphical user interface to the Conditions Database needs some libraries that are made available only if you use the package Tools/CondDBUI, which is part of LHCb. Of course, you can do SetupProject LHCb and get the browser, but you may be working with Panoramix and do not want to leave of modify (maybe corrupt) the environment to be able to start the browser, so SetupProject gives you the possibility of getting the environment specific to packages that are not in the project with the option --use:

[lxplus] ~ > SetupProject Panoramix --use Tools/CondDBUI
Configuring Panoramix v15r15 from /afs/cern.ch/lhcb/software/releases/PANORAMIX/PANORAMIX_v15r15
Using CMTPROJECTPATH = '/afs/cern.ch/user/m/marcocle/cmtuser:/afs/cern.ch/lhcb/software/releases:/afs/cern.ch/sw/Gaudi/releases:/afs/cern.ch/sw/lcg/app/releases'
Environment for Panoramix v15r15 ready.
(taken from Panoramix v15r15 from /afs/cern.ch/lhcb/software/releases/PANORAMIX/PANORAMIX_v15r15)
[lxplus] ~ > echo $CONDDBUIROOT
/afs/cern.ch/lhcb/software/releases/LHCB/LHCB_v24r0/Tools/CondDBUI/v2r10

The usefulness of --use is not limited to add a package to the list of the used ones. With special projects like DBASE and PARAM, you have many versions of a package available at the same time, but only the latest one us used. If you need a special version of one of these packages, for example the version of DecFiles has to be fixed for productions of simulated data, you can add it to the use parameter:

[lxplus] ~ > SetupProject Gauss v33r2 --use "Gen/DecFiles v14r4"
Configuring Gauss v33r2 from /afs/cern.ch/lhcb/software/releases/GAUSS/GAUSS_v33r2
Using CMTPROJECTPATH = '/afs/cern.ch/user/m/marcocle/cmtuser:/afs/cern.ch/lhcb/software/releases:/afs/cern.ch/sw/Gaudi/releases:/afs/cern.ch/sw/lcg/app/releases'
Environment for Gauss v33r2 ready.
(taken from Gauss v33r2 from /afs/cern.ch/lhcb/software/releases/GAUSS/GAUSS_v33r2)
[lxplus] ~ > echo $DECFILESROOT
/afs/cern.ch/lhcb/software/releases/DBASE/Gen/DecFiles/v14r4

Please, note the quotes around "Gen/DecFiles v14r4": they are needed.

More Advanced usage

Quick Reference

Set up the runtime environment for a released application
  • SetupProject DaVinci
  • SetupProject Brunel v33r1
  • SetupProject Panoramix --ask
Set up the runtime environment for an application in the nightly builds
  • SetupProject DaVinci --nightly lhcb2
  • SetupProject DaVinci --nightly lhcb2 Wed
Prepare a user local project
  • SetupProject --build-env LHCb v25r0
  • setenvLHCb v25r0
    (an alias to the previous command)

Custom Project Search Path (CMTPROJECTPATH)

Since beginning of October 2008, SetupProject does not use anymore the defined CMTPROJECTPATH (the environment variable used to declare where to find projects), but it always constructs it from User_release_area (by default ~/cmtuser), LHCBPROJECTPATH (the default basic project search path) and the command line.

The reason for the change of behavior is needed because the old behavior could lead to some inconsistencies across calls to SetupProject. Of course, something similar to the old behavior is still possible with the command line option --keep-CMTPROJECTPATH.

Here I describe how to have SetupProject constructing the CMTPROJECTPATH for you. The basic logic is that the minimal CMTPROJECTPATH is given by the environment variable LHCBPROJECTPATH, the options to modify the CMTPROJECTPATH are prepended to it and the User_release_area must always be the first one to be searched. The options that modify the CMTPROJECTPATH are:

  • --dev
  • --dev-dir <dir>
  • --nightly <slot> [<day>]
The basic call to SetupProject, without any special option will set CMTPROJECTPATH to ${User_release_area}:${LHCBPROJECTPATH}.

The option --dev prepends to the CMTPROJECTPATH the content of the environment variable LHCBDEV, so that a call

SetupProject LHCb --dev

will set CMTPROJECTPATH to ${User_release_area}:${LHCBDEV}:${LHCBPROJECTPATH}.

--dev-dir is used to specify a different "dev" area, so --dev is actually equivalent to --dev-dir ${LHCBDEV}. The call

SetupProject LHCb --dev-dir /afs/cern.ch/user/m/marcocle/public

will result in the value ${User_release_area}:/afs/cern.ch/user/m/marcocle/public:${LHCBPROJECTPATH} for CMTPROJECTPATH.

The option --nightly allows to prepare the environment to use the standard LHCb nightly build directories. It accepts a mandatory argument defining the slot to use (e.g. lhcb1 or lhcb2) and an optional argument for the day (3 chars abbreviation), which is defaulted to the current day. With this option, SetupProject checks the existence of the configuration file of the nightly build and uses the information in it to prepare the CMTPROJECTPATH (needed for some special configuration of the nightly build slot). The call

SetupProject LHCb --nightly lhcb1

will set CMTPROJECTPATH to ${User_release_area}:/afs/cern.ch/lhcb/software/nightlies/lhcb1/Fri:...:${LHCBPROJECTPATH} (assuming today is Friday), where the ... will contain the special entries required by the slot.

The options described can be combined for a more complex CMTPROJECTPATH. The specified directories are added to the CMTPROJECTPATH in the order specified on the command line, after User_release_area and before LHCBPROJECTPATH. So from

SetupProject LHCb --dev-dir /afs/cern.ch/user/m/marcocle/public --nightly lhcb1 Mon --dev --dev-dir /afs/cern.ch/user/s/somebody/public

you should expect CMTPROJECTPATH featuring, in this order,

  • ${User_release_area}
  • /afs/cern.ch/user/m/marcocle/public
  • /afs/cern.ch/lhcb/software/nightlies/lhcb1/Mon
  • ${LHCBDEV}
  • /afs/cern.ch/user/s/somebody/public
  • ${LHCBPROJECTPATH}.

Caveats

There are 2 main usages for SetupProject: the fist one is to setup the full environment of a project and the second one is to prepare the environment for the software development. This second usage which is performed by using the --build-env option (or equivalently by using the alias setenvProject) doesn't setup the full environment. It only updates the environment variable LHCBPROJECTPATH.

This is all you need to build your software. For example, here is a normal usage of SetupProject for both development and run:

  1. Prepare the environment which the custom directory from the nightly builds:
                SetupProject --build-env --nightly lhcb2 DaVinci v24r0  (setenvProject --nightly lhcb2 DaVinci v24r0)
  2. checkout the package you would like to modifiy
                getpack MyPack
  3. Modify your package
  4. Build it
                cd  MyPack/cmt
                cmt make 
  5. Do the full setup to run it
                SetupProject --nightly lhcb2 DaVinci v24r0  

It is important that the build is done before the full setup is done. Otherwise you could end up with a package that doesn't build in the normal environment.

Bug reporting

Please report bugs to the SetupProject category of the lhcbscripts Savannah bug tracker

Appendix

SetupProject online help

This is the help message you can get calling SetupProject --help:

Usage: SetupProject.py [options] <project_name> [version|--ask] [options] [externals]

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  --site SITE           enable site specific defaults
  --ask                 ask for the version of the project to use (overrides
                        the version specified)
  --disable-CASTOR      remove CASTOR from the added dependencies
  --tag_add TAG_ADD     specify extra CMT tags
  --use USE             add a CMT use statement
  --verbose             be a bit more verbose
  --debug               output useful for debugging
  --ignore-missing      do not fail if some externals are missing, just
                        complain
  --ignore-context      do not use CMTUSERCONTEXT even if it should be used
  --list-versions       print available versions of the specified project and
                        exit (all other options are ignored)
  --build-env           sets only the build time environment for the project
  --external-only       sets only the environment for the externals (the
                        project is used only to select the version of LCG)
  --dev                 prepend $LHCBDEV to the search path. Note: the
                        directories are searched in the order specified on the
                        command line.
  --dev-dir DEVDIR      prepend DEVDIR to the search path Note: the
                        directories are searched in the order specified on the
                        command line.
  -v VERSION            must be used after the name of an external to specify
                        a non default version for it
  --set-CMTPATH         Set CMTPATH to the value used internally by CMT
                        (DANGEROUS)
  --runtime-project PROJECT [VERSION]
                        Add a project to the runtime environment
  --overriding-project PROJECT [VERSION]
                        Add a project to override packages
  --no-auto-override    Do not automatically prepend the projects
                        [('ExtraPackages', [])]
  --use-grid            Enable auto selection of LHCbGrid project
  -q, --silent          Removes message printout during setup
  --keep-CMTPROJECTPATH
                        Do not override the value of the environment variable
                        CMTPROJECTPATH
  --nightly SLOT [DAY]  Add the required slot of the LHCb nightly builds to
                        the list of DEV dirs. DAY must be a 3 digit
                        abbreviation of the weekday, by default the current
                        day. Special settings of the CMTPROJECTPATH needed for
                        the nightly build slot are taken into account.

How SetupProject works

-- MarcoClemencic - 2015-01-15

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback