EditAttachPDF

In the following, we describe how to get started with the CORAL and COOL software, either using one of the releases pre-installed for ATLAS and LHCb on AFS, or rebuilding the software from scratch against the external libraries installed on AFS and CVMFS. CORAL and COOL are supported on Linux SLC5/SLC6 and Mac OSX using a variety of different compilers. To set up the runtime environment (and a fortiori to rebuild from scratch), the CMT configuration management tool is needed (see further below on this twiki page). The recipes described in the following are given for the tcsh shell, but equivalent recipes also exist for the bash shell family. A simple script CMT_env.csh bundled with both CORAL and COOL is used to set up CMT for the software: before sourcing it, please remember to choose a build platform and compiler by setting the CMTCONFIG environment variable. All of the following examples are described for gcc46 64bit debug builds on SLC6 (i.e. using setenv CMTCONFIG x86_64-slc6-gcc46-dbg): please logon lxplus6 or another SLC6 machine with AFS and CVMFS to follow these recipes.

# Introduction to CMT and LCGCMT

CMT is the configuration management tool that is used internally by the CORAL and COOL development team to build and test the software. It is also used by the developers and end users of ATLAS and LHCb to build their own software frameworks, integrating into them CORAL, COOL and many other external dependencies.

The 'glue' integrating the CMT setups in CORAL and COOL with those used in ATLAS and LHCb is provided by a special package called LCGCMT. The LCGCMT packages defines a set of common configuration policies which apply also to the way CMT is used internally by the CORAL and COOL development team.

CMT is used by CORAL and COOL developers for three purposes primarily:

• to define (and eventually analyze) the internal and external dependencies of the Persistency components on one another and on other software packages
• to define the build rules and targets (and eventually to build the software)
• to define the runtime environment (and eventually to set up all environment variables)

# Use a pre-installed release of CORAL or COOL on AFS

In the context of the CORAL and COOL release process, whenever new software versions are released for ATLAS and LHCb, both packages are built and tested with CMT by the SPI team using the nightly build infrastructure. The resulting libraries, binaries and scripts are then installed in the AFS release area on /afs/cern.ch/sw/lcg/app/releases. ATLAS and LHCb take the CORAL and COOL software already built from the AFS release area and integrate it into their build and runtime infrastructure using CMT. The process is different in CMS, where the the CORAL source code is built internally using another configuration management tool, scram (this is the reason why scram BuildFile's can be found in CORAL).

If you are a member of ATLAS or LHCb and want to set up CMT to use the CORAL and COOL software as an external user (i.e. without rebuilding the code from scratch!), you may refer to the documentation provided by your own experiment:

Alternatively, you may use the following recipe to setup the CMT runtime environment directly from the CORAL or COOL releases installed on AFS (these releases also exist on CVMFS, but the CMT setup scripts preinstalled in these release areas point to AFS and are not relocatable - while all libraries and binaries are perfectly relocatable if the environment is setup externally, e.g. from the ATLAS and LHCb software releases). The following shows how to use the %MyCoralRel% and %MyCoolRel% versions that were released for the latest %MyLcgcmtRel% configuration.

## Set up the CMT runtime environment for a CORAL release on AFS

To set up CORAL:

  ssh lxplus.cern.ch
tcsh
setenv CMTCONFIG %MyCmtconfig%
cd /afs/cern.ch/sw/lcg/app/releases/CORAL/%MyCoralRel%/src/config/cmt
source CMT_env.csh
source setup.csh

You may then execute a simple test
  test_unit_CoralBase_CoralBaseTest


## Set up the CMT runtime environment for a COOL release on AFS

To set up COOL:

  ssh lxplus.cern.ch
tcsh
setenv CMTCONFIG %MyCmtconfig%
cd /afs/cern.ch/sw/lcg/app/releases/CORAL/%MyCoolRel%/src/config/cmt
source CMT_env.csh
source setup.csh

You may then execute a simple test
  test_CoolKernel_RecordSelection


Note that setting up COOL automatically sets up the correct LD_LIBRARY_PATH for using the corresponding CORAL libraries, however it will include COOL-specific tests in the runtime PATH. If you want to execute CORAL tests, please set up CORAL instead so that those tests end up in the runtime PATH.

# Rebuild CORAL and/or COOL from scratch using the externals installed on AFS or CVMFS

The following describes all steps necessary for downloading, building and running CORAL and COOL from scratch using CMT on Linux. These recipes were tested using an account that is not normally used for CMT-based developments, so that no hidden assumption should be present. We will use the %MyCoralTag% and %MyCoolTag% tags in SVN to download the current release candidate as tested in the nightlies (feel free to use a different tag or the trunk if you know what you are doing). We assume that CORAL and COOL are installed in subdirectories of a private directory $MYLCG (that you may for instance configure by adding setenv MYLCG /home/valassi to your .cshrc), either on AFS or (better) on a local disk. ## Rebuild CORAL from scratch on AFS against %MyLcgcmtRel% To download CORAL from SVN:  ssh lxplus.cern.ch tcsh mkdir -p$MYLCG/CORAL/%MyCoralTag%
cd $MYLCG/CORAL/%MyCoralTag% svn co svn+ssh://svn.cern.ch/reps/lcgcoral/coral/tags/%MyCoralTag% src  You should then set up CMT for building (you only need to do this once) by sourcing the CMT_env.csh script (after having chosen your preferred CMTCONFIG). By default this script will look for external software packages under AFS. For faster builds, it is better to to look for external software packages under CVMFS, in particular from /cvmfs/sft.cern.ch/, which you can achieve by having a file called usecvmfs in your config/cmt directory (WARNING: this presently fails because of issue SPI-328). The version of the externals depends on the LCGCMT configuration for your build (determined by file$MYLCG/CORAL/%MyCoralTag%/src/cmt/project.cmt=, in this case LCGCMT_66) and by the location of the LCGCMT package itself (determined by the CMTPROJECTPATH environment variable, configured by the CMT_env.csh script. The following commands set up CMT using externals from CVMFS, build the CORAL-specific setup scripts and rebuild the whole of CORAL from scratch:

  setenv CMTCONFIG %MyCmtconfig%
cd $MYLCG/CORAL/%MyCoralTag%/src/config/cmt ###\cp usecvmfsSFT usecvmfs # This presently leads to failures because of SPI-328 source CMT_env.csh cmt config ./cmtBroadcast.sh cmt make all_groups  Having done that, you can execute the CORAL-specific setup script and try to execute a simple test:  setenv CMTCONFIG %MyCmtconfig% cd$MYLCG/CORAL/%MyCoralTag%/src/config/cmt
source setup.csh
test_unit_CoralBase_CoralBaseTest


## Rebuild COOL from scratch on AFS against %MyLcgcmtRel%

  ssh lxplus.cern.ch
tcsh
mkdir -p $MYLCG/COOL/%MyCoolTag% cd$MYLCG/COOL/%MyCoolTag%
svn co svn+ssh://svn.cern.ch/reps/lcgcool/cool/tags/%MyCoolTag% src


As explained above for CORAL, the following commands set up CMT using externals from CVMFS, build the COOL-specific setup scripts and rebuild the whole of COOL from scratch:

  setenv CMTCONFIG %MyCmtconfig%
cd $MYLCG/COOL/%MyCoolTag%/src/config/cmt ###\cp usecvmfsSFT usecvmfs # This presently leads to failures because of SPI-328 source CMT_env.csh cmt config ./cmtBroadcast.sh cmt make all_groups  Having done that, you can execute the COOL-specific setup script and try to execute a simple test:  setenv CMTCONFIG %MyCmtconfig% cd$MYLCG/COOL/%MyCoolTag%/src/config/cmt
source setup.csh
test_CoolKernel_RecordSelection


Note that in this example %MyCoolTag% is being rebuilt from scratch against all externals from %MyLcgcmtRel%, including CORAL! In other words, the official %MyCoralRel% release is used instead of the private build of %MyCoralTag% described above. To use a different version of CORAL or other externals (such as ROOT), the CMT configuration files must be tweaked, for instance using the approach described in the next section for building COOL against the nightlies.

# Rebuild COOL from scratch using the externals from the nightly builds

The following describes all steps necessary for downloading, building and running COOL from scratch, using externals from the ROOT6 slot of the nightly builds. One motivation for documenting this recipe is to allow developers from the ROOT team to build and test PyCool against the beta version of ROOT6 (see task #47955). While this can be done in a variety of way, the following recipe is based on a few specific choices:

• Externals (here CORAL, LCGCMT and ROOT6) are copied from AFS to a private directory (because the nightly builds are erased every seven days and this approach guarantees longer availability).
• A new platform %MyCmtconfig6% is introduced and used (to allow parallel builds for the default %MyCmtconfig% in the same directory). Note that gcc48 is used here because ROOT6 is only built for this compiler in the nightlies.
• All CMT tweaks are configured using the CMTUSERCONTEXT feature of CMT.

First, download COOL from SVN. We use the %MyCoolHead% tag here instead of the %MyCoolTag% tag to reproduce the results of the dev2 nightly slot testing the beta version of ROOT6.

  ssh lxplus.cern.ch
tcsh
mkdir -p $MYLCG/COOL/%MyCoolHead% cd$MYLCG/COOL/%MyCoolHead%


Then, copy the latest CORAL, LCGCMT and ROOT6 nightlies to a private directory (here %MyNightlies% is assumed on a private SLC6 node). If you do not want to change your cmt/project.cmt file (for instance because you are using the same installation also against the official externals for that LCGCMT release), you must also create a new %MyLcgcmtRel% directory. Alternatively, you can also work with the nightlies on AFS directly, in which case please change all paths accordingly in your CMT configuration files below (and you may need to change the cmt/project.cmt file too).

  \rm -rf %MyNightlies%
mkdir -p %MyNightlies%/LCGCMT
rsync -rlptDPu --exclude=InstallArea %MyNightliesAfs%/LCGCMT/LCGCMT_root6/ \
%MyNightlies%/LCGCMT/LCGCMT_root6
mkdir -p %MyNightlies%/LCGCMT/%MyLcgcmtRel%
pushd %MyNightlies%/LCGCMT/%MyLcgcmtRel%; ln -sf ../LCGCMT_root6/* .; mkdir cmt.new
cat cmt/project.cmt | sed "s/LCGCMT_root6/%MyLcgcmtRel%/" > cmt.new/project.cmt
\rm -f cmt; \mv cmt.new cmt; popd
mkdir -p %MyNightlies%/CORAL/CORAL-preview
rsync -rlptDPu %MyNightliesAfs%/CORAL/CORAL-preview/include/ \
%MyNightlies%/CORAL/CORAL-preview/include
rsync -rlptDPu %MyNightliesAfs%/CORAL/CORAL-preview/src/ \
%MyNightlies%/CORAL/CORAL-preview/src
rsync -rlptDPu %MyNightliesAfs%/CORAL/CORAL-preview/%MyCmtconfig%/ \
%MyNightlies%/CORAL/CORAL-preview/%MyCmtconfig%
mkdir -p %MyNightlies%/ROOT/ROOT_today
rsync -rlptDPu %MyNightliesAfs%/ROOT/ROOT_today/%MyCmtconfig%/ \
%MyNightlies%/ROOT/ROOT_today/%MyCmtconfig%


You should then tweak the CMT configuration so that COOL uses the CORAL, LCGCMT and ROOT6 copied from the nightlies. The COOL CMT_env.csh script assumes that your private configuration files are in a subdirectory config/cmt/USERCONTEXT/$USER (defined by environment variable $CMTUSERCONTEXT) whose name depends on your username. You should create this directory and the three files requirements, CMT_userenv.csh, CMT_userenv.sh as follows.

  bash
mkdir -p $MYLCG/COOL/%MyCoolHead%/src/config/cmt/USERCONTEXT/$USER
\rm -f $MYLCG/COOL/%MyCoolHead%/src/config/cmt/USERCONTEXT/$USER/CMT_userenv.csh
cat << EOF > $MYLCG/COOL/%MyCoolHead%/src/config/cmt/USERCONTEXT/$USER/CMT_userenv.csh
if ( "\$CMTCONFIG" == "%MyCmtconfig6%" ) then setenv CMTPROJECTPATH %MyNightlies% endif EOF \rm -f$MYLCG/COOL/%MyCoolHead%/src/config/cmt/USERCONTEXT/$USER/CMT_userenv.sh cat << EOF >$MYLCG/COOL/%MyCoolHead%/src/config/cmt/USERCONTEXT/$USER/CMT_userenv.sh if [ "\$CMTCONFIG" == "%MyCmtconfig6%" ]; then
export CMTPROJECTPATH=%MyNightlies%
fi
EOF
\rm -f $MYLCG/COOL/%MyCoolHead%/src/config/cmt/USERCONTEXT/$USER/requirements
cat << EOF > $MYLCG/COOL/%MyCoolHead%/src/config/cmt/USERCONTEXT/$USER/requirements
tag x86_64-slc6-gcc48-dbg-root6 target-x86_64 target-slc6 target-gcc48 target-dbg ROOT_GE_6_00
macro LCG_external \$(SITEROOT)/sw/lcg/external target-gcc48 \$(SITEROOT)/sw/lcg/experimental
macro CORAL_config_version \$(CORAL_config_version) %MyCmtconfig6% CORAL-preview macro CORAL_base \$(CORAL_base) %MyCmtconfig6% %MyNightlies%/CORAL/\$(CORAL_config_version) macro ROOT_config_version \$(ROOT_config_version) %MyCmtconfig6% ROOT_today
macro ROOT_base \$(ROOT_base) %MyCmtconfig6% %MyNightlies%/ROOT/\$(ROOT_config_version)
EOF
exit
cat $MYLCG/COOL/%MyCoolHead%/src/config/cmt/USERCONTEXT/$USER/CMT_userenv.csh
cat $MYLCG/COOL/%MyCoolHead%/src/config/cmt/USERCONTEXT/$USER/CMT_userenv.sh
cat $MYLCG/COOL/%MyCoolHead%/src/config/cmt/USERCONTEXT/$USER/requirements


You can then set up CMT, build the COOL-specific setup scripts, rebuild the whole of COOL from scratch and source the COOL-specific setup script:

  \rm -rf $MYLCG/COOL/%MyCoolHead%/%MyCmtconfig6% \rm -rf$MYLCG/COOL/%MyCoolHead%/src/*/%MyCmtconfig6%
\rm -rf $MYLCG/COOL/%MyCoolHead%/src/*/*/%MyCmtconfig6% setenv CMTCONFIG %MyCmtconfig6% cd$MYLCG/COOL/%MyCoolHead%/src/config/cmt
source CMT_env.csh
cmt config
source setup.csh


Having done that, you are now ready to try and execute some simple tests, such as:

  test_CoolKernel_RecordSelection

or
  python -c 'import PyCool'

or
  python $MYLCG/COOL/%MyCoolHead%/src/PyCool/tests/test_coralAttributeList.py  or  python$MYLCG/COOL/%MyCoolHead%/src/PyCool/tests/test_coralBlob.py

or
  setenv COOLTESTDB "sqlite://;schema=/tmp/sqliteTest-${USER}-COOLTEST.db;dbname=COOLTEST" python$MYLCG/COOL/%MyCoolHead%/src/PyCool/tests/test_IFolder.py


If you just need to rebuild PyCool (e.g. because you made changes to ROOT), it is enough to execute the following:

  cd \$MYLCG/COOL/%MyCoolHead%/src/PyCool/cmt
cmt make all_groups


-- AndreaValassi - 03-Oct-2013

Edit | Attach | Watch | Print version |  | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r8 - 2013-10-29 - AndreaValassi

 Cern Search TWiki Search Google Search Persistency All webs
Copyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback