In the following, we describe how to build the CORAL and COOL software using CMake and how to set up the runtime environment and run some simple tests using the CMake-based infrastructure.

Note the following about CMake and CMT.

  • Before 2015, CMT was the default way to build and configure CORAL and COOL. Please refer to the OLDPersistencyCMT page for legacy information about CMT for CORAL and COOL.
  • As of 2015, CMake has become the default way to build and configure CORAL and COOL. CMake is used for all new (ROOT6-based) releases.
  • In August 2016, CMake builds for CORAL/COOL and their integration with lcgcmake have been significantly modified (CORALCOOL-2846). This page only describes this new infrastructure.

Except for a few details, which will be pointed out below, the CMake build infrastructure and all relevant scripts are absolutely identical in CORAL and COOL. For this reason, in the following all external pointers to code fragments will refer to the CORAL trunk in SVN, although they could equivalently be taken from the COOL trunk in SVN.

The recipes described below are given for the bash shell. In most cases the csh shell family could be used as an alternative, but this is discouraged.

Table of contents

1 Introduction

The CORAL and COOL software projects include components written in C++ and Python. Both projects have dependencies on several external software packages, such as Boost.

In order to build CORAL and COOL using CMake, and/or to set up their runtime environment to run applications, several parameters must be specified, including:

  • the C++ compiler version, and the appropriate build options
  • the versions and installation paths for all external dependencies (including Python)

Different build scenarios exist, depending on how the C++ compiler, external dependencies and other parameters are resolved. Broadly speaking, two cases will be considered in the following:

  • builds of CORAL and COOL _standalone_ against existing installations of all external dependencies, e.g. on AFS or CVMFS (this is the build type preferred for internal development)
  • builds of CORAL and COOL _embedded_ within a full build of the LCG software stack using lcgcmake (this is the build type used for production releases and for automatic nightly builds)

Before diving into the CMake build system for CORAL and COOL, the following sub-sections give a few more details about the LCG software stack and the external dependencies of CORAL and COOL.

1.1 The LCG software stack

The LCG software stack is a self consistent set of package versions, providing a foundation layer on top of which the ATLAS and LHCb experiment software stacks are built.

Each version of the stack is identified by a unique string beginning by "LCG_". At the time of writing in January 2017, the most recent released version of the LCG stack is LCG_87.

The stack includes CORAL and COOL and all of their external dependencies, plus many other packages. The full list of stacks available and the detailed contents of each stack can be seen on http://lcgsoft.web.cern.ch.

1.1.1 BINARY_TAG

Each version of the stack is built and installed for one or more supported platforms. Each platform is identified by a single string BINARY_TAG (e.g. x86_64-slc6-gcc49-dbg) that corresponds to:

  • a specific architecture (e.g. 64-bit x86 hosts for x86_64-slc6-gcc62-dbg)
  • a specific O/S (e.g. SLC6 for x86_64-slc6-gcc62-dbg)
  • a specific C++ compiler (e.g. gcc 6.2 for x86_64-slc6-gcc62-dbg)
  • a specific build type (e.g. debug build options for x86_64-slc6-gcc62-dbg)

Following the convention used for the LCG software stack, the required version of Python is not part of BINARY_TAG, but is instead defined by the specific version of the LCG stack: for instance, LCG_87 uses Python 2.7.10.

1.1.2 Installation on AFS and CVMFS

Whenever requested by ATLAS or LHCb, a new version of the LCG stack is built (using lcgcmake) and the corresponding binaries are installed on AFS or CVMFS (using lcgjenkins) by the CERN EP-SFT group.

Different versions of the stack, and individual packages within them, are installed at the following locations on AFS and CVMFS:

  • on CVMFS, LCG_xx is under /cvmfs/sft.cern.ch/lcg/releases/<LCG_xx>/ (e.g. /cvmfs/sft.cern.ch/lcg/releases/LCG_87/)
  • on AFS, LCG_xx is under /afs/cern.ch/sw/lcg/releases/<LCG_xx>/ (e.g. /afs/cern.ch/sw/lcg/releases/LCG_87/)
  • individual packages are under <LCG_xx>/<pkg>/<pkg_version>/<BINARY_TAG>/ (e.g. /cvmfs/sft.cern.ch/lcg/releases/LCG_87/CORAL/3_1_6/x86_64-slc6-gcc62-dbg/)

Note that installations on AFS will soon cease and be replaced by installations on EOS instead.

A given version of an individual package can be built, with different external dependencies, in different LCG software stacks. For this reason, a unique hash is assigned to the combination of a package version and the versions of all its external dependencies. Software builds and installations for a given package on a given platform are only performed for package combinations with distinct hashes, while combinations with the same hash can be reused across different LCG stacks. The installation of a given package under <LCG_xx>/<pkg>/<pkg_version>/<BINARY_TAG>/ on AFS or CVMFS is actually a symbolic link to the distinct installation of that package hash on the given platform.

For instance, the CORAL 3_1_0 builds and installation for LCG_81b, LCG81c, LCG_81d (three stacks differing only in their version of ROOT, which is not a CORAL dependency) are the same, while those for LCG_81e and LCG_81f are different (because a new frontier_client version was introduced in LCG_81e, forcing a rebuild of CORAL):

ls -ld /cvmfs/sft.cern.ch/lcg/releases/LCG_81*/CORAL/3_1_0/x86_64-slc6-gcc49-opt
  lrwxrwxrwx. 1 cvmfs cvmfs 50 Feb 10  2016 /cvmfs/sft.cern.ch/lcg/releases/LCG_81b/CORAL/3_1_0/x86_64-slc6-gcc49-opt -> ../../.././CORAL/3_1_0-36f4b/x86_64-slc6-gcc49-opt/
  lrwxrwxrwx. 1 cvmfs cvmfs 50 Feb  3  2016 /cvmfs/sft.cern.ch/lcg/releases/LCG_81c/CORAL/3_1_0/x86_64-slc6-gcc49-opt -> ../../.././CORAL/3_1_0-36f4b/x86_64-slc6-gcc49-opt/
  lrwxrwxrwx. 1 cvmfs cvmfs 50 Mar 18 16:37 /cvmfs/sft.cern.ch/lcg/releases/LCG_81d/CORAL/3_1_0/x86_64-slc6-gcc49-opt -> ../../.././CORAL/3_1_0-36f4b/x86_64-slc6-gcc49-opt/
  lrwxrwxrwx. 1 cvmfs cvmfs 50 Apr 13 10:12 /cvmfs/sft.cern.ch/lcg/releases/LCG_81e/CORAL/3_1_0/x86_64-slc6-gcc49-opt -> ../../.././CORAL/3_1_0-7af7f/x86_64-slc6-gcc49-opt/
  lrwxrwxrwx. 1 cvmfs cvmfs 50 Jun 24 17:43 /cvmfs/sft.cern.ch/lcg/releases/LCG_81f/CORAL/3_1_0/x86_64-slc6-gcc49-opt -> ../../.././CORAL/3_1_0-7af7f/x86_64-slc6-gcc49-opt/

One rpm is also created for each package in the stack, for each hash of that package and for each platform. For instance, a single rpm CORAL-36f4b_3_1_0_x86_64_slc6_gcc49_opt-1.0.0-81.noarch.rpm was created for the CORAL 3_1_0 builds in LCG_81b, LCG_81c and LCG_81d, described above. All rpms created so far with this procedure can be downloaded from http://service-spi.web.cern.ch/service-spi/external/rpms/lcg.

1.1.3 LCG summary files for AFS and CVMFS installations

For each version of the LCG software stack and for each platform, a single file on AFS or CVMFS provides the full list of package versions/hashes in the stack, plus some additional details about the compilation environment used for the build:

  • this "LCG summary file" can be found under <LCG_xx>/LCG_externals_<BINARY_TAG>.txt (e.g. /cvmfs/sft.cern.ch/lcg/releases/LCG_87/LCG_externals_x86_64-slc6-gcc62-dbg.txt)
  • the first line contains the BINARY_TAG (e.g. PLATFORM: x86_64-slc6-gcc62-dbg)
  • the second line contains the LCG_xx version tag (e.g. VERSION: 87)
  • the third line contains the full version number of the C++ compiler (e.g. COMPILER: gcc;6.2.0)
  • the subsequent lines contain, one by one, details about every package in the stack and its dependencies

For instance, the CORAL 3_1_0 builds and installation for the LCG_81* releases discussed above are described in the following way in the corresponding LCG summary files:

grep '^CORAL' /cvmfs/sft.cern.ch/lcg/releases/LCG_81*/LCG_externals_x86_64-slc6-gcc49-opt.txt
  /cvmfs/sft.cern.ch/lcg/releases/LCG_81b/LCG_externals_x86_64-slc6-gcc49-opt.txt:CORAL; 36f4b;  3_1_0; ./CORAL/3_1_0/x86_64-slc6-gcc49-opt; XercesC-8ccd5,sqlite-4b60e,valgrind-6e9ea,Python-df007,Boost-71952,oracle-e33b7,igprof-dc060,libunwind-76996,mysql-c4d2c,CppUnit-31a6a,QMtest-68574,Frontier_Client-51b5a,expat-6c8ce,tcmalloc-f5a5c
  /cvmfs/sft.cern.ch/lcg/releases/LCG_81c/LCG_externals_x86_64-slc6-gcc49-opt.txt:CORAL; 36f4b;  3_1_0; ./CORAL/3_1_0/x86_64-slc6-gcc49-opt; XercesC-8ccd5,sqlite-4b60e,valgrind-6e9ea,Python-df007,Boost-71952,oracle-e33b7,igprof-dc060,libunwind-76996,mysql-c4d2c,CppUnit-31a6a,QMtest-68574,Frontier_Client-51b5a,expat-6c8ce,tcmalloc-f5a5c
  /cvmfs/sft.cern.ch/lcg/releases/LCG_81d/LCG_externals_x86_64-slc6-gcc49-opt.txt:CORAL; 36f4b;  3_1_0; ./CORAL/3_1_0/x86_64-slc6-gcc49-opt; XercesC-8ccd5,sqlite-4b60e,valgrind-6e9ea,Python-df007,Boost-71952,oracle-e33b7,igprof-dc060,libunwind-76996,mysql-c4d2c,CppUnit-31a6a,QMtest-68574,Frontier_Client-51b5a,expat-6c8ce,tcmalloc-f5a5c
  /cvmfs/sft.cern.ch/lcg/releases/LCG_81e/LCG_externals_x86_64-slc6-gcc49-opt.txt:CORAL; 7af7f;  3_1_0; ./CORAL/3_1_0/x86_64-slc6-gcc49-opt; XercesC-8ccd5,sqlite-4b60e,valgrind-6e9ea,Python-df007,Boost-71952,oracle-e33b7,igprof-dc060,libunwind-76996,mysql-c4d2c,CppUnit-31a6a,Frontier_Client-0bd09,QMtest-68574,expat-6c8ce,tcmalloc-f5a5c
  /cvmfs/sft.cern.ch/lcg/releases/LCG_81f/LCG_externals_x86_64-slc6-gcc49-opt.txt:CORAL; 7af7f; 3_1_0; ./CORAL/3_1_0/x86_64-slc6-gcc49-opt; XercesC-8ccd5,sqlite-4b60e,valgrind-6e9ea,Python-df007,Boost-71952,oracle-e33b7,igprof-dc060,libunwind-76996,mysql-c4d2c,CppUnit-31a6a,Frontier_Client-0bd09,QMtest-68574,expat-6c8ce,tcmalloc-f5a5c

1.2 External dependencies

CORAL and COOL have many dependencies on other "external" packages at build time and/or at run time. In the LCG Application Area build infrastructure, some of these are included in the LCG software stack, while others are expected to be found in the underlying operating system.

1.2.1 Dependencies included in the LCG stack

The full set of external dependencies needed to build and configure CORAL and COOL production releases within the LCG software stack are those specified within the lcgcmake projects/CMakeLists.txt file (which, for every release, ultimately determines the CORAL and COOL dependencies displayed in the "LCG summary file" described above).

Some dependencies are mandatory, leading to build errors if the corresponding external package cannot be found. Other dependencies are optional and should not lead to build failures (but this may still need some cleanup).

CORAL has the following external dependencies at build and run time:

  • Boost (mandatory foundation library)
  • CppUnit (needed for building and running tests)
  • expat (needed by Frontier_client)
  • Frontier_Client (needed by the CORAL FrontierAccess database plugin)
  • mysql (needed by the CORAL MySQLAccess database plugin)
  • oracle (needed by the CORAL OracleAccess database plugin)
  • Python (mandatory for scripts and needed by PyCoral)
  • QMtest (needed for running tests)
  • sqlite (needed by the CORAL SQLiteAccess database plugin)
  • XercesC (mandatory for the authentication and DB lookup XML plugins)

In addition, the following packages are also declared as CORAL external dependencies, although they are only needed at runtime for debugging purposes:

  • igprof (needed to run IgProf)
  • libunwind (needed by igprof)
  • gperftools (needed to run the Google performance tools) - this was previously known as tcmalloc (the name was changed in LCG_84)
  • valgrind (needed to run valgrind)

COOL has the following external dependencies at build and run time:

  • CORAL (mandatory, with all of its external dependencies)
  • Qt (needed to run the ACE graphical editor)
  • ROOT (mandatory to use PyCool)

As of LCG_85, two packages that are used to drive the build process are also included as lcgcmake dependencies of CORAL and COOL:

  • ccache (optional build cache to speed up compilation and link)
  • ninja (optional alternative to GNU make to speed up dependency resolution of build targets)

1.3 O/S pre-requirements

In addition to the external dependencies listed above and included in the LCG stack, many other packages are needed by CORAL and COOL at build time and/or run time, either directly or because some of their primary dependencies need them as their own dependencies. One such example is libaio, which is needed at runtime by the Oracle client libraries, and which is not installed by default on many operating systems in their most minimalistic deployments.

Within the LCG Application Area, for many years this issue has been addressed on Linux by requiring these packages (such as libaio) to be installed as part of the operating system. This has been achieved by including them in the list of requirements of the HEP_OSlibs meta-rpm. HEP_OSlibs was introduced in 2009 to represent the minimal set of packages needed by the LHC experiment software applications on top of bare RedHat Linux systems (SLC5, SLC6, CentOS7) and its contents have been regularly discussed and agreed with representatives of the four LHC experiments. In practice, the SLC6 represented by a BINARY_TAG such as x86_64-slc6-gcc62-dbg is defined as an SLC6 system where HEP_OSlibs has been installed.

More specifically for CORAL and COOL, for many years the assumption has been that:

  • C++ packages that cannot be taken from the O/S, because they use a non-default compiler version (e.g. Boost, which for gcc 6.2 needs recompilation on slc6, where the default is gcc 4.4.7), should be included in the LCG stack;
  • packages that cannot be taken from the O/S, because a more recent version is needed (e.g. Python, for which 2.7.10 is required in LCG_87, while the default version on slc6 is 2.6.6), should also be included in the LCG stack;
  • packages that cannot be taken from the O/S, because they are HEP-specific (e.g. Frontier-client) or in any case are not included in standard O/S distributions (e.g. Oracle), should also be included in the LCG stack;
  • all other packages should be taken from the O/S using their default version there, through the heposlibs mechanism.

In other words, for many years the assumption has been that whatever could be taken from the O/S should be taken from the O/S through heposlibs, while the LCG stack should be kept as little as possible, because:

  • it is unnecessary to include in the LCG stack packages that could be taken from the O/S;
  • any packages included in the LCG stack could potentially lead to a conflict with the O/S if the build time and/or runtime environments are not configured correctly (see for instance CORALCOOL-762 about the issues caused by expat in 2010).

In 2016, however, discussions about possible changes to this policy have been started (see for instance the presentations at the LIM meeting of July 2016), because of the upcoming start of support for non-RedHat platforms including Ubuntu and especially MacOSX, and because of the adoption of the LCG stack by the SWAN project. While these discussions have not yet come to a firm conclusion, some changes have already been implemented in the LCG stack. The situation (especially concerning CORAL and COOL) may need some clean up to remove some inconsistencies, once a more precise policy has been defined. Some of the recent changes include:

  • Package libaio has been added to the LCG stack in LCG_85 (SPI-815). A dependency of CORAL on libaio has been added for both standalone and lcgcmake builds (CORALCOOL-2955). This is mainly due to the need to support Ubuntu and the unavailability of an equivalent mechanism to HEP_OSlibs there.
  • Packages openssl, zlib and freetype have been added to the LCG stack in LCG_85 (SPI-815). No dependency of CORAL or COOL on these packages has been added yet, neither for standalone nor for lcgcmake builds, even if these packages are needed: the Frontier_client library (needed by CORAL FrontierAccess) needs openssl and zlib (as can be seen from ldd /cvmfs/sft.cern.ch/lcg/releases/LCG_87/frontier_client/2.8.19/x86_64-slc6-gcc62-dbg/lib/libfrontier_client.so), while the ACE tool in COOL needs freetype (as can be seen from ldd /cvmfs/sft.cern.ch/lcg/releases/LCG_87/COOL/3_1_6/x86_64-slc6-gcc62-dbg/bin/ace). While openssl and freetype are only included as static libraries in the LCG stack, there is a potential risk of conflict coming from zlib, because Frontier_client will pull in the O/S library version 1.2.3 at runtime, while another version 1.2.8 is included in the LCG stack. This should be followed up within the CORAL/COOL project (CORALCOOL-2975).

2 Building CORAL and COOL using CMake

As mentioned in the Introduction, broadly speaking two different build scenarios exist for CORAL and COOL, depending on how the C++ compiler and the external dependencies are resolved: standalone building against existing external packages, and embedded building within lcgcmake builds of the full LCG software stack. Some explanations about how cmake is used internally in these two scenarios are given in the following subsections, which are then followed by more practical instructions and examples for building CORAL and COOL.

2.1 Environment variables controlling CORAL and COOL cmake builds

As in most CMake-based projects, the internal invocation of cmake in CORAL/COOL builds is affected by several environment (and/or CMake) variables, including:

  • Some variables understood by CMake itself:
    • CC and CXX determine the C and C++ compiler used by CMake
    • CMAKE_PREFIX_PATH determines the search path for external packages used by Find<Package>.cmake modules
    • CMAKE_BUILD_TYPE determines the build type (e.g. Release or Debug)
    • CMAKE_INSTALL_PREFIX determines the location of the install directory
    • CMAKE_CXX_FLAGS determines additional C++ compilation flags (e.g. to enable c++11 or c++14)
  • Some variables that are understood by other software layers external to CORAL and COOL:
    • CCACHE_DIR, a standard ccache-related variable that determines the location of the ccache cache for the build (note that ccache is used by default if installed)
    • BINARY_TAG, the conventional way to identify a platform in the LCG stack, is used amongst other things to determine the location of the build directory build.$BINARY_TAG
  • Some variables that were introduced specifically for CORAL and COOL builds:
    • LCG_releases_base is not used in the build but is relevant to relocatability of the runtime environment setup scripts produced during the build
    • CMAKEFLAGS makes it possible to specify additional CMake variables for CORAL/COOL builds (including some of the above, if not set as environment variables)

Standalone and lcgcmake build scenarios essentially differ in the way the above variables are set to steer the build process.

2.2 Three steps: configure, build, install ("cmake", "make", "make install")

In both standalone and embedded lcgcmake builds, the build process logically includes three steps:

  1. The cmake configuration ("cmake") step. This creates a build directory if it does not exist yet and generates all relevant build rules there for the chosen generator (GNU make or ninja).
  2. The build ("make") step. This compiles and links the code using GNU make or ninja and produces build targets inside the same build directory.
  3. The install ("make install") step. Using GNU make or ninja as in the build step, this installs the build targets in an install directory, which is also created if it does not yet exist.

Extensive changes were applied in August 2016 to both the internal cmake strategy of CORAL and COOL and its integration in lcgcmake (CORALCOOL-2846). The general idea of this new approach was to use a simpler approach and follow cmake best practices, to allow easier integration into lcgcmake. In particular, one goal of this reengineering was to ensure that both types of builds proceed through exactly the same steps, and as much as possible through exactly the same cmake code. As a consequence, the implementation of the three-step configure/build/install strategy is very similar for standalone and lcgcmake builds of CORAL and COOL, and only differs in a few details.

2.2.1 Standalone developer builds

In standalone developer builds, the configure/build/install steps are implemented by calling the cmake executable three times, with different command line options. These commands take two user-defined parameters: a set of cmake flags specifying what cmake should do in the initial configuration step (represented below by the $CMAKEFLAGS environment variable), and the path to the build directory (represented by $BUILDDIR below). The three commands to execute, from the top-level source directories where the software has been installed from SVN (e.g. CORAL/trunk or COOL/trunk), are the following:

  1. cmake $CMAKEFLAGS -H. -B$BUILDDIR, for the configure/"cmake" step
  2. cmake --build $BUILDDIR --config ., for the build/"make" step
  3. cmake --build $BUILDDIR --config . --target install, for the install/"make install" step

The first step uses an undocumented signature of the cmake executable, which makes it possible to create a new build directory from the source directory. This command is completely equivalent to the more traditional way of using cmake as cd $BUILDDIR; cmake $CMAKEFLAGS <sourcedir>.

To simplify development and ensure that the same $CMAKEFLAGS are used consistently, a setupLCG.sh script and a Makefile have been added to both the CORAL and COOL top-level source directories. Thanks to this, building simply requires one preliminary call to source setupLCG.sh with the appropriate command-line options, to set $CMAKEFLAGS and $BUILDDIR, followed by three make commands, to proceed with the usual configure/build/install steps:

  1. make cmake, for the configure/"cmake" step
  2. make, for the build/"make" step
  3. make install, for the install/"make install" step

The choice of the parameters to the setupLCG.sh script is what ultimately determines which type of standalone build is going to be performed. This script is primarily meant to emulate the environment set up within lcgcmake builds of a full LCG stack. Using appropriate parameters (especially those in the -l or -r command-line options), it makes it possible to prepare a build against and consistent with an existing installation of the LCG stack. The script however also makes it possible to prepare a "native" build using the default compiler and packages installed on the local O/S (if -l and -r are both missing). Sourcing this script with the -h option provides useful help about its usage. In particular, the following points should be noted:

  • The $BINARY_TAG$ environment variable must be set during the build process. Setting it via the -b option takes priority: if this is not used, the variable must have been set externally in advance, otherwise the script will fail.
  • The -l and -r option are two alternatives to indicate the path to an existing LCG release (or nightly) installation where all relevant externals should be found. These parameters however are optional: the paths to the externals can also be defined by setting $CMAKE_PREFIX_PATH directly, and/or by installing the relevant externals in O/S system directories.
  • The compiler is set up depending on the value of $BINARY_TAG$, using the script compilerSetup.sh that is called internally by setupLCG.sh.

2.2.2 Embedded lcgcmake builds

Within lcgcmake the build proceeds through the steps above (as can be seen in files CORAL-HEAD-stamp/CORAL-HEAD-build.cmake and CORAL-HEAD-stamp/CORAL-HEAD-install.cmake from an lcgcmake build).

2.3 Builds against an existing installation of the LCG stack

Builds against an existing installation of the LCG stack are those preferred for all internal developments by the CORAL and COOL team. They essentially consist in using all packages from an existing LCG stack, except for CORAL and COOL that are rebuilt locally. In the case of COOL, this is actually rebuilt against externals from the LCG stack, and a previously created local rebuild of CORAL against the same LCG stack.

In this section, we consider builds performed by making use of an LCG summary file. For a given LCG stack version and a given BINARY_TAG, this file contains (almost) all relevant information to launch a build consistent with that LCG stack.

In particular, these builds are triggered by setting the environment variable CORALCOOL_CMAKE_LCGSUMMARY equal to the location of one such file. In this case, Makefile-cmake.mk parses that file using a custom script lcg_summary_parser.sh, which sets the values of BINARY_TAG, CC, CXX, CMAKE_PREFIX_PATH and CMAKE_BUILD_TYPE consistent with those used in the LCG stack. Note that equivalent builds could also be produced by setting these five variables manually instead (i.e. using the general procedure for free builds, but using CORALCOOL_CMAKE_LCGSUMMARY provides better internal consistency.

For this type of builds, the default build directory and install directory are ./build.$BINARY_TAG and ./$BINARY_TAG within the top-level CORAL and COOL directories.

2.3.1 Detailed example

Here's a detailed example of how to build CORAL and COOL trunk for BINARY_TAG=x86_64-slc6-gcc49-dbg against the existing LCG_84 stack on CVMFS, under directories /build/CORAL and /build/COOL on a private SLC6 VM.

The following comments explain the rationale behind some of the somewhat arbitrary choices that were made in this example, and discuss some of the possible alternatives.

  • CVMFS is used in the example instead of AFS. For faster builds, always prefer CVMFS if the desired LCG stack is available there and if CVMFS is installed on your VM. As mentioned above, LCG stacks are normally installed on both AFS and CVMFS; but the most recent builds are sometimes only available on AFS if replication to CVMFS has not happened yet.
  • LCG_84, the most recent release at the time of writing, is used in the example. With respect to the previous LCG_83, this fixes bug ROOT-7864 in ROOT, which was causing segmentation faults in PyCool. The installation of LCG_84 still has a few pending issues affecting runtime environment setup, but these can be worked around as described below.
  • CORAL and COOL in this example are checked out from SVN using an authenticated account via svn+ssh. Alternatively, you can use guest checkout via http as described in PersistencyRepository.
  • Builds are performed in /build/CORAL and /build/COOL on a private SLC6 VM in this example. For faster builds, always prefer local disks to AFS. You could also use lxplus and AFS.
  • COOL is built against the local CORAL installation by explicitly adding the latter to the CMAKE_PREFIX_PATH for the COOL build. If this was not done, COOL would build against the CORAL installation available in LCG_84 (as is done for all other COOL externals).
  • Directories /build/CORAL/.ccache and /build/COOL/.ccache are used as ccache caches for the builds in this example. Using ccache speeds up re-builds enormously: deleting the build and install directory completely and rebuilding from scratch, which is an operation that developers frequently perform, becomes extremely fast. Invoking ccache -s is just a way to check what is in the cache before (re-)building.
  • CMake version 3.3.2 from CVMFS is used in this example. You can also use the default system version of CMake if you prefer.
  • The VERBOSE variable is set to 1 in this example in order to produce verbose printouts for the builds. Its effect is slightly different with ninja and with make; it is recommended that you install ninja on your system.

Check out CORAL from SVN and build it using cmake, against LCG_84 externals.

bash
cd /build/CORAL
\rm -rf trunk
svn co svn+ssh://svn.cern.ch/reps/lcgcoral/coral/trunk
cd trunk
export PATH=/cvmfs/sft.cern.ch/lcg/contrib/CMake/3.3.2/Linux-x86_64/bin:$PATH
export VERBOSE=1
export BINARY_TAG=x86_64-slc6-gcc49-dbg
export LCG_releases_base=/cvmfs/sft.cern.ch/lcg/releases/LCG_84
export CORALCOOL_CMAKE_LCGSUMMARY=${LCG_releases_base}/LCG_externals_${BINARY_TAG}.txt
export CCACHE_DIR=`pwd`/.ccache
ccache -s
make > log-cmake-${BINARY_TAG}.txt 2>&1
make install >> log-cmake-${BINARY_TAG}.txt 2>&1

Check out COOL from SVN and build it using cmake, against the local CORAL and LCG_84 externals.

bash
cd /build/COOL
\rm -rf trunk
svn co svn+ssh://svn.cern.ch/reps/lcgcool/cool/trunk
cd trunk
export PATH=/cvmfs/sft.cern.ch/lcg/contrib/CMake/3.3.2/Linux-x86_64/bin:$PATH
export VERBOSE=1
export BINARY_TAG=x86_64-slc6-gcc49-dbg
export LCG_releases_base=/cvmfs/sft.cern.ch/lcg/releases/LCG_84
export CORALCOOL_CMAKE_LCGSUMMARY=${LCG_releases_base}/LCG_externals_${BINARY_TAG}.txt
export CCACHE_DIR=`pwd`/.ccache
export CMAKE_PREFIX_PATH=/build/CORAL/trunk/$BINARY_TAG:$CMAKE_PREFIX_PATH
ccache -s
make > log-cmake-${BINARY_TAG}.txt 2>&1
make install >> log-cmake-${BINARY_TAG}.txt 2>&1

2.4 Builds within an lcgcmake build of the LCG stack

... useful links about using lcgcmake: official EP-SFT doc, genser doc, ATLAS HPC doc...

... coming soon ...

2.4.1 Special case: debugging an lcgcmake issue in-situ on a nightly build node

In some cases it may be necessary to debug an lcgcmake issue in-situ on a nightly build node. This was the case for instance for the LCG81f release (CORALCOOL-2916). The procedure that was followed in that case is documented in a specific comment to that Jira ticket and can be used in similar occasions if needed.

2.5 Free builds (including builds against the default system compiler and system libraries)

Under the third category of "free builds" we include all builds that do not fall in one of the two above categories: that is to say, they are standalone builds outside lcgcmake, and they are performed without setting the environment variable CORALCOOL_CMAKE_LCGSUMMARY. Under these circumstances, it is the responsibility of the user to set the values of BINARY_TAG, CC, CXX, CMAKE_PREFIX_PATH and CMAKE_BUILD_TYPE, if appropriate. If BINARY_TAG is not set, CORAL/COOL define a default value using the hsf_get_platform.py script. As for the other four variables, if these are not set then CMake uses its own internal defaults.

For this type of builds, the default build directory and install directory are ./build.$BINARY_TAG and ./$BINARY_TAG within the top-level CORAL and COOL directories.

These builds may arise in many different situations. In the following, we will consider only one specific example: a local build on a MacOSX build node of the EP-SFT jenkins nightlies infrastructure, against externals previously built within the nightlies. This is a typical situation where CORAL or COOL builds failed in the nightlies, but all other externals completed their builds successfully.

2.5.1 Detailed example

Here's a detailed example of how to build CORAL and COOL trunk for BINARY_TAG=x86_64-mac1011-clang70-opt on a Mac build node under directories /home/avalassi/CORAL and /home/avalassi/COOL, against the externals previously built using the jenkins nightlies infrastructure on that same node.

The following comments explain the rationale behind some of the choices that were made in this example and provide additional relevant information.

  • Build node macitois17 is used in this example. This is the build node that was used in April 2016 to port CORAL and COOL to the latest MacOSX 10.11 "El Capitan" (as described in CORALCOOL-2880, CORALCOOL-2881, CORALCOOL-2882, CORALCOOL-2883, CORALCOOL-2884, CORALCOOL-2885).
  • CVMFS and AFS are not available on this node. External software and XML files for database lookup and authentication must all be taken from local disks.
  • Instead of building CORAL and COOL directly against the externals built by jenkins nightlies under /Volumes/MacintoshHD2/build/electric_commander/build/workspace/lcg_experimental/BUILDTYPE/Release/COMPILER/native/LABEL/mac1011/install/ (which are automatcally rebuilt in place every day or even more frequently), a permanent copy of those files is made under /Users/avalassi/nightlies/20160420/. This ensures that a CORAL/COOL build may be prototyped and debugged over several days against the same externals. Copies of CORAL and COOL from the nightlies are included for completeness, even if these are then ignored when these projects are rebuilt from scratch.
  • The CORAL/COOL code is checked out from SVN using svn+ssh. On Mac, this may lead to one or more password prompts for every checkout and other SVN operations. As a workaround, make sure that the ~/.ssh/config file includes two lines containing HOST * and GSSAPITrustDns yes, respectively. For more details see Accessing CERN Linux machines via Kerberos.
  • The values of CC and CXX are set to match those used in the jenkins nightlies, as defined by CMAKE_CC_COMPILER and CMAKE_CXX_COMPILER in /Volumes/MacintoshHD2/build/electric_commander/build/workspace/lcg_experimental/BUILDTYPE/Release/COMPILER/native/LABEL/mac1011/build/CMakeCache.txt.
  • The jenkins nightlies produce a file /Volumes/MacintoshHD2/build/electric_commander/build/workspace/lcg_experimental/BUILDTYPE/Release/COMPILER/native/LABEL/mac1011/build/LCG_experimental_x86_64-mac1011-clang70-opt.txt that is very similar to those used for "type-1" builds using CORALCOOL_CMAKE_LCGSUMMARY. However, the format of this file is not exactly the expected one (see SPI-897) and for this reason this build was prepared using this "type-3" procedure, adding externals to CMAKE_PREFIX_PATH one by one.
  • COOL is built against the local CORAL installation by explicitly adding the latter to the CMAKE_PREFIX_PATH for the COOL build.
  • Neither ccache nor ninja are installed on this build node (see SPI-898).

Prepare a permanent copy of all externals from the most recent jenkins nightlies.

bash
mkdir /Users/avalassi/nightlies
mkdir /Users/avalassi/nightlies/20160420
for pkg in Boost CppUnit expat Frontier_Client mysql oracle Python QMtest sqlite XercesC gperftools tcmalloc igprof libunwind valgrind CORAL ROOT Qt COOL; \
  do rsync -rlptDPu /Volumes/MacintoshHD2/build/electric_commander/build/workspace/lcg_experimental/BUILDTYPE/Release/COMPILER/native/LABEL/mac1011/install/$pkg/ \
    /Users/avalassi/nightlies/20160420/$pkg; done

Check out CORAL and COOL from SVN.

bash
mkdir /Users/avalassi/CORAL
cd /Users/avalassi/CORAL
svn co svn+ssh://svn.cern.ch/reps/lcgcoral/coral/trunk
mkdir /Users/avalassi/COOL
cd /Users/avalassi/COOL
svn co svn+ssh://svn.cern.ch/reps/lcgcool/cool/trunk

Build CORAL using cmake, against the private copy of the externals from the most recent jenkins nightlies.

bash
unset CORALCOOL_CMAKE_LCGSUMMARY
export VERBOSE=1
export BINARY_TAG=x86_64-mac1011-clang70-opt
export LCG_releases_base=/Users/avalassi/nightlies/20160420
export CC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang
export CXX=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++
export CMAKE_BUILD_TYPE=Release
for pkg in Boost CppUnit expat Frontier_Client mysql oracle Python QMtest sqlite XercesC gperftools tcmalloc igprof libunwind valgrind; \
  do pkg1=/Users/avalassi/nightlies/20160420/$pkg; if [ -d $pkg1 ]; then export CMAKE_PREFIX_PATH=`find $pkg1 -mindepth 2 -maxdepth 2 -name $BINARY_TAG`:$CMAKE_PREFIX_PATH; fi; done
cd /Users/avalassi/CORAL/trunk
make > log-cmake-${BINARY_TAG}.txt 2>&1
make install >> log-cmake-${BINARY_TAG}.txt 2>&1

Build CORAL using cmake, against the local CORAL and the private copy of the externals from the most recent jenkins nightlies.

bash
unset CORALCOOL_CMAKE_LCGSUMMARY
export VERBOSE=1
export BINARY_TAG=x86_64-mac1011-clang70-opt
export LCG_releases_base=/Users/avalassi/nightlies/20160420
export CC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang
export CXX=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++
export CMAKE_BUILD_TYPE=Release
for pkg in Boost CppUnit expat Frontier_Client mysql oracle Python QMtest sqlite XercesC gperftools tcmalloc igprof libunwind valgrind ROOT Qt; \
  do pkg1=/Users/avalassi/nightlies/20160420/$pkg; if [ -d $pkg1 ]; then export CMAKE_PREFIX_PATH=`find $pkg1 -mindepth 2 -maxdepth 2 -name $BINARY_TAG`:$CMAKE_PREFIX_PATH; fi; done
export CMAKE_PREFIX_PATH=/Users/avalassi/CORAL/trunk/$BINARY_TAG:$CMAKE_PREFIX_PATH
cd /Users/avalassi/COOL/trunk
make > log-cmake-${BINARY_TAG}.txt 2>&1
make install >> log-cmake-${BINARY_TAG}.txt 2>&1

3 Setting up the CORAL and COOL runtime environment for CMake-based builds

Using non-standard locations for all external packages at CORAL/COOL build time means that the corresponding shared libraries and binaries will not be found at runtime, unless the runtime environment is modified accordingly.

  • On Linux, this is done by changing the values of the $LD_LIBRARY_PATH and $PATH environment variables.
  • Other environment variables must also be set to configure the behaviour of CORAL and COOL themselves or specific external packages (e.g. $PYTHONPATH for Python, $ROOTSYS and $ROOT_INCLUDE_PATH for ROOT, $TNS_ADMIN for Oracle),

In the CMake-based builds of CORAL and COOL, the configuration of the runtime environment is taken care of by the cc-run script, which is produced and installed during the build.

  • The name cc-run stands for "CORAL/COOL-run" and was chosen by analogy with the LHCb lb-run script, on which it is based (bear in mind however that cc-run has a significantly simpler scope and set of features than lb-run).
  • As described in the LHCB Software Tools twiki, cc-run and lb-run are modeled after the standard UNIX env command. Contrary to the philosophy of runtime environment setup for CMT-based builds, where the runtime environment of the current shell was modified forever by sourcing ad-hoc setup.sh and setup.csh scripts, cc-run allows the execution of a single command within a temporary shell with a modified environment.
  • A second script cc-sh, which is essentially equivalent to "cc-run bash", is also available to start a new subshell with the modified environment, where one can execute more than one commands and cleanly exit at the end; this emulates the effect of "bash followed by source setup.sh" in a CMT-based build.

Two different versions of the cc-run script are produced in the build and in the install directory.

  • In the vast majority of cases, you will normally only need to work with the version available in the install directory. This is actually the only version available for CORAL/COOL production releases installed on AFS and CVMFS, where only the install directory is available on these permanenet shared file systems, while the temporary build directory produced on the build nodes is deleted when the build process is finished.
  • The main difference between the two sets of scripts is that they use CORAL/COOL libraries and binaries from the build and install directories, respectively: the cc-run script in the build directory is only needed, for instance, to debug possible issues in the installation of libraries and binaries to the install directory.

The setup of the CORAL/COOL runtime environment has been extensively tested and debugged in several different configurations, including variants of the three build scenarios described above. In this context, however, it is largely irrelevant how BINARY_TAG, CC, CXX, CMAKE_PREFIX_PATH, CMAKE_BUILD_TYPE or other variables are set to steer the CMake build in the three different scenarios: what is really important is that some of these scenarios involve the relocation of the install directory to a different path and/or filesystem than that on which it was initially created. In particular, the two following scenarios will be considered in the following:

  • Local builds where no relocation occurs. This is typically the case of development builds (section 2.1), where a developer continuously changes, rebuilds and tests the code in an interactive and iterative way.
  • Relocated builds, where the install directory is moved elsewhere than where it was produced, for instance by direct file copy or by packing and unpacking of rpm or tar packages. This is typically the case of automated production and nightly builds using lcgcmake (section 2.2), with relocation to AFS, CVMFS or another remote system.

3.1 Environment setup for local builds

For the local CORAL build in /build/CORAL/trunk with BINARY_TAG=x86_64-slc6-gcc49-dbg of the example described in section 2.1, you may for instance

  • run /build/CORAL/trunk/x86_64-slc6-gcc49-dbg/cc-run <command> [<arguments>] to execute a command with the appropriate CORAL environment
  • run /build/CORAL/trunk/x86_64-slc6-gcc49-dbg/cc-sh to enter a temporary bash shell with the appropriate CORAL environment

To see which environment variables are set by the script, you may for instance execute the following (where "env -i" starts with a completely empty environment):

  • env -i /build/CORAL/trunk/x86_64-slc6-gcc49-dbg/cc-run env

3.2 Environment setup for relocated builds

... coming soon ...

... cc-run includes basic relocation for simple file copies ($LCG_release_area and $CORALSYS are meant to ease this) ... but lcgcmake and rpm installations used to complicate things because of post-install scripts ...

... ideally just try /afs/cern.ch/sw/lcg/releases/LCG_84/CORAL/3_1_3/x86_64-slc6-gcc49-dbg/cc-sh ... this works, but /afs/cern.ch/sw/lcg/releases/LCG_84/CORAL/3_1_3/x86_64-slc6-gcc49-opt/cc-sh does not and will be fixed in the next release, see CORALCOOL-2873 and SPI-855) ...

4 Executing CORAL and COOL tests for CMake-based builds

... This may be covered more in detail in a separate twiki page, also mentioning the distinction between qmtest tests and ad-hoc CoralServer tests? For the moment here we describe basic functional tests and the qmtest suite ...

... Execute a simple CORAL test: /build/CORAL/trunk/x86_64-slc6-gcc49-dbg/cc-run test_CoralBase_CoralBaseTest or /build/CORAL/trunk/x86_64-slc6-gcc49-dbg/cc-sh followed by test_CoralBase_CoralBaseTest; exit ...

... Execute the fast CORAL test suite (used in lcgcmake nightlies and production releases): /build/CORAL/trunk/x86_64-slc6-gcc49-dbg/run_nightly_tests_cmake.sh ...

... Execute the full CORAL test suite (for private development): /build/CORAL/trunk/x86_64-slc6-gcc49-dbg/qmtestRun.sh ... This is what ideally you should do on installed releases in AFS to check that there are no issues ...

... The above are all scripts that internally use cc-run to set up the environment ...

... For some of the above you may need your own database accoounts and/or access to some private AFS areas...

... On Mac you need to copy the xml files from AFS and set the dblookup and authentication path accordingly...

mkdir /Users/$USER/private
scp -o GSSAPIDelegateCredentials=yes $USER@lxplus:/afs/cern.ch/sw/lcg/app/pool/db/authentication.xml /Users/$USER/private/.
scp -o GSSAPIDelegateCredentials=yes $USER@lxplus:/afs/cern.ch/sw/lcg/app/pool/db/dblookup.xml /Users/$USER/private/.
mkdir /Users/$USER/private/admin/
scp -o GSSAPIDelegateCredentials=yes $USER@lxplus:/afs/cern.ch/sw/lcg/app/releases/CORAL/internal/oracle/admin/adminNoKerberos/tnsnames.ora /Users/$USER/private/admin/
scp -o GSSAPIDelegateCredentials=yes $USERi@lxplus:/afs/cern.ch/sw/lcg/app/releases/CORAL/internal/oracle/admin/adminNoKerberos/sqlnet.ora /Users/$USER/private/admin/
export CORAL_AUTH_PATH=/Users/$USER/private
export CORAL_DBLOOKUP_PATH=/Users/$USER/private
export TNS_ADMIN=/Users/$USER/private/admin
export COOL_QMTEST_USER=sftnight

-- AndreaValassi - 2015-06-22

Edit | Attach | Watch | Print version | History: r36 < r35 < r34 < r33 < r32 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r36 - 2017-02-21 - AndreaValassi
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Persistency All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback