Compiling G4 for ATLAS

This is a draft version -- Currently a copy of the instructions for compiling Geant4 for ATLAS with gmake. Created -- JohnApostolakis - 30-Nov-2012

The original of this page is in the ATLAS Twiki at URL

( Q: Why is this not reached as wiki location ? )


Preparation of a new G4 version to be used with AtlasSimulation can be divided into the following steps:

  • Obtain the G4 source code
  • Small modifications to physics lists and run manager
  • Setting up the compilation environment
  • Compiling G4
  • Installing G4
  • Compiling AtlasSimulation
  • Running

We will now go through the steps one by one, and provide detailed instructions.

Obtain the G4 source code

Official test builds of G4 by the ATLAS simulation team will be made in the /afs/ directory. If you are making a private build for your own purposes, you will need a suitably large area, such as a private scratch directory. Official G4 releases, including patches and betas, are available from /afs/

Most G4 builds for ATLAS will be patches of existing builds in the ATLAS software: for these, the best approach is to get the source code and other files from the latest of the already-built ATLAS releases in /afs/*/geant4.*. This avoids having to re-find and re-apply any previous patches. The non-architecture-specific copy of the G4 build is the one that you will want to use (i.e. no "i686-slc5-gcc34-opt" or similar suffix).

If building an entirely new G4 release for ATLAS, you will need to get an official source release, either from the official Geant4 web site or (usually more conveniently, due to lack of availability of a direct URL to wget onto lxplus) from the G4 release area on AFS: /afs/ If you are trying an unofficial release (i.e. a release candidate) you will need to contact the G4 developers and ask them for a copy of the source code. Once you have it, unpack it in your scratch area.

If you wish to build only one library from the Geant4 toolkit, it is best to take the lib, config, source, and include directories in their entirety from the ATLAS installation area (see below). You can then do

gmake global
within the directory you wish to modify (e.g. source/tracking).

Small modifications to physics lists and run manager

On top of the code distributed by G4, we apply two small modifications:

  • Modify in order to add some meaningful information to the G4 banner. A typical banner should look like this:
    G4cout << G4endl
    << "*************************************************************" << G4endl
    << versionString << G4endl
    << "                      Copyright : Geant4 Collaboration" << G4endl
    << "                      Reference : NIM A 506 (2003), 250-303" << G4endl
    << "                            WWW :" << G4endl

    << "Internal ATLAS build geant4.9.3.patch01.atlas01" << G4endl
    << "Created on 2010-02-12;" << G4endl
    << "Using CLHEP; XercesC 3.1.1; gcc43; SLC5; 32 bit; optimised" <<G4endl << G4endl

    << "Differences w.r.t. the \"plain\" geant4-09-03-01 are:"<<G4endl
    << "\t Bugfix to XXX"<<G4endl
    << "\t Patch to reduce memory churn in YYY"<<G4endl
    // etc.

    << "*************************************************************" << G4endl
    << G4endl;
If you are modifying an existing ATLAS build, don't forget to update this header.

If you want to include any other physics lists then make sure that you disable the physics list consistency checking in these physics list source files if you aren't copying them from an earlier Atlas build (unless G4 have started doing this "upstream".)

If you need to apply any other patch, do it now. New patches should be put into the builds/patches directory, in a new folder following the naming scheme $yyyy-$mm-G4.9.$x-$g4contact-$description, so that ATLAS patch builds may be reconstructed (say, when a new G4 patch is issued which does not include all the ATLAS patches from the previous G4 version). Please add a README file to the new patch drectory to detail what this patch contains and if there are any "gotchas" in its application, e.g. clashes with other patches.

Setting up the compilation environment

First we need to provide environment variables to tell the G4 build system which Makefiles etc. to use. Create a file like the following example:


## Choose correct architecture
export G4SYSTEM=Linux-g++
#export G4SYSTEM=Darwin-g++

## Choose architectures for CLHEP
#export CLHEP_BASE_DIR=/afs/
export CLHEP_BASE_DIR=/afs/
#export CLHEP_BASE_DIR=/afs/
#export CLHEP_BASE_DIR=/afs/

## Choose architectures for XercesC
#export XERCESCROOT=/afs/
export XERCESCROOT=/afs/
#export XERCESCROOT=/afs/
#export XERCESCROOT=/afs/


#export G4DEBUG=1

Note that this file will need to be customised for each architecture's build. The CLHEP and XercesC paths must match those of the LCG externals package against which the ATLAS software is being built. Those above are for the 15.9.0 series and later, corresponding to LCG_59c: builds for the 15.6.* production releases (at the time of writing) require older CLHEP and XercesC versions. The future intention is to update the LCG externals package more frequently than has been the case in the past, so some care will be required to make sure that compatible G4 builds are available for each release. Currently, this means that the G4.9.2 series is built for 15.6.X, G4.9.3 (unsupported) for 16.* releases, and the G4.9.4 series are built with the development/reprocessing 17.* releases. The external software versions can be found in the cmt/requirements file of the LCG_Configuration package, e.g.

Compiling with a release configuration

We now use the AtlasSetup system for ATLAS buiild configuration. For reproducibility, we recommend that you create an file as follows:

export AtlasSetup=/afs/
alias asetup='source $AtlasSetup/scripts/'

asetup $RELEASE,$PLATFORM,notest

echo "Using build area = $AtlasArea"
echo "Using CMTPATH = $CMTPATH"

Again, note that this will need to be adapted for each architecture being built.

Finally, as release setups also set specific G4 variables which are not wanted for a G4 built, we must unset some before continuing. Create a file containing the following:

unset G4UI_USE_GAG
unset G4LIB
unset G4PATH
unset G4SYSTEM
unset G4VERS
unset G4UI_USE_WO

This completes the directory setup to be used for all your G4 builds -- we now create several copies of the G4 directory for the architecture-specific and shared builds:

Creating multiple build directories

We will use the directory that you have been using so far for the architecture-independent part of the build: before doing this, we make copies of the source tree for the architecture:

cd ..
rsync -raz geant4.9.3.patch01.atlas02{,.i686-slc5-gcc43-dbg}/
rsync -raz geant4.9.3.patch01.atlas02{,.i686-slc5-gcc43-opt}/
rsync -raz geant4.9.3.patch01.atlas02{,.x86_64-slc5-gcc43-opt}/
rsync -raz geant4.9.3.patch01.atlas02{,.x86_64-mac106-gcc42-opt}/

While this may seem a little over-prescriptive, I've found that doing this before starting the build helps to keep the several build copies under control, and avoids having to copy large numbers of build-generated files which then need to be deleted.

Building the include files

Enter the architecture-independent build dir, and do the following to set up the correct environment, create the correct G4 includes directory, and make useful log files:

cd source
env > env.log
make includes &> make-includes.log &
less make-includes.log

Assuming that all worked fine, that's it for the architecture-independent directory. Note that the build can be done safely on lxplus without overrunning the allowed job length, because the make job forks smaller jobs, none of which run for long enough to trigger the process killer. However, due to this process-forking you cannot "nohup" the build jobs and log out... you will need (or at least want!) a stable network connection for all these build steps.

Compiling G4

Now we compile each architecture as appropriate. Make sure you are logged into a suitable build node (i.e SLC4 or SLC5 lxplus, or a special Mac build machine as appropriate) before starting! Enter your architecture-specific build dir of choice and make the following customisations:

  • Edit config/architecture.gmk and add
    CPPFLAGS += -DG4HadSignalHandler_off
    at the bottom: this should be done for all builds and ensures that the internal G4 signal handler is disabled... otherwise it will interfere with the Athena signal handler and result in no tracebacks from runtime crashes: a big problem for production error debugging.
  • Still in config/architecture.gmk, add
    LDFLAGS += -Wl,--hash-style=both
    CXXFLAGS += -Wl,--hash-style=both
    at the same point as the CPPFLAGS modification above. I don't know if this flag is meant to apply to both the compiler and linker: maybe just modifying LDFLAGS is enough, but I've played it safe so far wink These flags implement a faster, hash-based symbol lookup scheme which may help with Athena initialisation performance.
  • Edit source/run/src/ to specify the correct architecture in the banner (the patches should all have been commonly documented before you made the architecture-specific copies of the build dir)
  • Edit to pass the correct CMT tags (slc4/slc5, 32/64, gcc34/gcc43) to get the appropriate architecture setup
  • Edit to choose the correct architectures for the CLHEP and Xerces (XML parser) packages
  • If you are making a Max OSX build, edit to specify the correct architecture to the G4 build (Darwin-g++). For all Linux builds, the Linux-g++ setting is appropriate, as for the default setup in the template above.

Depending on your platform, you may now need to edit the G4 Makefiles to pass flags appropriate for 32 bit or SLC5-64 builds.

Compiling in 32 bit mode

In the script, uncomment the 32 bit line and comment out the 64 bit line. In G4RunManagerKernel, change the banner to read '32bit'. In geant4.x.y.z/config/architecture.gmk, add

CXXFLAGS += -m32
at the end.

And in geant4.x.y.z/config/sys/Linux-g++.gmk (or geant4.x.y.z/config/sys/Darwin-g++.gmk for 32 Mac builds... which we don't currently do), change the two occurances of $(CXX) to $(CXX) -m32 (they appear at the end of the file).

You can check that the build has worked properly after it has finished, by doing readelf -h geant4.x.y.z.../lib/Linux-g++/libG4*.so and looking for Class: ELF32. For the debug symbols build, check that you get a non-empty output from readelf -e geant4.x.y.z/lib/Linux-g++/libG4*.so | grep debug_info... the other builds should give an empty output from the grep.

Compiling in 64 bit mode on SLC5

In geant4.x.y.z/config/sys/Linux-g++.gmk, change the two occurances of $(CXX) (at the end of the file) to include (leave the other options as well):


Again, check that the architecture is correct after the build, with readelf -h geant4.x.y.z.../lib/Linux-g++/libG4*.so and this time look for Class: ELF64.

Compiling in 64 bit mode on Macs

The Mac builds only exist in mig11, so set up your environment (in with asetup mig11,rel_0,builds,64. Note that the old AtlasLogin system will not set up the Mac architecture correctly -- not that we recommend using that anyway.

You must also make sure that the G4SYSTEM variable in is set to Darwin-g++ and that there are no -m32 hanging around in the config directory... if you've copied the platform-independent directory to make the Mac platform-specific build dir (as described above), this shouldn't be a problem, but it's worth checking.

Compiling G4 in opt with debug symbols


setenv G4OPTDEBUG 1
and do NOT set G4DEBUG or G4OPTIMIZE

You can check that the debug symbols are initially present using readelf -e geant4.x.y.z/lib/Linux-g++/libG4*.so | grep debug_info (should give a a non-empty output)

cd geant4.x.y.z/lib/Linux-g++/
for library in `ls libG4*.so`; do
objcopy --only-keep-debug $library $library.dbg 
objcopy --strip-debug $library
objcopy --add-gnu-debuglink=$library.dbg $library
cd -
The first line in the loop creates a file containing the debugging info. The second line strips the debug symbols out of the library. The third line in the loop adds a link to the debugging info into the stripped library. You can check the linking worked by checking that readelf -e geant4.x.y.z/lib/Linux-g++/libG4*.so | grep gnu_debuglink gives a non-empty output for each library.

Run the build

Now run the following series of commands:

cd source
env > env.log
make glob &> make-glob.log &
less make-glob.log

As for the includes directory, you can run this on lxplus without the build jobs being killed, but beware that nohup cannot be used to make the build robust against logging out, because new jobs launched by a logged-out process will be killed. So once again you will need a stable network connection... or the patience to log back in several times per architecture to restart the stalled build!

After doing the above for each architecture, you should now have a set of build directories, one of which contains the include files in geant4.x.y.z.patchXX.atlasYY/include/, and the others which contain architecture-specific library files in geant4.x.y.z.patchXX.atlasYY.*/lib/. The final step before testing is to create a set of symlinks similar to the official ATLAS G4 installation, so that a test version of AtlasSimulation can be compiled against it.

Making a test-installation of G4

G4 needs to be installed in a standard AFS location in order to be found by the AtlasSimulation packages. The official location is /afs/, and the structure under the releases directory is the one that needs to be mimicked for testing before officially deploying your new builds. The structure is as follows:

  + share/
    + geant4.9.2.patch02.atlas04/
    + ...
    + geant4.9.3.patch01.atlas02/
      + data
      + include
      + source
  + specific/
    + x86_64-mac106-gcc42-opt/
    + i686-slc5-gcc43-dbg/
    + ...
    + x86_64-slc5-gcc43-opt/
      + geant4.9.2.patch02.atlas04/
      + ...
      + geant4.9.3.patch01.atlas02/
        + lib

The data, include, source, and lib files in this directory structure are symlinks to the corresponding directories from your build. Personally, I put my test releases directory at the same level as the G4 build directories, so the symlinks resolve as follows:

abuckley@lxplus201:~/.../G4$ ls releases/share/geant4.9.3.patch01.atlas02 -l
total 3
lrwxr-xr-x  1 abuckley zp 60 Dec 24 18:46 data -> /afs/
lrwxr-xr-x  1 abuckley zp 43 Jan  5 16:04 include -> ../../../geant4.9.3.patch01.atlas02/include
lrwxr-xr-x  1 abuckley zp 42 Jan  5 16:04 source -> ../../../geant4.9.3.patch01.atlas02/source

abuckley@lxplus201:~/.../G4$ ls releases/specific/i686-slc5-gcc43-opt/geant4.9.3.patch01.atlas02 -l
total 1
lrwxr-xr-x  1 abuckley zp 56 Jan  5 16:06 lib -> ../../../../geant4.9.3.patch01.atlas02.i686-slc5-gcc43-opt/lib

Note that the data symlink for testing will point somewhere outside your build area, either to the G4 copy (as here) or it may be a directory containing specific symlinks to the ATLAS copies of the G4 data files in /afs/ For this test form of the releases directory, it is not important whether the links are relative or absolute, but for the official install they must be relative due to the way that the releases are deployed on to the Grid and into release kits: you may as well get in the relative symlinking habit now.

Now that we have working (?) G4 builds, and a mock-up of the releases area structure expected by the AtlasSimulation CMT build setup, we will test our builds against a copy of AtlasSimulation before installing for production use.

Testing G4 with AtlasSimulation

Builds of AtlasSimulation against the testing versions of G4 are done in the simulation group area, /afs/ This contains a directory for each G4 test version, each of which contains the AtlasSimulation project build area structure, i.e. a directory level called AtlasSimulation, in which there are one or more ATLAS release number directories, each containing a source/build tree of the appropriate AtlasSimulation release, e.g.

abuckley@lxplus201:~$ ls G4.9.3.p1.a2/AtlasSimulation/
15.6.9  15.6.10

We will now make a similar build for the new version of G4 we wish to test:

Building AtlasSimulation against a trial G4 build

So you should make a similar directory structure for your new G4 build and the version(s) of AtlasSimulation that you want to test, e.g.

mkdir -p G4.9.3.p01.a02/AtlasSimulation/15.6.10
and cd into that directory.

We will now check out the AtlasSimulationRelease package, at the tag appropriate for the release we wish to build - this is a special package for tracking the tag contents of the AtlasSimulation project, and will help us to check out the rest. To find the correct tag version, the easiest way (IMO) is to look at the tags link on the NICOS page for the stable release of interest: you can get to this from the appropriate version number link at Here's how to check it out when you know the tag name:

cmt co -r AtlasSimulationRelease-AtlasSimulation-xx-yy-zz Projects/AtlasSimulationRelease -d AtlasSimulationRelease

Now we use this package to check out all the tags in the project for that release:

cmt co -requirements AtlasSimulationRelease/cmt/requirements
This will take some time to run: you may wish to nohup the process and get a coffee.

Once done, you need to make a few modifications: again, I recommend making the file as described above for the G4 compilation... or copy it from an existing G4 testing AtlasSimulation build. This time, don't add the notest tag to use the AtlasEvent, here tags instead, to make sure that the current directory is used as the primary build/test area for the AtlasSimulation part of the build.

Now source this setup file, e.g. source from the AtlasSimulation/XX.YY.ZZ directory in which it is located.

You now have to modify External/Geant4/cmt/requirements to point to the new G4 version. Change the definition of the Geant4_native_version macro to your new version (in the standard form as encoded in your releases directory) and the Geant4_home macro to point to your releases directory, e.g.

# Adapt standard macros for export
#macro Geant4_home "${SITEROOT}/atlas/offline/external/geant4/releases" \
#    BNL         "${SITEROOT}/cernsw/geant4/releases" \
#    LBNL        "${SITEROOT}/sw/geant4/releases" \
#    EXTSITE     "${SITEROOT}/geant4/releases" \
#    STANDALONE  "${SITEROOT}/geant4/releases"
#macro Geant4_home "${SITEROOT}/atlas/offline/geant4/releases"

macro Geant4_home "/afs/"
macro Geant4_native_version "geant4.9.3.patch01.atlas04"
You may as well build this external package now, by calling make from inside the External/Geant4/cmt directory: check the output to make sure that the correct G4 library versions are being symlinked into your AtlasSimulation build's InstallArea.

If the version of G4 that you are testing requires changes to the G4 API calls from the ATLAS sim software, make the changes to the code now.

Now go to the AtlasSimulationRelease/cmt directory and launch the rest of the build with cmt br - "cmt config && make -j2" &> make-$CMTCONFIG.log &. As for the G4 builds, this will take a long time, and a reliable network connection is highly recommended. You will probably want to build for several platforms, by changing the platform definition in avoiding the clash on the build log files is the reason for including $CMTCONFIG in the logfile name.

Finally, to make this build area usable we need to add a cmt/version.cmt file for our AtlasSimulation project area. This should go in the project directory (the one which directly contains InstallArea), like so:

cp -r /afs/ .

Running with the trial AtlasSimulation build

In order to use the custom AtlasSimulation build, call asetup with the --prependarea=XXX flag, e.g.

asetup  --prependarea=/afs/ AtlasSimulation,AA.BB.CC,32,notest

Everything should now work... test the new G4 build until content, and then move on to the final step: the official installation.

Installing G4 in the official ATLAS AFS area

First, note that the G4 versions in the official ATLAS AFS area are not to be installed lightly: they should first be tested as outlined above. Also, you should only install G4 releases into this area if you intend for them to be used in ATLAS production releases: once installed to the releases directory, G4 builds should generally never be uninstalled, at least once a release has started to use them. For test versions of G4 that you'd e.g. like to use in a MIG nightly build, you should leave your G4 builds in the projects/geant4 area and collect a specially modified External/Geant4 tag into the MIG.

If you do still wish to install a build for use in an ATLAS release, this final step is essentially identical to those above, but is slightly complicated by AFS read-only volume issues. Essentially, you want to install your built libraries, headers, data files, and symlinks into /afs/, but this path is read-only. Instead you must modify /afs/ (spot the extra dot in front of and send a request to the ALAS librarian(s) for the volumes to be synchronised when you are done.

In the /afs/ directory, there are several directories named like g4.X.Y, which obviously correspond to Geant4 release numbers. Each of these is actually a separate AFS volume, as the G4 builds take up a lot of space. You will be installing into the appropriately named directory for your G4 release: if it doesn't exist or there is not enough free space (use fs listquota), ask the ATLAS librarian to create a new volume or extend the too-small one: space should always be provided for production G4 releases.

In your chosen g4.9.Y directory, make G4 directories corresponding to the ones in your private build, i.e. geant4.9.4.patch01.atlas02, geant4.9.4.patch01.atlas02.i686-slc5-gcc43-opt, etc. Note that the names here are strictly structured: the ATLAS internal name scheme is geant4.x.y.patchKK.atlasZZ. The "patch" field is the official G4 patch number. The "atlas" field takes into account unofficial ATLAS patches (put "00" if no unofficial patches are applied). The architextures are identified by the CMTCONFIG code, as before.

Copy the appropriate parts of your build directories into the new folders. Specifically, you should copy the include and source directories, plus any log files that you made during the "make includes" process, into the architecture-independent target directory, and the appropriate lib directory plus "make glob" log files into each of the architecture-specific targets. Do not copy all the files from each directory, even to delete them later: there are a huge number of files that you will need to delete and this just wastes a lot of time.

Now you need to create the symlinks in the /afs/ directory: this is exactly as you did for your test releases area, but relative symlinks are now imperative. If your new G4 release uses new data libraries, download the libraries from the G4 site and install them into /afs/ You will need to symlink the appropriate data file (with version name) from the ATLAS area to a versionless location in your releases/share/geant4.*/data directory (check existing releases for the naming scheme), e.g. ln -s /afs/ G4EMLOW.

At this point G4 is compiled, installed and nearly ready to be used by AtlasSimulation. The final step is to sync the read-write volume to the read-only one. If you have AFS admin permissions, you can do this using the command afs_admin vos_release DIR to release the volume asociated with directory DIR. You will need to do this separately for the main geant4 releases volume, and for each of the g4.X.Y volumes. If you don't have AFS permissions, ask the ATLAS librarian to do this bit for you.

And you're done! Now just pull a new tag of External/Geant4 into the appropriate nightlies using the Tag Collector and relax with a well-deserved beer.

Discussion on building G4 with CMake

Major Updates: -- AndyBuckley - 16 Sep 2011 -- JohnApostolakis - 30-Nov-2012

%RESPONSIBLE% JohnApostolakis


Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2012-12-05 - JohnApostolakis
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback