Concurrency For High Energy Physics twiki

Welcome to the Concurrency For High Energy Physics twiki!

Our goal is to bring concurrency in the modern HEP data processing. We work on Concurrent Gaudi, in ordet to allow to process multiple events concurrently and execute several data processing algorithms simultaneously.

Our Approach

We decided to approach the problem very pragmatically, trying to develop the framework and simultaneously testing it in a real life case, taken from the LHCb reconstruction. See C4HepMiniBrunel for more details.


For the Atlas flavoured exercise, please see AthenaHive !

Getting Started

Concurrent Gaudi

Instructions to get started with Concurrent Gaudi here:

# set up the minimal environment
export CMTCONFIG=x86_64-slc6-gcc48-opt
export PATH=/afs/$PATH
source /afs/$CMTCONFIG/

# clone
git clone -b dev/hive Gaudi
cd Gaudi

# configure for push via ssh
git remote set-url --push origin ssh://

# Set mail before pushing
git config --global "$USER"

# Set locale to C to avoid problems with boost::filesystem
export LC_ALL=C

# Build! 
make  -j20

Some useful aliases (if you like using them):

alias gaudirun="$PWD/build.$CMTCONFIG/run"
alias gaudipython="$PWD/build.$CMTCONFIG/run  python"

And to see the concurrent infrastructure in action:

gaudirun GaudiHive/options/

This options file contains also quite some comments explaining how to use the components for concurrency. Do not hesitate to look into it to understand how this works!

Concurrent Gaudi for Ubuntu

To install Gaudi Hive on ubuntu, you have to install LCG externals first:

# Create folder for LCG externals
mkdir -p ~/hive/lib; cd ~/hive/lib;

# Set environment, must be at least gcc 4.8, use SLC 6
export CMTCONFIG=x86_64-slc6-gcc48-opt

# Download installer & download LCG (here version 68)
chmod a+x
python -b LCGCMT 68

Now you need to execute LbLogin, with slc6 in CMTCONFIG it will not work. To make LbLogin find the missing encryption modules (e.g. md5) do:

# Install libssh 1.0.0, in case create folder lib where links have to be
ln -s /lib/x86_64-linux-gnu/ lhcb/COMPAT/COMPAT_v1r18/CompatSys/$CMTCONFIG/lib/
ln -s /lib/x86_64-linux-gnu/ lhcb/COMPAT/COMPAT_v1r18/CompatSys/$CMTCONFIG/lib/

# Do the LbLogin:
. ~/hive/lib/ -c x86_64-slc6-gcc48-opt

Now go ahead with installing concurrent Gaudi:

export LC_ALL=C
cd ~/hive/Gaudi
git clone -b dev/hive Gaudi
cd Gaudi

In case of compile errors install the correspondent libraries (e.g. sudo apt-get install libmpc3). Make sure that demanded files are accessible. Some errors with ubuntu 13.10 might be regarding definitions in the /usr/include/bits/byteswap.h file. Copy it from a SLC 6 machine to the same directory on ubuntu. Make sure, that the differences are only due to the missing functions, e.g.:

In file included from /usr/include/endian.h:60,
                from /usr/include/ctype.h:40,
                from _/data/gaudi-hive/lib/lcg/external/gcc/4.8.1/x86_64-slc6/bin/_../lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../include/c++/4.8.1/cctype:42,
                from _/data/gaudi-hive/lib/lcg/external/gcc/4.8.1/x86_64-slc6/bin/_../lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../include/c++/4.8.1/bits/localefwd.h:42,
                from _/data/gaudi-hive/lib/lcg/external/gcc/4.8.1/x86_64-slc6/bin/_../lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../include/c++/4.8.1/string:43,
                from /data/gaudi-hive/Gaudi/RootCnv/RootCnv/RootRefs.h:16,
                from /data/gaudi-hive/Gaudi/RootCnv/dict/RootCnv_dict.h:1:
/usr/include/bits/byteswap.h: In function 'unsigned int __bswap_32(unsigned int)':
/usr/include/bits/byteswap.h:47: error: '__builtin_bswap32' was not declared in this scope
/usr/include/bits/byteswap.h: In function '__uint64_t __bswap_64(__uint64_t)':
/usr/include/bits/byteswap.h:111: error: '__builtin_bswap64' was not declared in this scope
--->> genreflex: INFO: Parsing file /data/gaudi-hive/Gaudi/RootCnv/dict/RootCnv_dict.h with GCC_XML
--->> genreflex: ERROR: processing file with gccxml. genreflex command failed.

Concurrent Gaudi and LHCb stack

Instructions to get started with Concurrent Gaudi and the LHCb stack here:

These instruction will bring to your local area the whole LHCb stack, therefore also Gaudi. There is no need to install Gaudi separately.

export CMTCONFIG=x86_64-slc6-gcc48-opt

# Clone repository
git clone -b dev/hive /afs/

cd LHCbStack

# Used as optimization in the compiler wrappers
export LCG_hostos=x86_64-slc6

# Set locale to C to avoid problems with boost::filesystem
export LC_ALL=C


cmake .
make -j 20

# to set the environment
./run bash

# Setup COMPAT for the compressed CondDB


  • If you wish to recompile Gaudi once in the context of the LHCb software stack you have to clean the Stamps directory:
         rm  work/Stamp/Gaudi/Gaudi-build 
         make -j 20

  • If you don't have the permissions to access the LHCb Castor storage, you could add to your configuration the following lines for having some input:
         # Change input file
         EventSelector().Input = ["DATAFILE='mdf:/afs/' SVC='LHCb::MDFSelector'"]
An example file for running minibrunel is distributed with the stack. In order to run it: Brunel/Rec/Brunel/options/MiniBrunel/

LHCb stack for Concurrency: C4HepReleases

How to upgrade the versions of the LHCb projects in the LHCb stack:

# Clone repository
git clone -b dev/hive /afs/

cd LHCbStack

# Set up the LHCb login environement
. /afs/

# Upgrade LHCb releases branch to v37r0
./ LHCb v37r0

# Enter the temporary directory that have been created and push the changes
cd tmp/LHCb
git push
git push --tags

# Merge the changes in the releases on the dev/hive branch
git checkout dev/hive
git merge v37r0

# Fix the conflicts

Important: before merging you should check the changes in dev/hive wrt to the latest merged version to know in advance where you need to update the changes.

For example, to check which files have been modified in LHCb from v36r2 in both dev/hive and v37r0:

$ (git diff --name-only v36r2 v37r0 ; git diff --name-only v36r2 dev/hive) | sort | uniq -d

This gives the list of files that have been modified both between v36r2 and v37r0, and between v36r2 dev/hive, so these are the files that need to be checked carefully during the merge.


In this section, the goals we achieved and the ones we would like to achieve are listed.

Milestone 4: Provide a beta of a framework containing all the functionalities needed for MT execution

Also called Concurrent Gaudi release 1.0 (click for the project history in JIRA). Provide solutions to historical ideocyncracies of the Atlas and LHCb experiment software to provide a smooth transition to parallel processing. In addition, provide a series of recipes and recommendations to migrate the users' code for offline and online execution in a state compatible with the new framework.

Milestone 3: Provide ingredients necessary to build parallel experiment software for future experiments

With the Concurrent Gaudi v0.6 milestone the insights gained from the v0.5 release are evolved to production level solutions. Future experiments can design their experiment software with concurrency in mind from the beginning and use Gaudi as their framework of choice.

Milestone 2: Run a restricted set of algorithms (Mini-Brunel) with the GaudiHive framework

Also called Concurrent Gaudi release 0.5 (click for the project history in JIRA). Consider a restricted set of algorithms of the LHCb reconstruction application Brunel and run them with the new concurrent Framework. All details about the workflow can be found at this page C4HepMiniBrunel.

Completed in May 2013 (Q2 as planned and at the beginning of it) Presentation with results here.

Runtime of event loop scaled to the sequential case as a function of the number of the maximum number of algorithms processed simultaneously.

Milestone 1: Integration of the Concurrent Whiteboard demonstrator into Gaudi and characterisation of its performance

Allow Gaudi to process multiple events simultaneously and algorithms concurrently. To achieve this goal these features were implemented:

  • A scheduler which schedules the algorithms as soon as their input is available
  • The event loop manager which pulls in an arbitrary number of events
  • Thread safety of a restricted core of services and registration mechanisms: message service (see here the presentation of Marco C.), reference counting, minimal features of the data service.
  • CPUCruncher algorithm which keeps the CPU busy (finding prime numbers) for a configurable amount of time. Calibration performed in order to mimic the real behaviour of the application. No real algorithms scheduled, just CPUCrunchers.
Completed in December 2012 See the papers session for more informations.

Runtime of event loop scaled to the sequential case as a function of the number of the maximum number of algorithms processed simultaneously. Here number crunching algorithms aiming only to keep the CPUs busy were used instead of the real ones.

Open Tasks

See the C4Hep project in Jira.

Testing, Debugging and Profiling

See C4HepDebuggingAndProfiling

Git Tips and Tricks

See C4HepTipsAndTricks

Useful Links

  • Second Atlas-LHCb joint sprint: here
  • Concurrent Gaudi Nightlies (from outside CERN a tunnel is needed): here
  • GaudiMT repository web interface here
  • Concurrent Framework project in the Concurrency Forum website here
  • Minutes of the working meetings here
  • Concurrent Framework in Jira here
  • Concurrent Whiteboard Demonstrator in the Concurrency Forum website here
  • The Gaudi Project here and the Gaudi long paper here

Publications and other useful documents

Please refer to the Concurrent Gaudi Project page on the Concurrency forum website here

Twiki Web Utilities

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng MiniBrunelScaling.png r1 manage 39.1 K 2013-06-28 - 10:56 DaniloPiparo Minibrunel Scaling
PNGpng ProtoPerf.png r1 manage 251.4 K 2013-10-22 - 18:43 DaniloPiparo Performance of the Concurrent Gaudi Prototype
Edit | Attach | Watch | Print version | History: r58 < r57 < r56 < r55 < r54 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r58 - 2014-10-20 - MarcoClemencic
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    C4Hep All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback