TWiki> LHCb Web>LHCbComputing>LHCbDevOps (revision 20)EditAttachPDF

This pages describes the services maintained by the LHCb Core Software teams and the associated maintenance procedures.

LHCb Software Management tools

Projects in current development

Continuous Integration

The following diagram describes the LHCb Continuous Integration system:


LHCb Physics Software sources

The LHCb projects are hosted in the gitlab instance, hosted at CERN, in the "lhcb" group:

Permissions: * Group Admin: The LHCb Group is owned by the egroup (using LDAP synchronization). * Project Admin: Should be granted project per project, using the dedicated page in the project.

Nightly builds & Unit testing

The LHCb Core Software runs a continuous ntegration system described in:

The status of the builds can be found at:

The configuration for the system can be found in git at:

The nightly builds are driven by the following instance of Jenkins:

Installation of the nightly builds

After build the nightly slots are installed on the CVMFS volume: /cvmfs/

CIAgent and messaging

When a slot is build, a message is pushed to RabbitMQ BuildReadyCIAgent queue. The message is then converted by CIAgent to a message for RabbitMQ CVMFSDevActions queue and assigned a low priority. This queue combines messages from Jenkins Remote Cron Jobs with a higher priority and build ready messages with a low priority. The messages are consumed on this priority.

In order to debug the installation system, please refer to

CVMFS Dev release documentation:

Messaging between Jenkins and the CVMFS LHCb Stratum-0 node (cvmfs-lhcbdev) is hosted via an instance of the rabbitmq messaging engine described below.

CVMFS dev interactive mode

In order to execute administration operations on CVMFS, you can use -m  [Reason for the intervation] which guarantees that the scheduler has finished any unpublish transactions. In this shell, the administrator can start, publish or abort new transactions. When the shell is exited, the schedulers retakes possession of the transactions lock. If no reason is given, you will be prompted to enter a reason.

CVMFS dev disk usage statistics:


or see here

Monitoring of the CVMFS dev stratum 0

Kibana monitoring dashboard, see here



  • Main documentation:

  • The status of the build can be seen:

  • Deployment task management system:

Update of the project list on CVMFS

a cgi-bin on ariadne-lhcb provides this information read-only, without needing credentials:

/home/cvlhcb/bin/ that polls that URL and downloads the file to:


A cron was added to run that script every hour and the log can be found here:


Infrastructure Services

Jenkins for physics projects

Jenkins for Core Software tools

Local Pypi repository

Neo4j graph database

RabbitMQ messaging engine

The server runs on

To connect to the admin interface use port

To automatically provision a local instance of RabbitMQ instance with the LHCb configuration use:

To add new users or virtual hosts please look at (use scripts instead)

For some adminitration tips and tricks, see here:

Kibana monitoring dashboard, see here

LHCb Installation and environment configuration tools

Most of the core software tools are versioned in gitlab, at the following address:


* Git:

* Doc:

LbScripts & LbLogin

To release:

-- BenjaminCouturier - 2016-04-21

Topic attachments
I Attachment History Action Size Date Who Comment
Unix shell scriptsh r1 manage 1.9 K 2018-01-16 - 10:40 UnknownUser  
PNGpng sdev.png r1 manage 178.4 K 2017-08-31 - 13:49 BenjaminCouturier  
PNGpng srel.png r1 manage 188.3 K 2017-08-31 - 13:49 BenjaminCouturier  
Edit | Attach | Watch | Print version | History: r22 < r21 < r20 < r19 < r18 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r20 - 2019-02-21 - BenjaminCouturier
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback