This pages describes the services maintained by the LHCb Core Software teams and the associated maintenance procedures.

LHCb Software Management tools

Continuous Integration

The following diagram describes the LHCb Continuous Integration system:

LHCb Physics Software sources

The LHCb projects are hosted in the gitlab instance, hosted at CERN, in the "lhcb" group:

https://gitlab.cern.ch/lhcb

Permissions: * Group Admin: The LHCb Group is owned by the egroup lhcb-software-librarians@cernNOSPAMPLEASE.ch (using LDAP synchronization). * Project Admin: Should be granted project per project, using the dedicated page in the project.

Nightly builds & Unit testing

The LHCb Core Software runs a continuous ntegration system described in:

https://twiki.cern.ch/twiki/bin/view/LHCb/LHCbNightlies

The status of the builds can be found at:

https://lhcb-nightlies.cern.ch/nightly/

The configuration for the system can be found in git at:

https://gitlab.cern.ch/lhcb-core/LHCbNightlyConf

(Some description of the Nightly build slots is here)

The nightly builds are driven by the following instance of Jenkins:

https://jenkins-lhcb-nightlies.web.cern.ch/

Installation of the nightly builds

After build the nightly slots are installed on the CVMFS volume: /cvmfs/lhcbdev.cern.ch.

CIAgent and messaging

When a slot is build, a message is pushed to RabbitMQ BuildReadyCIAgent queue. The message is then converted by CIAgent to a message for RabbitMQ CVMFSDevActions queue and assigned a low priority. This queue combines messages from Jenkins Remote Cron Jobs with a higher priority and build ready messages with a low priority. The messages are consumed on this priority.

In order to debug the installation system, please refer to https://twiki.cern.ch/twiki/bin/view/LHCb/NightlyBuildsInstallationDebug

CVMFS Dev release documentation: https://twiki.cern.ch/twiki/bin/view/LHCb/CVMFSDevReleaseDocumentation

Messaging between Jenkins and the CVMFS LHCb Stratum-0 node (cvmfs-lhcbdev) is hosted via an instance of the rabbitmq messaging engine described below.

CVMFS dev interactive mode

In order to execute administration operations on CVMFS lhcbdev.cern.ch, you can use lbcvmfsinteractive.sh -m  [Reason for the intervation] which guarantees that the scheduler has finished any unpublish transactions. In this shell, the administrator can start, publish or abort new transactions. When the shell is exited, the schedulers retakes possession of the transactions lock. If no reason is given, you will be prompted to enter a reason.

CVMFS dev disk usage statistics:

or see here

Monitoring of the CVMFS dev stratum 0

http://cernvm-monitor.cern.ch/cvmfs-monitor/cb/browser/lhcbdev.cern.ch/latest/?page=0

Kibana monitoring dashboard, see here

Releases

• Main documentation:

https://twiki.cern.ch/twiki/bin/view/LHCb/ProjectRelease

• The status of the build can be seen:

https://lhcb-nightlies.cern.ch/release/

• Deployment task management system:

https://sft.its.cern.ch/jira/projects/LHCBDEP

Update of the project list on CVMFS

a cgi-bin on ariadne-lhcb provides this information read-only, without needing credentials:

http://ariadne-lhcb.cern.ch/cgi-bin/lb-projects

/home/cvlhcb/bin/update_lb_projects.sh that polls that URL and downloads the file to:

/cvmfs/lhcb.cern.ch/lib/etc/lb_projects.json

A cron was added to run that script every hour and the log can be found here:

/home/cvlhcb/logs/update_lb_projects.log

Infrastructure Services

Jenkins for physics projects

• Jenkins: https://jenkins-lhcb-nightlies.web.cern.ch/

Jenkins for Core Software tools

• Jenkins: https://jenkins-lhcb-core-soft.web.cern.ch/

Local Pypi repository

• Usage: https://twiki.cern.ch/twiki/bin/view/LHCb/PythonPyPIServer

• Location: https://lhcb-pypi.web.cern.ch/

Neo4j graph database

• Neo4J: https://ariadne-lhcb.cern.ch/webadmin/

• URL for requested releases JSON: http://ariadne-lhcb.cern.ch/cgi-bin/getreleases

• URL for the LHCb list of projects: http://ariadne-lhcb.cern.ch/cgi-bin/lb-projects

RabbitMQ messaging engine

The server runs on lbmessagingbroker.cern.ch

To connect to the admin interface use port http://lbmessagingbroker.cern.ch:15672.

To automatically provision a local instance of RabbitMQ instance with the LHCb configuration use: https://twiki.cern.ch/twiki/pub/LHCb/LHCbDevOps/LHCbProvisionRabbitMQ.sh

Obsolete

To add new users or virtual hosts please look at https://twiki.cern.ch/twiki/bin/view/LHCb/CreatingAccessToRabbitMq (use scripts instead)

For some adminitration tips and tricks, see here:

Kibana monitoring dashboard, see here

LHCb Installation and environment configuration tools

Most of the core software tools are versioned in gitlab, at the following address:

https://gitlab.cern.ch/lhcb-core

LbInstall

* Git: https://gitlab.cern.ch/lhcb-core/lbinstall

* Doc: http://lbinstall.web.cern.ch/lbinstall/

LbScripts & LbLogin

https://gitlab.cern.ch/lhcb-core/lbscripts

To release:

https://twiki.cern.ch/twiki/bin/view/LHCb/LbScriptsRelease

-- BenjaminCouturier - 2016-04-21