This pages describes the services maintained by the LHCb Core Software teams and the associated maintenance procedures.
LHCb Software Management tools
Continuous Integration
The following diagram describes the LHCb Continuous Integration system:
LHCb Physics Software sources
The LHCb projects are hosted in the gitlab instance, hosted at CERN, in the "lhcb" group:
https://gitlab.cern.ch/lhcb
Permissions:
* Group Admin: The LHCb Group is owned by the egroup
lhcb-software-librarians@cernNOSPAMPLEASE.ch (using LDAP synchronization).
* Project Admin: Should be granted project per project, using the dedicated page in the project.
Nightly builds & Unit testing
The LHCb Core Software runs a continuous ntegration system described in:
https://twiki.cern.ch/twiki/bin/view/LHCb/LHCbNightlies
The status of the builds can be found at:
https://lhcb-nightlies.cern.ch/nightly/
The configuration for the system can be found in git at:
https://gitlab.cern.ch/lhcb-core/LHCbNightlyConf
(Some description of the Nightly build slots is
here)
The nightly builds are driven by the following instance of Jenkins:
https://jenkins-lhcb-nightlies.web.cern.ch/
Installation of the nightly builds
After build the nightly slots are installed on the CVMFS volume:
/cvmfs/lhcbdev.cern.ch
.
CIAgent and messaging
When a slot is build, a message is pushed to
RabbitMQ BuildReadyCIAgent queue. The message is then converted by CIAgent to a message for
RabbitMQ CVMFSDevActions queue and assigned a low priority. This queue combines messages from Jenkins Remote Cron Jobs with a higher priority and build ready messages with a low priority. The messages are consumed on this priority.
In order to debug the installation system, please refer to
https://twiki.cern.ch/twiki/bin/view/LHCb/NightlyBuildsInstallationDebug
CVMFS Dev release documentation:
https://twiki.cern.ch/twiki/bin/view/LHCb/CVMFSDevReleaseDocumentation
Messaging between Jenkins and the CVMFS LHCb Stratum-0 node (cvmfs-lhcbdev) is hosted via an instance of the
rabbitmq
messaging engine described below.
CVMFS dev interactive mode
In order to execute administration operations on CVMFS lhcbdev.cern.ch, you can use
lbcvmfsinteractive.sh -m [Reason for the intervation]
which guarantees that the scheduler has finished any unpublish transactions. In this shell, the administrator can start, publish or abort new transactions. When the shell is exited, the schedulers retakes possession of the transactions lock. If no reason is given, you will be prompted to enter a reason.
CVMFS dev disk usage statistics:
or
see here
Monitoring of the CVMFS dev stratum 0
http://cernvm-monitor.cern.ch/cvmfs-monitor/cb/browser/lhcbdev.cern.ch/latest/?page=0
Kibana monitoring dashboard, see
here
Releases
https://twiki.cern.ch/twiki/bin/view/LHCb/ProjectRelease
- The status of the build can be seen:
https://lhcb-nightlies.cern.ch/release/
- Deployment task management system:
https://sft.its.cern.ch/jira/projects/LHCBDEP
Update of the project list on CVMFS
a cgi-bin on ariadne-lhcb provides this information read-only, without needing credentials:
http://ariadne-lhcb.cern.ch/cgi-bin/lb-projects
/home/cvlhcb/bin/update_lb_projects.sh that polls that URL and downloads the file to:
/cvmfs/lhcb.cern.ch/lib/etc/lb_projects.json
A cron was added to run that script every hour and the log can be found here:
/home/cvlhcb/logs/update_lb_projects.log
Infrastructure Services
Jenkins for physics projects
Jenkins for Core Software tools
Local Pypi repository
Neo4j graph database
RabbitMQ messaging engine
The server runs on lbmessagingbroker.cern.ch
To connect to the admin interface use port
http://lbmessagingbroker.cern.ch:15672
.
To automatically provision a local instance of
RabbitMQ instance with the LHCb configuration use:
https://twiki.cern.ch/twiki/pub/LHCb/LHCbDevOps/LHCbProvisionRabbitMQ.sh
- Obsolete
- To add new users or virtual hosts please look at https://twiki.cern.ch/twiki/bin/view/LHCb/CreatingAccessToRabbitMq (use scripts instead)
For some adminitration tips and tricks, see
here:
Kibana monitoring dashboard, see
here
LHCb Installation and environment configuration tools
Most of the core software tools are versioned in gitlab, at the following address:
https://gitlab.cern.ch/lhcb-core
* Git:
https://gitlab.cern.ch/lhcb-core/lbinstall
* Doc:
http://lbinstall.web.cern.ch/lbinstall/
https://gitlab.cern.ch/lhcb-core/lbscripts
To release:
https://twiki.cern.ch/twiki/bin/view/LHCb/LbScriptsRelease
--
BenjaminCouturier - 2016-04-21