Difference: LHCbDevOps (1 vs. 22)

Revision 222019-08-14 - BenjaminCouturier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Deleted:
<
<
 
Changed:
<
<
This pages describes the services maintained by the LHCb Core Software teams and the associated maintenance procedures.

LHCb Software Management tools


Projects in current development

https://twiki.cern.ch/twiki/bin/view/LHCb/InDevelopmentProjects

Continuous Integration

The following diagram describes the LHCb Continuous Integration system:

sdev.png

LHCb Physics Software sources

The LHCb projects are hosted in the gitlab instance, hosted at CERN, in the "lhcb" group:

https://gitlab.cern.ch/lhcb

Permissions: * Group Admin: The LHCb Group is owned by the egroup lhcb-software-librarians@cernNOSPAMPLEASE.ch (using LDAP synchronization). * Project Admin: Should be granted project per project, using the dedicated page in the project.

Nightly builds & Unit testing

The LHCb Core Software runs a continuous ntegration system described in:

https://twiki.cern.ch/twiki/bin/view/LHCb/LHCbNightlies

The status of the builds can be found at:

https://lhcb-nightlies.cern.ch/nightly/

The configuration for the system can be found in git at:

https://gitlab.cern.ch/lhcb-core/LHCbNightlyConf

The nightly builds are driven by the following instance of Jenkins:

https://jenkins-lhcb-nightlies.web.cern.ch/

Installation of the nightly builds

After build the nightly slots are installed on the CVMFS volume: /cvmfs/lhcbdev.cern.ch.

CIAgent and messaging

When a slot is build, a message is pushed to RabbitMQ BuildReadyCIAgent queue. The message is then converted by CIAgent to a message for RabbitMQ CVMFSDevActions queue and assigned a low priority. This queue combines messages from Jenkins Remote Cron Jobs with a higher priority and build ready messages with a low priority. The messages are consumed on this priority.

For some administration tips and tricks about the messages and the messages queues on the CIAgent , see here:

In order to debug the installation system, please refer to https://twiki.cern.ch/twiki/bin/view/LHCb/NightlyBuildsInstallationDebug

CVMFS Dev release documentation: https://twiki.cern.ch/twiki/bin/view/LHCb/CVMFSDevReleaseDocumentation

Messaging between Jenkins and the CVMFS LHCb Stratum-0 node (cvmfs-lhcbdev) is hosted via an instance of the rabbitmq messaging engine described below.

CVMFS dev interactive mode

In order to execute administration operations on CVMFS lhcbdev.cern.ch, you can use lbcvmfsinteractive.sh -m  [Reason for the intervation] which guarantees that the scheduler has finished any unpublish transactions. In this shell, the administrator can start, publish or abort new transactions. When the shell is exited, the schedulers retakes possession of the transactions lock. If no reason is given, you will be prompted to enter a reason.

CVMFS dev disk usage statistics:

cvmfs_stats.png

or see here

Monitoring of the CVMFS dev stratum 0

http://cernvm-monitor.cern.ch/cvmfs-monitor/cb/browser/lhcbdev.cern.ch/latest/?page=0

Kibana monitoring dashboard, see here

Releases

srel.png

  • Main documentation:

https://twiki.cern.ch/twiki/bin/view/LHCb/ProjectRelease

  • The status of the build can be seen:

https://lhcb-nightlies.cern.ch/release/

  • Deployment task management system:

https://sft.its.cern.ch/jira/projects/LHCBDEP

Update of the project list on CVMFS

a cgi-bin on ariadne-lhcb provides this information read-only, without needing credentials:

http://ariadne-lhcb.cern.ch/cgi-bin/lb-projects

/home/cvlhcb/bin/update_lb_projects.sh that polls that URL and downloads the file to:

/cvmfs/lhcb.cern.ch/lib/etc/lb_projects.json

A cron was added to run that script every hour and the log can be found here:

/home/cvlhcb/logs/update_lb_projects.log

Infrastructure Services


Jenkins for physics projects

Jenkins for Core Software tools

Local Pypi repository

Neo4j graph database

RabbitMQ messaging engine

The server runs on lbmessagingbroker.cern.ch

To connect to the admin interface use port http://lbmessagingbroker.cern.ch:15672.

To automatically provision a local instance of RabbitMQ instance with the LHCb configuration use: https://twiki.cern.ch/twiki/pub/LHCb/LHCbDevOps/LHCbProvisionRabbitMQ.sh

Obsolete
To add new users or virtual hosts please look at https://twiki.cern.ch/twiki/bin/view/LHCb/CreatingAccessToRabbitMq (use scripts instead)

For some administration tips and tricks, see here:

Kibana monitoring dashboard, see here

LHCb Installation and environment configuration tools


Most of the core software tools are versioned in gitlab, at the following address:

https://gitlab.cern.ch/lhcb-core

LbInstall

* Git: https://gitlab.cern.ch/lhcb-core/lbinstall

* Doc: http://lbinstall.web.cern.ch/lbinstall/

LbScripts & LbLogin

https://gitlab.cern.ch/lhcb-core/lbscripts

To release:

https://twiki.cern.ch/twiki/bin/view/LHCb/LbScriptsRelease

-- BenjaminCouturier - 2016-04-21

>
>
Please see: https://cern.ch/lbdevops/
 
META FILEATTACHMENT attachment="sdev.png" attr="" comment="" date="1504180173" name="sdev.png" path="sdev.png" size="182658" user="bcouturi" version="1"
META FILEATTACHMENT attachment="srel.png" attr="" comment="" date="1504180173" name="srel.png" path="srel.png" size="192813" user="bcouturi" version="1"

Revision 212019-02-21 - StefanGabrielChitic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 54 to 54
  When a slot is build, a message is pushed to RabbitMQ BuildReadyCIAgent queue. The message is then converted by CIAgent to a message for RabbitMQ CVMFSDevActions queue and assigned a low priority. This queue combines messages from Jenkins Remote Cron Jobs with a higher priority and build ready messages with a low priority. The messages are consumed on this priority.
Added:
>
>
For some administration tips and tricks about the messages and the messages queues on the CIAgent , see here:
 In order to debug the installation system, please refer to https://twiki.cern.ch/twiki/bin/view/LHCb/NightlyBuildsInstallationDebug

CVMFS Dev release documentation: https://twiki.cern.ch/twiki/bin/view/LHCb/CVMFSDevReleaseDocumentation

Line: 146 to 148
 
Obsolete
To add new users or virtual hosts please look at https://twiki.cern.ch/twiki/bin/view/LHCb/CreatingAccessToRabbitMq (use scripts instead)

Changed:
<
<
For some adminitration tips and tricks, see here:
>
>
For some administration tips and tricks, see here:
  Kibana monitoring dashboard, see here

Revision 202019-02-21 - BenjaminCouturier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 42 to 42
  https://gitlab.cern.ch/lhcb-core/LHCbNightlyConf
Deleted:
<
<
(Some description of the Nightly build slots is here)
 The nightly builds are driven by the following instance of Jenkins:

https://jenkins-lhcb-nightlies.web.cern.ch/

Revision 192018-08-28 - StefanGabrielChitic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 7 to 7
 

LHCb Software Management tools


Added:
>
>

Projects in current development

https://twiki.cern.ch/twiki/bin/view/LHCb/InDevelopmentProjects

 

Continuous Integration

The following diagram describes the LHCb Continuous Integration system:

Revision 182018-04-11 - BenjaminCouturier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 91 to 91
  https://sft.its.cern.ch/jira/projects/LHCBDEP
Added:
>
>

Update of the project list on CVMFS

a cgi-bin on ariadne-lhcb provides this information read-only, without needing credentials:

http://ariadne-lhcb.cern.ch/cgi-bin/lb-projects

/home/cvlhcb/bin/update_lb_projects.sh that polls that URL and downloads the file to:

/cvmfs/lhcb.cern.ch/lib/etc/lb_projects.json

A cron was added to run that script every hour and the log can be found here:

/home/cvlhcb/logs/update_lb_projects.log

 

Infrastructure Services


Line: 114 to 132
 
Added:
>
>
 

RabbitMQ messaging engine

The server runs on lbmessagingbroker.cern.ch

Revision 172018-04-06 - StefanGabrielChitic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 58 to 58
  Messaging between Jenkins and the CVMFS LHCb Stratum-0 node (cvmfs-lhcbdev) is hosted via an instance of the rabbitmq messaging engine described below.
Added:
>
>

CVMFS dev interactive mode

In order to execute administration operations on CVMFS lhcbdev.cern.ch, you can use lbcvmfsinteractive.sh -m  [Reason for the intervation] which guarantees that the scheduler has finished any unpublish transactions. In this shell, the administrator can start, publish or abort new transactions. When the shell is exited, the schedulers retakes possession of the transactions lock. If no reason is given, you will be prompted to enter a reason.

 

CVMFS dev disk usage statistics:

cvmfs_stats.png

Revision 162018-03-29 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 91 to 91
 

Jenkins for physics projects

Changed:
<
<
>
>
 

Jenkins for Core Software tools

Revision 152018-02-15 - BenjaminCouturier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 101 to 101
 
Changed:
<
<
>
>
 

Neo4j graph database

Revision 142018-01-22 - BenjaminCouturier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 19 to 19
  https://gitlab.cern.ch/lhcb
Added:
>
>
Permissions: * Group Admin: The LHCb Group is owned by the egroup lhcb-software-librarians@cernNOSPAMPLEASE.ch (using LDAP synchronization). * Project Admin: Should be granted project per project, using the dedicated page in the project.
 

Nightly builds & Unit testing

The LHCb Core Software runs a continuous ntegration system described in:

Revision 132018-01-19 - StefanGabrielChitic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 63 to 63
  http://cernvm-monitor.cern.ch/cvmfs-monitor/cb/browser/lhcbdev.cern.ch/latest/?page=0
Added:
>
>
Kibana monitoring dashboard, see here
 

Releases

srel.png

Line: 114 to 116
  For some adminitration tips and tricks, see here:
Added:
>
>
Kibana monitoring dashboard, see here
 

LHCb Installation and environment configuration tools

Revision 122018-01-16 - BenjaminCouturier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 43 to 43
  After build the nightly slots are installed on the CVMFS volume: /cvmfs/lhcbdev.cern.ch.
Added:
>
>

CIAgent and messaging

 When a slot is build, a message is pushed to RabbitMQ BuildReadyCIAgent queue. The message is then converted by CIAgent to a message for RabbitMQ CVMFSDevActions queue and assigned a low priority. This queue combines messages from Jenkins Remote Cron Jobs with a higher priority and build ready messages with a low priority. The messages are consumed on this priority.

In order to debug the installation system, please refer to https://twiki.cern.ch/twiki/bin/view/LHCb/NightlyBuildsInstallationDebug

Line: 51 to 53
  Messaging between Jenkins and the CVMFS LHCb Stratum-0 node (cvmfs-lhcbdev) is hosted via an instance of the rabbitmq messaging engine described below.
Changed:
<
<
CVMFS dev disk usage statistics:
>
>

CVMFS dev disk usage statistics:

  cvmfs_stats.png

or see here

Changed:
<
<
Monitoring of the CVMFS dev stratum 0:
>
>

Monitoring of the CVMFS dev stratum 0

  http://cernvm-monitor.cern.ch/cvmfs-monitor/cb/browser/lhcbdev.cern.ch/latest/?page=0

Revision 112018-01-16 - BenjaminCouturier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 107 to 107
  To connect to the admin interface use port http://lbmessagingbroker.cern.ch:15672.

To automatically provision a local instance of RabbitMQ instance with the LHCb configuration use: https://twiki.cern.ch/twiki/pub/LHCb/LHCbDevOps/LHCbProvisionRabbitMQ.sh

Added:
>
>
Obsolete
To add new users or virtual hosts please look at https://twiki.cern.ch/twiki/bin/view/LHCb/CreatingAccessToRabbitMq (use scripts instead)

For some adminitration tips and tricks, see here:

 
Deleted:
<
<
To add new users or virtual hosts please look at https://twiki.cern.ch/twiki/bin/view/LHCb/CreatingAccessToRabbitMq
 

LHCb Installation and environment configuration tools


Revision 102018-01-16 - StefanGabrielChitic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 106 to 106
  To connect to the admin interface use port http://lbmessagingbroker.cern.ch:15672.
Added:
>
>
To automatically provision a local instance of RabbitMQ instance with the LHCb configuration use: https://twiki.cern.ch/twiki/pub/LHCb/LHCbDevOps/LHCbProvisionRabbitMQ.sh
  To add new users or virtual hosts please look at https://twiki.cern.ch/twiki/bin/view/LHCb/CreatingAccessToRabbitMq

LHCb Installation and environment configuration tools

Line: 133 to 135
 
META FILEATTACHMENT attachment="sdev.png" attr="" comment="" date="1504180173" name="sdev.png" path="sdev.png" size="182658" user="bcouturi" version="1"
META FILEATTACHMENT attachment="srel.png" attr="" comment="" date="1504180173" name="srel.png" path="srel.png" size="192813" user="bcouturi" version="1"
Added:
>
>
META FILEATTACHMENT attachment="LHCbProvisionRabbitMQ.sh" attr="" comment="" date="1516095651" name="LHCbProvisionRabbitMQ.sh" path="LHCbProvisionRabbitMQ.sh" size="1974" user="stchitic" version="1"

Revision 92018-01-10 - StefanGabrielChitic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 106 to 106
  To connect to the admin interface use port http://lbmessagingbroker.cern.ch:15672.
Added:
>
>
To add new users or virtual hosts please look at https://twiki.cern.ch/twiki/bin/view/LHCb/CreatingAccessToRabbitMq
 

LHCb Installation and environment configuration tools


Revision 82017-12-20 - StefanGabrielChitic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 43 to 43
  After build the nightly slots are installed on the CVMFS volume: /cvmfs/lhcbdev.cern.ch.
Added:
>
>
When a slot is build, a message is pushed to RabbitMQ BuildReadyCIAgent queue. The message is then converted by CIAgent to a message for RabbitMQ CVMFSDevActions queue and assigned a low priority. This queue combines messages from Jenkins Remote Cron Jobs with a higher priority and build ready messages with a low priority. The messages are consumed on this priority.

In order to debug the installation system, please refer to https://twiki.cern.ch/twiki/bin/view/LHCb/NightlyBuildsInstallationDebug

 CVMFS Dev release documentation: https://twiki.cern.ch/twiki/bin/view/LHCb/CVMFSDevReleaseDocumentation

Messaging between Jenkins and the CVMFS LHCb Stratum-0 node (cvmfs-lhcbdev) is hosted via an instance of the rabbitmq messaging engine described below.

Revision 72017-11-20 - JoelClosier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 88 to 88
 
Changed:
<
<
>
>
 

Neo4j graph database

Revision 62017-11-17 - BenjaminCouturier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 98 to 98
 

RabbitMQ messaging engine

Changed:
<
<
The server runs on lbmessagingbroker.cern.ch.
>
>
The server runs on lbmessagingbroker.cern.ch
 
Changed:
<
<
To connect to the admin interface use port 15672.
>
>
To connect to the admin interface use port http://lbmessagingbroker.cern.ch:15672.
 

LHCb Installation and environment configuration tools


Revision 52017-11-10 - StefanGabrielChitic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 43 to 43
  After build the nightly slots are installed on the CVMFS volume: /cvmfs/lhcbdev.cern.ch.
Changed:
<
<
XXX Link CVMFS Dev release documentation
>
>
CVMFS Dev release documentation: https://twiki.cern.ch/twiki/bin/view/LHCb/CVMFSDevReleaseDocumentation
  Messaging between Jenkins and the CVMFS LHCb Stratum-0 node (cvmfs-lhcbdev) is hosted via an instance of the rabbitmq messaging engine described below.
Line: 82 to 82
 

Jenkins for Core Software tools

Added:
>
>

Local Pypi repository

 

Neo4j graph database

Line: 103 to 111
 

LbInstall

Changed:
<
<
https://gitlab.cern.ch/lhcb-core/lbinstall
>
>
* Git: https://gitlab.cern.ch/lhcb-core/lbinstall

* Doc: http://lbinstall.web.cern.ch/lbinstall/

 

LbScripts & LbLogin

Revision 42017-11-06 - MaciejSzymanski

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

This pages describes the services maintained by the LHCb Core Software teams and the associated maintenance procedures.

Deleted:
<
<


 

LHCb Software Management tools


Line: 84 to 73
  https://sft.its.cern.ch/jira/projects/LHCBDEP
Deleted:
<
<


 

Infrastructure Services


Line: 104 to 90
 

RabbitMQ messaging engine

Changed:
<
<
The server runs on lbjenkins.cern.ch

To connect to the admin interface, create a tunnel to port 15672

>
>
The server runs on lbmessagingbroker.cern.ch.
 
Added:
>
>
To connect to the admin interface use port 15672.
 
Deleted:
<
<



 

LHCb Installation and environment configuration tools


Revision 32017-11-02 - BenjaminCouturier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 52 to 52
  Messaging between Jenkins and the CVMFS LHCb Stratum-0 node (cvmfs-lhcbdev) is hosted via an instance of the rabbitmq messaging engine described below.
Added:
>
>
CVMFS dev disk usage statistics:
 
Added:
>
>
cvmfs_stats.png

or see here

Monitoring of the CVMFS dev stratum 0:

http://cernvm-monitor.cern.ch/cvmfs-monitor/cb/browser/lhcbdev.cern.ch/latest/?page=0

 

Releases

Revision 22017-08-31 - BenjaminCouturier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Changed:
<
<

LHCb Software Dev infrastructure description

>
>
This pages describes the services maintained by the LHCb Core Software teams and the associated maintenance procedures.



LHCb Software Management tools


 

Continuous Integration

Changed:
<
<
The nightly builds status can be found at the following address:
>
>
The following diagram describes the LHCb Continuous Integration system:

sdev.png

LHCb Physics Software sources

The LHCb projects are hosted in the gitlab instance, hosted at CERN, in the "lhcb" group:

https://gitlab.cern.ch/lhcb

Nightly builds & Unit testing

The LHCb Core Software runs a continuous ntegration system described in:

https://twiki.cern.ch/twiki/bin/view/LHCb/LHCbNightlies

The status of the builds can be found at:

  https://lhcb-nightlies.cern.ch/nightly/
Changed:
<
<
The description of the Nightly build slots is here
>
>
The configuration for the system can be found in git at:

https://gitlab.cern.ch/lhcb-core/LHCbNightlyConf

(Some description of the Nightly build slots is here)

The nightly builds are driven by the following instance of Jenkins:

https://jenkins-lhcb-nightlies.web.cern.ch/

Installation of the nightly builds

After build the nightly slots are installed on the CVMFS volume: /cvmfs/lhcbdev.cern.ch.

XXX Link CVMFS Dev release documentation

Messaging between Jenkins and the CVMFS LHCb Stratum-0 node (cvmfs-lhcbdev) is hosted via an instance of the rabbitmq messaging engine described below.

 

Releases

Added:
>
>

srel.png

 
  • Main documentation:

https://twiki.cern.ch/twiki/bin/view/LHCb/ProjectRelease

Line: 27 to 74
  https://sft.its.cern.ch/jira/projects/LHCBDEP
Changed:
<
<

Infrastructure URLs

>
>


Infrastructure Services


Jenkins for physics projects

 
Added:
>
>

Jenkins for Core Software tools

Neo4j graph database

 

Added:
>
>

RabbitMQ messaging engine

The server runs on lbjenkins.cern.ch

To connect to the admin interface, create a tunnel to port 15672

 
Added:
>
>


LHCb Installation and environment configuration tools


Most of the core software tools are versioned in gitlab, at the following address:

https://gitlab.cern.ch/lhcb-core

LbInstall

https://gitlab.cern.ch/lhcb-core/lbinstall

LbScripts & LbLogin

https://gitlab.cern.ch/lhcb-core/lbscripts

To release:

https://twiki.cern.ch/twiki/bin/view/LHCb/LbScriptsRelease

 -- BenjaminCouturier - 2016-04-21
Added:
>
>
META FILEATTACHMENT attachment="sdev.png" attr="" comment="" date="1504180173" name="sdev.png" path="sdev.png" size="182658" user="bcouturi" version="1"
META FILEATTACHMENT attachment="srel.png" attr="" comment="" date="1504180173" name="srel.png" path="srel.png" size="192813" user="bcouturi" version="1"

Revision 12016-04-21 - BenjaminCouturier

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="LHCbComputing"

LHCb Software Dev infrastructure description

Continuous Integration

The nightly builds status can be found at the following address:

https://lhcb-nightlies.cern.ch/nightly/

The description of the Nightly build slots is here

Releases

  • Main documentation:

https://twiki.cern.ch/twiki/bin/view/LHCb/ProjectRelease

  • The status of the build can be seen:

https://lhcb-nightlies.cern.ch/release/

  • Deployment task management system:

https://sft.its.cern.ch/jira/projects/LHCBDEP

Infrastructure URLs

-- BenjaminCouturier - 2016-04-21

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback