System Configuration

The LHCb Nightly Build System consists of a few key components that need to be configured in a consistent way to function correctly.

Here we describe the required settings divided per key component. The order of the following sections was chosen to simplify the connection step, but is not compulsory.

The required applications, if not specified otherwise, are installed with the regular Scientific Linux CERN 6 packages.

Service Account (lhcbsoft)

A service account is used to run the build jobs and perform all the automated tasks.

It is important that the chosen service account have password-less ssh access across the hosts used to operate the Nightly Build System. It should also have the home directory on a shared filesystem with a directory called private with restricted access (read/write only for the user).

The regular tasks performed by the service account (clean up of temporary directories, etc.) are triggered using the acrontab tool.

We use the account lhcbsoft for this purpose, with the home directory on AFS.

Hosts

Of course, to set up the Nightly Build System, we need a few machines:

• the master host: it runs the main services and coordinates the activities
• build hosts (optional): the machines on which the builds will run (one could use the master as a build host too, if it's powerful enough)

The different services do not not need to run on the same host, but it simplify the configuration.

At the moment, some scripts rely on the master host to be called buildlhcb.cern.ch.

Marster Host

The master host must be configured with a big /data partition that will hold configuration data, databases and build products.

The /data directory must contain the following directories:

• artifacts, owned by the service account (lhcbsoft)
• couchdb, owned by the account used to run the CouchDB server
• jenkins_home, owned by the account running the Tomcat server
• jenkins_home/nightlies, still owned by the Tomcat account, but writable by a group which the service account belongs to

Web Server

The services, tools, data files, summaries, etc. are accessed via an Apache HTTPD instance (running on the master host) which must be configured with the following features/modules:

• HTTPS
• rewrite
• proxy
• proxy_http
• proxy_ajp
• cache
• disk_cache
• Shibboleth SSO

The server should be also configured to follow symlinks on the filesystem.

The parts of the configuration for the different services will be described together with the configuration of the services in the following sections.

Dashboard

The Nightly Builds Dashboard is a web application backed by a CouchDB server, running on the master host.

Several instances of the Dashboard can be configured, each one with its own database, to support different flavours of the builds (regular nightly builds, testing builds, periodic tests, etc.).

CouchDB

The CouchDB server (installed with rpm and configured to start at boot), must be set up with an administrator account (e.g. using the service account name). The configuration file /etc/couchdb/local.ini must be modified to use /data/couchdb as database_dir and view_index_dir.

To support multiple build flavours, we must create one database per flavour, called nightlies-.

Each database should contain the CouchDB Web Application (CouchApp) that presents the content of the database as an HTML-based dashboard. The code of the application have been developed using the erica CouchApp framework and is kept in the couchdb/dashboard directory of the project LbNightlyTools.

The file /etc/sysconfig/couchdb should be modified adding the line

export ERL_ZFLAGS="-env HEART_BEAT_TIMEOUT 60"

to change the default heart-beat from 11s to 60s (sometimes the host may be a bit unresponsive).

The Web Server should be configured to map, via a reverse proxy call, the HTTP path /nightlies-/* to http://localhost:5984/nightlies-/_design/dashboard/_rewrite/*. Those paths should also be cached with the disk_cache module.

The admin credentials must be stored in the file private/couchdb-admin in the home directory of the service account.

Jenkins

Jenkins CI is an open source continuous integration system that we use to schedule the jobs of the Nightly Build System.

Jenkins is configured as a Tomcat servlet.

Master

Slaves

-- MarcoClemencic - 12 Feb 2014