BEWARE - This page is obsolete and has been superseded by the ProjectRelease Guide.

Software Release Guide

Scope

The following documentation applies to the release of LHCb Physics projects as well as data packages. Ganga is installed on a separate CVMFS volume, and there are specific instructions for DIRAC and LHCbDIrac.

Warning, important These instructions do not apply to LbScripts. Use LbScriptsRelease instead.

Overview of the procedure

The software is build by the Jenkins continuous integration engine. The configuration of the projects, their dependencies and which stacks should be built is defined in the software configuration database, with the typical procedure including the following steps:

  1. Release manager tags a project (or several) and triggers the build by imported the projects to the software configuration database.
  2. Jenkins builds the project(s) and prepares the RPM packages
  3. Release manager checks the outcome of the build and either restarts the build or requests a release in JIRA
  4. Release shifter copies the RPM packages to the official LHCb package repository (YUM format). At this point, anybody can perform local installations using lbinstall
  5. Release shifter marks the RPM as "released" in the software configuration database, and specifies which platforms have effectively been released. This is necessary for the build system to know what is released and isn't when building stacks, but also for the automated release procedures on AFS and CVMFS to know which platforms should be installed.
  6. Release shifter performs the installation on AFS and updates the documentation (still AFS dependent).
  7. Release shifter performs the installation on CVMFS

N.B. Installations on CVMFS and AFS can be performed in parallel.

Detailed instructions for all the steps are detailed below.

For Release Managers - Preparation of the build

For Projects

  • prepare the project as usual (release notes, tags, etc.), following the instructions at PrepareProjectReleaseGit
  • register the project/version to the software database
    lb-sdb-import MyProject vXrY
    
(If the tag is in git while the project is still by default maintained in SVN you need to tell lb-sdb-import to look in git, e.g.: lb-sdb-import  --sourceuri="gitlab-cern:LHCb/Panoramix#v23r2" Panoramix v23r2)

Within 10-15 min the build should start to appear in the nightlies page https://lhcb-nightlies.cern.ch/release. Note that, for a dependency tree of projects, lb-sdb-import can be called just once for the top level project as it will pull in all the dependencies (unless you need to provide the --sourceurl option, in which case each project has to be added individually, in the dependency order).

  • if there is a problem in the build, it can be re-started via the dedicated button (it will not restart by itself after a retag)
  • once satisfied by the build, take note of the build id (you can use the direct link icon) and make the request via LHCb Deployment Request form

For Data Packages

For Release Shifters - Release of the build

See the LHCb shift database to find who is this week's shifter.

Special information

Starting from 13 October 2016, you should set your environment to DEV as : ". /afs/cern.ch/lhcb/software/releases/LBSCRIPTS/dev/InstallArea/scripts/LbLogin.sh"

Prerequisites for new release shifters

If you are a new release shifter, please make sure you have the prerequisites that are needed at some point of the release procedure.

  • You should be added as a "developer" in the LHCb Deployment Jira project (https://sft.its.cern.ch/jira/browse/LHCBDEP/), to receive and track release requests
  • You must be member of the z5:librarians AFS group in order to have the correct access to the AFS areas where the software is installed.
  • You must be member or the _lhcb_ AFS group to have to have the AFS administrator rights that allow you to create AFS volumes for LHCb and to install the rpms in the LHCb rpm repository
  • You must have a valid grid certificate (For release of DecFiles package)
  • You must be added to the lhcb-cvmfs-librarians E-group (for access to the IT managed CVMFS installation machine and for communication with other release shifters)
  • Check that the CMTPATH environment variable is not defined in the shell which you are using.
  • You need to be registered as a software deployment shifter in the shifts database
  • You need to be registered to use Jenkins, to be able to trigger release builds from the web interface (for data packages)

As the shifter you can self-assign shifts through the shift database shift database. Usually this is done by block-booking weeks for individual shifters.

Using the checklist tool

The script lb-deployment-checklist simplifies the release shifter task by checking what needs to be done and either prints the next command to run or points to this page.

So connect to lxplus (or equivalent SLC6 machine) and run

lb-deployment-checklist MyProject vXrY
and follow the instructions.

The latest version of lb-deployment-checklist may cache some data to speed up the execution. If you think it's showing wrong messages, try to remove the directory ~/.cache/lb-deployment/.

Release of projects

Update of the RPM repository

The goal of this step is to make sure that:

  1. The RPMs have been copied to the YUM repository (curently in EOS)
  2. The YUM metadata has been updated (the lb-release-rpm internally uses the "createrepo" yum command).

Warning, important This step has to be performed from the host lhcb-archive.cern.ch

  • go there
    ssh lhcb-archive.cern.ch
    bash
    . /afs/cern.ch/lhcb/software/releases/LBSCRIPTS/dev/InstallArea/scripts/LbLogin.sh
    kinit Your_Username@CERN.CH
    

  • publish the produced RPMs (all of them, unless explicitly requested otherwise, see partial install section)
    • try to copy the RPMs to the repository
      build_id=1234
      lb-release-rpm --rpm-dir=/eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2018 /eos/project/l/lhcbwebsites/www/lhcb-nightlies-artifacts/release/lhcb-release/$build_id/rpms
      
      • For old stacks only released as tar files:
        lb-release-oldtar /eos/project/l/lhcbwebsites/www/lhcb-nightlies-artifacts/release/lhcb-release/$build_id/olddist
               
    • if everything is fine, do the real copy and update the repository info
      lb-release-rpm --rpm-dir=/eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2018 --copy /eos/project/l/lhcbwebsites/www/lhcb-nightlies-artifacts/release/lhcb-release/$build_id/rpms
      
      • For old stacks only released as tar files:
        lb-release-oldtar --copy /eos/project/l/lhcbwebsites/www/lhcb-nightlies-artifacts/release/lhcb-release/$build_id/olddist
        
  • Check it's all there
    ls -ltr /eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2018/ | tail -10
    
    In case of problems see here

BE CAREFUL: Since 20141010, the release procedure automatically bumps up the release number for packages for which there is already a version in the RPM repo. If your slot contains more projects that the ones requested for release, you need to ONLY copy the needed ones to the RPM repo. In that case the --rpm-regex option of lb-release-rpm should be used.

N.B. If you want to release all rpms EXCEPT some matching a specific string (e.g. gcc62) you can use:

lb-release-rpm -c --rpm-regex '^(?!.*(gcc62)).*'  --rpm-dir=/eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2018 /eos/project/l/lhcbwebsites/www/lhcb-nightlies-artifacts/release/lhcb-release/$build_id/rpms

Update of the Software configuration Database

The goal of this step is to make sure that:

  1. The project is referenced properly in the Software configuration DB
  2. The list of platforms released for the project have been updated

  • Verify that the software configuration DB has been updated:
    MyProject=<ProjectName>
    MyVersion=vXrY
    echo ${MyProject^^} ${MyVersion}
    lb-sdb-query d $MyProject $MyVersion
    
    This should show projects in the slot or already released. If some are missing re-run:
    lb-sdb-import $MyProject $MyVersion
    

  • update the software database.
    First try in dry-run mode:
    lb-deployment-updatesdb /eos/project/l/lhcbwebsites/www/lhcb-nightlies-artifacts/release/lhcb-release/$build_id
    
    If more platforms/projects are being updated than necessary you can limit using the options of the lb-deployment-updatesdb command.If all ok, just run:
    lb-deployment-updatesdb --update /eos/project/l/lhcbwebsites/www/lhcb-nightlies-artifacts/release/lhcb-release/$build_id
    
    In case of SSO, or certificate problems:
    ssh -fN  -L12345:localhost:7474 lbariadne
    export SDBURL=http://localhost:12345/db/data/
    lb-deployment-updatesdb [...]
    
    You can ignore the warnings such as:
WARNING: Error encountered while acquiring CERN SSO cookie for Ariadne. Attempting to connect with no SSO..

  • From here on you will be installing the software on AFS and on CVMFS. The two installations can be done in parallel*
  • For Dirac and LHCbDirac you can ONLY do CVMFS installation

AFS Software Installation

Warning, important This step has to be performed from the host lhcb-archive.cern.ch

Install on AFS

  • First, set some environment variables that define the project name and version to be installed. Used in the following commands (so you can simply cut and paste them directly). Works for bash shells, so recommend switching to this if it isn't your default
    MyProject=<ProjectName>
    MyVersion=vXrY
    
    To check the following commands will work for you, now run
    echo ${MyProject^^} ${MyVersion}
    
    which should return the project name, in upper case.
  • prepare the AFS volume
    lb-project-manage-volume -c $MyProject $MyVersion
    
  • install the new project
    lb-deployment-afs-install $MyProject $MyVersion
    
    In case of problems or missing dependencies you can alter the options to lbinstall by setting the OPTS environment variable e.g.:
    OPTS=--no-strict lb-deployment-afs-install $MyProject $MyVersion
    
  • Check it's all there
    ls -l /afs/cern.ch/lhcb/software/releases/${MyProject^^}/${MyProject^^}_${MyVersion}/
    ls -l /eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2018/ | grep ${MyProject^^} | grep ${MyVersion}
    
    In case of problems see here

All cases:

  • release and lock the AFS volume
    lb-project-manage-volume -r $MyProject $MyVersion
    lb-project-manage-volume -l $MyProject $MyVersion
    
  • prepare the web pages links (NOT for DIRAC and LHCbDirac)
    $LHCBDOC/scripts/addrel.py $MyProject $MyVersion
    

  • FOR OLD STACKS NOT RELEASED VIA RPM: For Gauss, it is necessary to create the Genser tarball
    mkMCGentarFromRPM ${MyProject} ${MyVersion} ${CMTCONFIG}
    
    but make sure that ${CMTCONFIG} is a platform valid for Gauss. Else use another one.

CVMFS Software Installation

  • install the software on CVMFS if not already done in parallel. See here for instructions

Release of Data packages

Warning, important WARNING The procedure for the release of data packages via RPMs is still under heavy development and the automation is not yet in place.

Specifics

Data packages are different from standard projects:

  • They do not need to be compiled for several platforms (make is run nonetheless to perform basic actions)
  • They are part of the DBASE or PARAM projects
  • They are not registered in the software configuration Database and the builds may not always be triggered by the release manager.

The release steps are therefore different from normal projects.

Procedure

Trigger by hand the lhcb-release Jenkins job (if needed)

  • Once the build is completed, check the checkout log
    • go to https://lhcb-nightlies.cern.ch/release/
    • click on the small Jenkins icon in the "Project" column of the build summary look for the package build log (after the line "...:INFO... building Hat/Package")

publish the RPMs

  • If the build was fine, publish the RPMs (from lhcb-archive.cern.ch)
    • Since default SW is now CVMFS, switch to AFS
      ssh lhcb-archive.cern.ch
      bash
      /afs/cern.ch/lhcb/software/releases/LBSCRIPTS/dev/InstallArea/scripts/LbLogin.sh
      kinit Your_Username@CERN.CH
      
    • try to copy the RPMs to the repository
      build_id=1234
      lb-release-rpm --rpm-dir=/eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2018 \
            /eos/project/l/lhcbwebsites/www/lhcb-nightlies-artifacts/release/lhcb-release/$build_id/rpms
      
    • if everything is fine, do the real copy and update the repository info
      lb-release-rpm --rpm-dir=/eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2018 \
           --copy /eos/project/l/lhcbwebsites/www/lhcb-nightlies-artifacts/release/lhcb-release/$build_id/rpms
      
    • check it's all there
      ls -ltr /eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2018/ | tail -10

Install the new RPM on AFS

(must be done from lhcb-archive)

/afs/cern.ch/lhcb/software/lbinstall/afslbinstall install DBASE_Hat_Package_version

Gen/DecFiles usually come in pairs (two versions for one JIRA task): one for dev and one for legacy Gauss versions. Michal will give instructions in this case (always follow his instructions, which take precedence over what is written below).
/afs/cern.ch/lhcb/software/lbinstall/afslbinstall install DBASE_Gen_DecFiles
will install the latest version. To get the legacy version do
/afs/cern.ch/lhcb/software/lbinstall/afslbinstall install DBASE_Gen_DecFiles_vXXrYY
where XX, YY are given in the JIRA task.

Then one needs an extra step to update the bookkeeping. It requires to have a valid Grid certificate

  1. Get your Grid proxy:
              lb-run LHCbDirac lhcb-proxy-init
         
  2. Update the bookkeeping:
             cd /afs/cern.ch/lhcb/software/releases/DBASE/Gen/DecFiles/<version>/cmt  
             lb-run LHCbDirac dirac-bookkeeping-eventtype-mgt-insert ../doc/table_event.sql
             lb-run LHCbDirac dirac-bookkeeping-eventtype-mgt-update ../doc/table_event.sql
             lb-run LHCbDirac dirac-bookkeeping-eventtype-mgt-update ../doc/table_obsolete.sql
         

The dirac-bookkeeping-eventtype-mgt-insert command will produce many errors like

{'EVTTYPEID': '12165431', 'DESCRIPTION': 'Bu_D02pi+pi-,K+pi-pi0=PHSP,DecProdCut', 'PRIMARY': '[B+ -> pi+ pi+ pi- (D0bar -> K+ pi- pi0)]cc'} : Excution failed.: ( ORA-00001: unique constraint (LHCB_DIRACBOOKKEEPING.SYS_C00302542) violated
ORA-06512: at "LHCB_DIRACBOOKKEEPING.BOOKKEEPINGORACLEDB", line 1131
ORA-06512: at line 1
 ) 
This is normal as most event types already exist in the Oracle table. Things are OK if the last line is like
The following event types are inserted: ['42102201', '42102200', '42163203', '42102221', '42102220', '15104451', '42100210', '42100211', '42100230', '42100231', '42142201', '42142200', '42102210', '42102211', '42162203', '42162202', '15574100', '42100221', '42100220', '12163436', '12163437', '42102601', '42102600', '12163236', '42163202', '12163235', '42112200', '42112201'] 

Note that in the commands below, the name <Package> does not contain the hat (i.e. <Package>=SQLDDDB and not Det/SQLDDDB)

  • Check all is fine
              ls -l /afs/cern.ch/lhcb/software/releases/DBASE/<Hat>/<Package>/<version>
  • Lock the AFS directory. After the build of the package release, the directory holding it can be locked. This will remove the write permissions to the directory and this can be achieved with:
              lockAFSDir -R /afs/cern.ch/lhcb/software/releases/DBASE/<Hat>/<Package>/<version>
  • Next? If this is not a special case, the next thing to do is probably to deploy on CVMFS, instructions here.

Install the new RPM on CVMFS

Deployment on CVMFS of a data package is similar to the installation of a project;

CVMFS Software Installation

Connection to "cvlhcb" account on host "cvmfs-lhcb"

All installation of software is done on host "cvmfs-lhcb", under the account called "cvlhcb".

So first connect to "cvmfs-lhcb":

ssh yourusername@cvmfs-lhcb

Then check that nobody else is connected to install software:

$ who -q ; pgrep -fl -u cvlhcb

If someone else is connected and if there are processes running as cvlhcb, perhaps try sending them an email to see what they are doing before proceeding. If nobody else is connected, sudo into the all powerful account...

$ sudo -i -u cvlhcb

You need to start a transaction on the server to be able to write to the disk.

$ cvmfs_transaction

If the command returns that a transaction was already created, somebody else is installing at the same time. Check with the who command before going any further.

N.B.1 cvmfs_transaction is a wrapper around the CVMFS command that creates the transaction on the repository lhcb.cern.ch, and logs the command and result. It is best to use it instead of using the direct command cvmfs_server transaction lhcb.cern.ch

N.B.2 Even if you decide to use cvmfs_server instead of the wrapper, it is best to specify the repository in the command as this server could potentially host several repositories in the future.

Install using standard script

Data packages and Projects are installed on CVMFS roughly the same way (except for the substitution <Project> ==> <Package> for Data packages).

For stacks >= LHCb v35r4 (with Gaudi v23r6), the debug version will be installed on CVMFS. The software should be installed the scripts from the ${HOME}/bin directory.

At this stage you may need to determine which stack a software release is based on. In this case in a different window on any lxplus machine run:

lb-sdb-query --readonly d <Project> <version> 

Install a project from recent stacks, for the default platforms

If the stack is the most recent, and only default platforms are required

install_software.sh <Project> <version> 

In case of problems or missing dependencies you can alter the options to lbinstall by setting the OPTS environment variable e.g.:

OPTS=--no-strict install_software.sh $MyProject $MyVersion

Install a Data package

You need to know which project the data package is part of (DBASE or PARAM) and the optional subdirectory (called "hat") in which it was copied (e.g. WG). Then you can run:

cvmfslbinstall install PROJECT_Hat_Package_version

e.g.

cvmfslbinstall install DBASE_WG_CharmConfig_v3r51

or

cvmfslbinstall install DBASE_PRConfig_v1r19

Check and Update

Check: Sometimes what seems to be a warning is actually a fatal error. You can then check it all went fine with

ls -ltr /cvmfs/lhcb.cern.ch/lib/lhcb/<PROJECT>/<PROJECT>_<version>/InstallArea/
The platforms you have installed should be there, the default platforms are currently only slc6-gcc48, opt and dbg. (the directory to check is different for a Data package, but should be easy to guess.)

Update: Once you have done all the installs, you have to update the global catalog (needs to be done just once at the end, to publish everything that you have installed)

cvmfs_publish

N.B.1 cvmfs_publish is a wrapper around the CVMFS command that publishes the transaction on the repository lhcb.cern.ch, and logs the command and result. It is best to use it instead of using the direct command cvmfs_server publish lhcb.cern.ch

N.B.2 Even if you decide to use cvmfs_server instead of the wrapper, it is best to specify the repository in the command as this server could potentially host several repositories

Exit: When done, do not forget to exit the sudo session so that others can proceed with other installations if necessary

Note: the various install_software and cvmfs_publish commands can be chained by separating them with ;

CVMFS Appendix

IT CVMFS Information

All information from IT Department can be found at https://twiki.cern.ch/twiki/bin/view/CvmFS/Installers

LHCB Software on cvmfs-lhcb

Scripts

All updates to software deployed on cvmfs should be done under account "cvlhcb". "${HOME}/bin" contains the necessary scripts to add/remove software:

  • cvmfs_transaction: To start a transaction to deploy software on the server

  • install_software.sh: To call install_project to add new projects/versions

  • remove_software.sh: To Remove software versions

  • cvmfs_publish: Synchronize the CVMFS file system

Some other scripts are also available (same name as above but prefixed with std_ perform the same action without logging)

The output of those command is automatically logged to "${HOME}/logs/install.log

Sometimes a package is not known to the install_software.sh script. In that case

cvmfslbinstall install DBASE_HAT_PKG_vXrY
may help.

CRON

The local CRON is running on this host to automatically update SQLDDDB every hour. The logs for this cron is in: ${HOME}/logs/update_sqldddb.log

A backup of the CRONTAB is available in ${HOME}/conf.crontab.backup

Special Instructions and troubleshooting

Removing Software (archival)

Projects

  • On lxplus or one of the lxbuild machines (AFS and CVMFS access, as well as the afs_admin command are needed)
    • Set up the AFS environment with the command
      afsLbLoginDev
    • Run the command:
      archive_project <Project> <version>
      This removes the software from the AFS release area and from the old Grid distribution area, and archives the tar files to castor. You need a grid proxy, since the first thing the script does is to contact the book-keeping to check that there isn't an active production step using the software to be archived. You have to hit several times to confirm that you want to remove the tar balls (don't say yes if there was some problem copying to castor)

  • On cvmfs-lhcb, start a transaction in the usual way, then run the command:
    remove_software.sh <Project> <version>
    . Unfortunately this is currently (28th Nov 2017) broken, see https://its.cern.ch/jira/browse/LBCORE-1488. Remember to run cvmfs_publish at the end, as usual, to close the transaction.

Data Packages

  • unlock the package directory
    unlockAFSDir -R /afs/cern.ch/lhcb/software/releases/DBASE/<Hat>/<Package>/<version>
  • remove the package
    /afs/cern.ch/lhcb/software/lbinstall/lbinstall remove <package_name>
  • got to the CVMFS publishing machine (cvmfs-lhcb) and uninstall
    cvmfs_transaction
    remove_software.sh <Package> <version>
    cvmfs_publish
    

Troubleshooting (For Releasers)

If there is a quota problem

When copying tars. First check what's wrong
fs lq $LHCBTAR/*
Then increase quota with
afs_admin sq $LHCBTAR/<PROJECT> <quota>

In a project:

fs lq /afs/cern.ch/lhcb/software/releases/<PROJECT>/*
Then increase quota with
afs_admin sq /afs/cern.ch/lhcb/software/releases/<PROJECT>/<PROJECT>_<version>/*

In rpms... on 2/12/2016 Ben did (see JIRA):

for r in $LHCBTAR/rpm/lhcb/<PROJECT>_<version>_*rpm; do 
          for p in `rpm -qp --requires $r | grep LCG`; 
               do ./afslbinstall --disable-yum-check --just-db install  $p; 
           done; 
    done

In case of problems with incomplete installs

Sometimes the automatic installation may fail, for example because not all platforms were correctly built in the release build, or because some of the needed external dependencies are missing. Depending on the reason for the incomplete install, you may want to ignore the missing dependencies, and install anyway, or just install those platforms for which the dependencies are OK.

  • First find the available packages:
/afs/cern.ch/lhcb/software/lbinstall/afslbinstall  query "${MyProject^^}_${MyVersion}"

  • There are two useful options of afslbinstall to help with partial deployment of the packages returned by the above query:
/afs/cern.ch/lhcb/software/lbinstall/afslbinstall install --nodeps <package_name>
installs just the package but none of its dependencies. Be careful with this because you have to deploy by hand, one package at a time, the whole dependency stack for the deployment to be useful

/afs/cern.ch/lhcb/software/lbinstall/afslbinstall install --no-strict <package_name>
installs the complete set of dependencies, but skipping any missing dependencies

For completeness, afslbinstall called without options installs the complete set of dependencies but stops with an error if a dependency is missing. It is what is called internally by lb-deployment-afs-install

/afs/cern.ch/lhcb/software/lbinstall/afslbinstall install <package_name>

  • Hint: all the above can be condensed in a single command to install all platforms ignoring missing dependencies:
for r in $(/afs/cern.ch/lhcb/software/lbinstall/afslbinstall query "${MyProject^^}_${MyVersion}"); do
/afs/cern.ch/lhcb/software/lbinstall/afslbinstall install --no-strict $r
done

  • Or, if you want to exclude a platform (e.g. because it has missing dependencies):
for r in $(/afs/cern.ch/lhcb/software/lbinstall/afslbinstall query "${MyProject^^}_${MyVersion}" | grep -v ThePlatformToExclude); do
/afs/cern.ch/lhcb/software/lbinstall/afslbinstall install $r
done

  • Don't forget to do the same on cvmfs, where you replace /afs/cern.ch/lhcb/software/lbinstall/afslbinstall with cvmfslbinstall

Troubleshooting (For Librarians)

If the nightly builds do not work

The nightly build system is nothing more than wrappers around simple tools that can be called by any user, so it is possible to prepare the RPMs for the release by hand.

  • Go to an Centos7 machine with AFS (e.g. lxplus or a build machine), to a temporary working directory (e.g. /build/$USER)

  • Prepare the configuration file to drive the build
    lbn-gen-release-config MyProject vXrY > lhcb-release.json
    
    or
    lbn-gen-release-config --cmt  --pack 'Hat/Package vXrY' > lhcb-release.json
    
    for data packages

  • Checkout the sources and prepare the shared RPM
    lbn-checkout --verbose lhcb-release.json
    lbn-rpm --shared --builddir tmp/checkout --verbose lhcb-release.json
    

  • For each required platform build, test and prepare the RPM (not needed for data packages)
    export CMTCONFIG=...
    lbn-build --verbose --clean --with-tests -j 4 lhcb-release.json
    lbn-rpm --verbose lhcb-release.json
    

  • Check if the builds/tests are ok (not needed for data packages)
    • start the Python mini web server
      cd artifacts && python -m SimpleHTTPServer
      
    • with a web browser connect to the port 8000 of the build machine used (e.g. http://lxbuild00.cern.ch:8000/) and navigate to the
      • build log: summaries.<CMTCONFIG>/<Project>/build_log.html
      • test reports: summaries.<CMTCONFIG>/<Project>/html

  • Publish the RPMs (from lhcb-archive.cern.ch)
    cp -v -n artifacts/MYPROJECT*.rpm $LHCBTAR/rpm/lhcb
    
    then, from lhcb-archive.cern.ch
    createrepo --workers=20 --update $LHCBTAR/rpm/lhcb
    

  • If lhcb-archive.cern.ch works, continue with the procedure to install to AFS, otherwise you should install on AFS from the generated tar files, then continue with the normal procedure

If lb-release-rpm prints messages like "error: not an rpm package"

It might mean that the RPM file is probably corrupted.

This could happen during the copy, in which case it is enough to remove the corrupted files from $LHCBTAR/rpm/lhcb and call again lb-release-rpm.

If it still does not work, check that there are no permanent problems in $LHCBTAR/rpm/lhcb (e.q. quota) and that the files are copied correctly with something like (bash)

for f in /eos/project/l/lhcbwebsites/www/lhcb-nightlies-artifacts/release/lhcb-release/${build_id}/*.rpm ; do
  cmp $f $LHCBTAR/rpm/lhcb/$(basename $f)
done

If everything looks correct, try to rebuild the project via the rebuild button in the dashboard.

If everything has gone so wrong that you you want to sit in a corner and cry... or restart from scratch

THIS SHOULD NOT BE DONE, but I document it in any case.

To undo a release, so that it can be replayed from the beginning you should (not that it is a somehow reverse order to what done in the normal procedure):

  • uninstall from CVMFS... you shouldn't have gone that far... TODO
  • revert the software database (should not be needed) TODO
  • remove the old-style tarballs and html files
    rm -i $LHCBTAR/html/${MyProject^^}_${MyProject^^}_${MyVersion}*
    rm -i $LHCBTAR/${MyProject^^}/${MyProject^^}_${MyProject^^}_${MyVersion}*
    
  • remove web page linksTODO
  • uninstall the RPMs
    • check that you know what you want to remove
      /afs/cern.ch/lhcb/software/lbinstall/lbinstall list ${MyProject^^}_${MyVersion}
      
    • actually remove them
      /afs/cern.ch/lhcb/software/lbinstall/lbinstal remove $(/afs/cern.ch/lhcb/software/rpmrel/afslbpkr list ${MyProject^^}_${MyVersion})
      
  • remove the AFS volume in the release area
    afs_admin delete /afs/cern.ch/lhcb/software/releases/${MyProject^^}/${MyProject^^}_${MyVersion}
    
  • remove the RPMs from the repository
    rm -i $LHCBTAR/rpm/lhcb/${MyProject^^}_${MyVersion}*.rpm
    createrepo --update --workers=20 $LHCBTAR/rpm/lhcb
    

For the Librarian

Releasing a new version of LCG externals

Until Gaudi v23r2

Until Gaudi v23r2, the dependencies to be included in LCGCMT were included in the Gaudi requirements files. The procedure to build the tarball was therefore the following:

For Gaudi, one more setup needs to be done: the generation of the tarball of its dependencies. This has to be done for each and every optimized >binary< (x86-64-slc6-gcc46-opt, x86-64-slc5-gcc46-opt, x86-64-slc5-gcc43-opt, i686-slc5-gcc43-opt)

          cd $Gaudi_release_area
          mkLCGCMTtar -n GAUDI_<version> -b <binary>
This script produces a log file in the local directory from where it has been run. Check it to see if everything was OK.

As from Gaudi v23r3...

The generation of the tarball of its dependencies. This has to be done for each and every optimized and debug >binary< (x86-64-slc6-gcc48-opt, x86-64-slc6-gcc48-dbg, x86-64-slc6-gcc49-opt, x86-64-slc6-gcc49-dbg).

The project LHCbExternals contains the list of dependencies to be included in LCGCMT. It is therefore necessary to:

  1. Make sure that the dependency list is up to date
  2. Tag a new version of LHCbExternals, with a version matching LCGCMT e.g. LCGCMT 63 -> LHCbExternals v63r0 LCGCMT 63a -> LHCbExternals v63r1 etc...

Release LHCbExternals to AFS:

          cd /afs/cern.ch/lhcb/software/releases/
          mkproject -p LHCbExternals -v vXrY -a ngc 
          mkproject -p LHCbExternals -v vXrY -a K

Now prepare the tar ball of LCGCMT, which has to be done differently depending on the version of LCGCMT:

* Up to LCGCMT 66:

          cd /afs/cern.ch/lhcb/software/releases/
          mkLCGCMTtar -n LHCBEXTERNALS_<version> -b <binary>

This script produces a log file in the local directory from where it has been run. Check it to see if everything was OK.

* As from LCGCMT 68:

          mkLCGCMTtarFromRPM LHCbExternals v69r0p1 $CMTCONFIG

As from LCG 86

We do not extract dependencies any more, the LHCBEXTERNALS project is abandoned.

To Generate the install project tar files LCGCMT for x86_64-slc6-gcc49-opt, just run (LbScript v8r7 at least is needed)

makeTarFromRPM LCGCMT v86r0 x86_64-slc6-gcc49-opt ./LHCBEXTERNALS_86.json 

Where LHCBEXTERNALS_86.json is a file of the form:

{
    "heptools": {
    "version": 86,
    "packages":[
        "AIDA",
        "Boost",
        "eigen",
   "CLHEP",
   "COOL",
   "CORAL",
   "CppUnit",
   "Frontier_Client",
   "GSL",
   "HepMC",
   "HepPDT",
   "Python",
   "QMtest",
   "Qt",
   "RELAX",
   "ROOT",
   "XercesC",
   "fastjet",
   "fftw",
   "graphviz",
   "libunwind",
   "neurobayes_expert",
   "oracle",
   "pyanalysis",
   "pygraphics",
   "pytools",
   "sqlite",
   "tcmalloc",
   "vdt",
   "xqilla",
   "xrootd",
   "tbb",
   "rangev3",
   "cppgsl",
   "ipython"]

    }
}

As from LCG 87

We now just need to pre-install the build dependencies so that the nightly builds can continue working. The RPM needed at install time are pulled in as needed.

The metadata is contained in files such as:

LHCBEXTERNALS_88.json:

{
    "heptools": {
    "version": 88,
    "packages":[
        "AIDA",
        "Boost",
        "eigen",
   "CLHEP",
   "COOL",
   "CORAL",
   "CppUnit",
   "Frontier_Client",
   "GSL",
   "HepMC",
   "HepPDT",
   "Python",
   "QMtest",
   "Qt",
   "RELAX",
   "ROOT",
   "XercesC",
   "fastjet",
   "fftw",
   "graphviz",
   "libunwind",
   "neurobayes_expert",
   "oracle",
   "pyanalysis",
   "pygraphics",
   "pytools",
   "sqlite",
   "tcmalloc",
   "vdt",
   "xqilla",
   "xrootd",
   "tbb",
   "rangev3",
   "cppgsl",
   "ipython"]

    }
}

The metadata files are kept and versioned in https://gitlab.cern.ch/lhcb-core/rpm-recipes (LHCBEXTERNALS sub directory).

Follow the instructions in the README.md to get the list of rpms to install, and to install them on CVMFS and AFS.

Then update the software configuration DB, e.g.

lb-sdb-addplatform LCG 91 x86_64-slc6-gcc62-opt
lb-sdb-addplatform LCG 91 x86_64-slc6-gcc62-dbg
lb-sdb-addplatform LCG 91 x86_64-centos7-gcc62-dbg
lb-sdb-addplatform LCG 91 x86_64-centos7-gcc62-opt
lb-sdb-addplatform LCG 91 x86_64-centos7-gcc7-opt
lb-sdb-addplatform LCG 91 x86_64-centos7-gcc7-dbg

Updating the Software Configuration DB for a new LCG or Gaudi

First import Gaudi without asking for a release:

lb-sdb-import --norelease Gaudi v28r3

If we are dealing with a new LCG, add the corresponding platforms:

lb-sdb-addplatform LCG 89 x86_64-slc6-gcc62-opt
lb-sdb-addplatform LCG 89 x86_64-slc6-gcc62-dbg
lb-sdb-addplatform LCG 89 x86_64-centos7-gcc62-dbg
lb-sdb-addplatform LCG 89 x86_64-centos7-gcc62-opt
lb-sdb-addplatform LCG 89 x86_64-centos7-gcc7-opt
lb-sdb-addplatform LCG 89 x86_64-centos7-gcc7-dbg

If Gaudi features more platforms than LCG, then specify the list of platforms to release:

lb-sdb-addplatform --release Gaudi v28r3 x86_64-slc6-gcc62-opt
[...]

Then schedule Gaudi for release:

lb-sdb-release Gaudi v28r3

Then check if everything is ok:

$ lb-sdb-query show Gaudi v28r3
Node 382601 Properties
------------------------------
project   : GAUDI
version   : v28r3

Node 382601 relationships
------------------------------
2683451:REQUIRES(O)     -> (ID:382602, project:LCG, version:89)
2683462:REQUESTED_PLATFORM(O) -> (ID:382306, platform:x86_64-centos7-gcc62-dbg)
2683463:REQUESTED_PLATFORM(O) -> (ID:382305, platform:x86_64-centos7-gcc62-opt)
2683464:REQUESTED_PLATFORM(O) -> (ID:382607, platform:x86_64-centos7-gcc7-dbg)
2683466:REQUESTED_PLATFORM(O) -> (ID:382608, platform:x86_64-centos7-gcc7-opt)
2683468:REQUESTED_PLATFORM(O) -> (ID:382303, platform:x86_64-slc6-gcc62-dbg)
2683469:REQUESTED_PLATFORM(O) -> (ID:382302, platform:x86_64-slc6-gcc62-opt)
2683470:REQUESTED_PLATFORM(O) -> (ID:382307, platform:x86_64-centos7-gcc62-do0)
2683471:REQUESTED_PLATFORM(O) -> (ID:382609, platform:x86_64-centos7-gcc7-do0)
2683473:REQUESTED_PLATFORM(O) -> (ID:382304, platform:x86_64-slc6-gcc62-do0)
2683449:PROJECT(I)      <- (ID:122379, project:GAUDI, sourceuri:gitlab-cern:gaudi/Gaudi)
2683474:RELEASEREQ(I)   <- (ID:122366, project:NONE, type:RELEASE, version:NONE)

Releasing a new version of LCG Grid

After LHCbDirac v8r0 (before then mkLCGCMTTar was necessary)

mkLCGCMTtarFromRPM  LHCbDirac vXrY $CMTCONFIG --nonatives

If some RPMs are missing from the middleware, it is possible to create them and to add them to the LHCb externals repository in $LHCBTAR/rpm/lcg The code necessary is in the LCGRPM repo. It can be used to create the spec.

git clone https://gitlab.cern.ch/lhcb-core/LCGRPM.git
cd LCGRPM/package 
./createExternalRPMSpec.py  -s Grid -o ext.spec voms 2.0.12-3 x86_64-slc6-gcc48-opt
rpmbuild -bb ext.spec

After that step, the file should be copied to the RPM repo and the RPM repo should be reindexed:

cp /tmp/rpmbuild/RPMS/noarch/voms_2.0.12_3_x86_64_slc6_gcc48_opt-1.0.0-1.noarch.rpm $LHCBTAR/rpm/lcg
 createrepo --workers=20 --update $LHCBTAR/rpm/lcg

N.B. /tmp/rpmbuild is a default of createExternalRPMSpec, and the -b option can be used to put the files to another directory e.g.

./createExternalRPMSpec.py -b /tmp/lben/toto -s Grid -o ext.spec voms 2.0.12-3 x86_64-slc6-gcc48-opt && rpmbuild -bb ext.spec

Creates the RPM: /tmp/lben/toto/rpmbuild/RPMS/noarch/voms_2.0.12_3_x86_64_slc6_gcc48_opt-1.0.0-1.noarch.rpm

Preparing the meta RPM with the software to be installed locally on the Online farm

First get a local version of LbNightlyTools:

mkdir -p /build/$USER
cd /build/$USER
git clone http://git.cern.ch/pub/LbNightlyTools
cd LbNightlyTools
. setup.sh

Now prepare the Meta RPM spec. There are two ways:

  • By specifying explicitly the packages to be included:
    lbn-generate-metaspec -o ofm.spec OnlineFarmMeta 1.1.1 MOOREONLINE_v23r6_x86_64_slc6_gcc48_opt [...]
    
(You need to pass all necessary applications RPMson the command lines, dependencies are dealt with automatically) And build it:
rpmbuild -bb ofm.spec

  • Using the ONLINE tag in the software configuration DB:

First add the project/version/combination to the soft configuration DB and check that the update worked:

lb-sdb-tag -p x86_64-slc6-gcc48-opt MooreOnline v23r7p15 ONLINE
lb-sdb-query listTag ONLINE

Removal can be done with

lb-sdb-tag -r -p x86_64-slc6-gcc48-opt MooreOnline v23r7pq5 ONLINE
lb-sdb-query listTag ONLINE

Then produce the RPM :

. lb-sdb-env.sh
lbn-generate-metaspec -o ofm.spec OnlineFarmMeta 1.1.1 -t ONLINE
(You need to pass all necessary applications RPMson the command lines, dependencies are dealt with automatically) Build the RPM:
rpmbuild -bb ofm.spec

Now copying it to the RPM repository:

lb-release-rpm --rpm-dir=/eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2018 /tmp/rpmbuild/RPMS/noarch/
lb-release-rpm --rpm-dir=/eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2018 --copy /tmp/rpmbuild/RPMS/noarch/

Now install on plus:

cd /group/hltsw
./onlinelbpkr install OnlineFarmMeta

Releases of the nightly builds on /cvmfs/lhcbdev.cern.ch

The releases should be completely automated, but here are the details needed to pcheck the system, perform manual installations/add slots etc...

To connect:

ssh cvmfs-lhcbdev.cern.ch 
sudo -i -u cvlhcbdev

Then you can start a transaction and copy files to /cvmfs/lhcbdev.cern.ch

Preparing the RPM and releasing a new version of CMake

LHCb uses the Linux_x86_64 binaries from Kitware:

https://cmake.org/download/

The tools to repqckage the RPM can be found in:

https://gitlab.cern.ch/lhcb-core/rpm-recipes

In the cmake folder.

The RPM prepared just needs to be copied to the yum repository and the metadata rebuilt.

Installing new generators

The simulation team tests generators on /cvmfs/lhcbdev.cern.ch. Once validated they need to be installed on the production cvmfs.

The list will be expressed as a YAML file such as:

packages:   
- name: 'lhapdf'     
  LCG: [ '88']     
  versions: [ '6.1.6.cxxstd' ]
- name: 'pythia8'
  LCG: [ '88' ]
  versions: [ '230' ]   
- name: 'rivet'
  LCG: [ '88' ]
  versions: [ '2.5.2' ]   
- name: 'thepeg'
  LCG: [ '88' ]
  versions: [ '2.0.3' ]   

To perform the installation, it is first necessary to derive the list of RPM packages to install. This can be done in the following manner:

virtualenv yamlToPackages
. ./yamlToPackages/bin/activate
pip install --extra-index-url https://lbmultipython03.cern.ch/simple/ --trusted-host lbmultipython03.cern.ch lbinstall lbdevmanager

lbGetPackagesFromYAML <YAML FILE>

This will print out a list of packages. It is then necessary to login to cvmfs-lhcb and install them using the cvmfslbinstall command, e.g.

cvmfs_transaction
cvmfslbinstall install <packages>
cvmfs_publish

-- MarcoClemencic - 30 Jun 2014

-- MarcoCattaneo - 3 Jun 2017

Edit | Attach | Watch | Print version | History: r273 < r272 < r271 < r270 < r269 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r273 - 2018-01-22 - MarcoClemencic
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback