Deployment of Data Packages

Data packages are different from standard projects:

• They do not need to be compiled for several platforms (make is run nonetheless to perform basic actions)
• They are part of the DBASE or PARAM projects
• They are not registered in the software configuration Database and the builds may not always be triggered by the release manager.

The release steps are therefore different from normal projects.

Trigger by hand the lhcb-release Jenkins job (if needed)

• go to https://jenkins-lhcb-nightlies.web.cern.ch/job/nightly-builds/job/release/build
• add the list of packages (with their versions) in the field packages_list (e.g. RawEventFormat v1r1 Gen/DecFiles vXrY), the name of the container can be specified with something like "container":"name" (e.g. PARAM:TMVAWeights v1r0). The container can be omitted for DBASE.
• click on the Build button (leave all the others fields empty)
• watch it being built at https://jenkins-lhcb-nightlies.web.cern.ch/job/nightly-builds/job/checkout/

• Once the build is completed, check the checkout log
  • go to https://lhcb-nightlies.cern.ch/release/
  • click on the project name (DBASE or PARAM) in the build summary, and look for the package build log (after the line "building data packages in...")

Update of the RPM repository

The goal of this step is to make sure that:

1. The RPMs have been copied to the YUM repository (currently in EOS)
2. The YUM metadata has been updated (the lb-release-rpm internally uses the "createrepo" yum command).

Actions:

• If you didn't trigger the build, check that everything is OK (see trigger data package build section)
• If there are problems, ask the project manager to fix them, and restart the build
• Update the RPMs repository ( *to be done from any host *)
  • connect to lxplus.cern.ch

    ssh lxplus7.cern.ch
    bash
    kinit ${USER}@CERN.CH
    

  • publish the produced RPMs (all of them, unless explicitly requested otherwise, see partial install section)
    • Define the build ID to extract RPMS from

      build_id=1234
      

    • Try to copy the RPMs to the repository

      lb-release-rpm --rpm-dir=/eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2019 /eos/project/l/lhcbwebsites/www/lhcb-nightlies-artifacts/release/lhcb-release/$build_id/rpms
      

      • For old stacks only released as tar files:

        lb-release-oldtar /eos/project/l/lhcbwebsites/www/lhcb-nightlies-artifacts/release/lhcb-release/$build_id/olddist
        

    • If everything is fine, do the real copy and update the repository info

      lb-release-rpm --rpm-dir=/eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2019 --copy /eos/project/l/lhcbwebsites/www/lhcb-nightlies-artifacts/release/lhcb-release/$build_id/rpms
      

      • For old stacks only released as tar files:

        lb-release-oldtar --copy /eos/project/l/lhcbwebsites/www/lhcb-nightlies-artifacts/release/lhcb-release/$build_id/olddist
        

  • Check it's all there

    ls -ltr /eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2019/ | tail -10
    

    In case of problems see here

BE CAREFUL: Since 20141010, the release procedure automatically bumps up the release number for packages for which there is already a version in the RPM repo. If your slot contains more projects that the ones requested for release, you need to ONLY copy the needed ones to the RPM repo. In that case the --rpm-regex option of lb-release-rpm should be used.

N.B. If you want to release all rpms EXCEPT some matching a specific string (e.g. gcc62) you can use:

lb-release-rpm -c --rpm-regex '^(?!.*(gcc62)).*'  --rpm-dir=/eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2019 /eos/project/l/lhcbwebsites/www/lhcb-nightlies-artifacts/release/lhcb-release/$build_id/rpms

Software Installation (CVMFS)

Actions:

• Connect to "cvlhcb" account on host "cvmfs-lhcb"

  ssh yourusername@cvmfs-lhcb
  

• Check that nobody else is connected to install software

  who -q ; pgrep -fl -u cvlhcb
  

  if someone else is connected and if there are processes running as cvlhcb, perhaps try sending them an email to see what they are doing before proceeding
• sudo into the all powerful account

  sudo -i -u cvlhcb
  

• Start a transaction on the server to be able to write to the disk

  cvmfs_transaction
  

  • if the command returns that a transaction was already created, somebody else is installing at the same time. Check with the who command before going any further.
  • Note:
    1. cvmfs_transaction is a wrapper around the CVMFS command that creates the transaction on the repository lhcb.cern.ch, and logs the command and result. It is best to use it instead of using the direct command cvmfs_server transaction lhcb.cern.ch
    2. even if you decide to use cvmfs_server instead of the wrapper, it is best to specify the repository in the command as this server could potentially host several repositories in the future.

• Install the data package

  cvmfslbinstall install <PROJECT>_<Hat>_<Package>_<version>
  

• Check that it all went fine with

  ls -ltr /cvmfs/lhcb.cern.ch/lib/lhcb/<PROJECT>/<Hat>/<Package>/<version>/
  

• Update the CVMFS catalogs

  cvmfs_publish
  

  • Note:
    1. cvmfs_publish is a wrapper around the CVMFS command that publishes the transaction on the repository lhcb.cern.ch, and logs the command and result. It is best to use it instead of using the direct command cvmfs_server publish lhcb.cern.ch
    2. Even if you decide to use cvmfs_server instead of the wrapper, it is best to specify the repository in the command as this server could potentially host several repositories
• Remember to log out

Deployment of LbEnv

Install new versions

LbEnv is the set of LHCb environment scripts and tools used for development. It will be released to CVMFS by a CRON running on cvmfs-lhcb.cern.ch, but in the meantime some tools have been prepared for this task (namely update_lbenv.sh).

First check in DRYRUN mode whether the tool is going to update correctly:

CVMFS-Shared ~ > DRYRUN=1 update_lbenv.sh
2019-03-07T16:39:30+0100 Will install:
lbenv-kit.stable.x86_64-centos7.347-4a87636109.tar.bz2
lbenv-kit.stable.x86_64-slc6.347-4a87636109.tar.bz2
lbenv-kit.unstable.x86_64-centos7.347-68420333c0.tar.bz2
lbenv-kit.unstable.x86_64-slc6.347-68420333c0.tar.bz2
2019-03-07T16:39:30+0100 Unpacking lbenv-kit.stable.x86_64-centos7.347-4a87636109.tar.bz2 ...
2019-03-07T16:39:30+0100 DRYRUN mode. Would add link /cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/testing/x86_64-centos7 -> ../347/stable/x86_64-centos7
2019-03-07T16:39:30+0100 Unpacking lbenv-kit.stable.x86_64-slc6.347-4a87636109.tar.bz2 ...
2019-03-07T16:39:30+0100 DRYRUN mode. Would add link /cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/testing/x86_64-slc6 -> ../347/stable/x86_64-slc6
2019-03-07T16:39:30+0100 Unpacking lbenv-kit.unstable.x86_64-centos7.347-68420333c0.tar.bz2 ...
2019-03-07T16:39:30+0100 DRYRUN mode. Would add link /cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/unstable/x86_64-centos7 -> ../347/unstable/x86_64-centos7
2019-03-07T16:39:30+0100 Unpacking lbenv-kit.unstable.x86_64-slc6.347-68420333c0.tar.bz2 ...
2019-03-07T16:39:30+0100 DRYRUN mode. Would add link /cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/unstable/x86_64-slc6 -> ../347/unstable/x86_64-slc6

If this seems satisfactory:

CVMFS-Shared ~ > update_lbenv.sh
Starting TRANSACTION
Created TRANSACTION
2019-03-07T16:39:37+0100 Will install:
lbenv-kit.stable.x86_64-centos7.347-4a87636109.tar.bz2
lbenv-kit.stable.x86_64-slc6.347-4a87636109.tar.bz2
lbenv-kit.unstable.x86_64-centos7.347-68420333c0.tar.bz2
lbenv-kit.unstable.x86_64-slc6.347-68420333c0.tar.bz2
2019-03-07T16:39:37+0100 Unpacking lbenv-kit.stable.x86_64-centos7.347-4a87636109.tar.bz2 ...
2019-03-07T16:39:37+0100 Downloading https://lhcb-rpm.web.cern.ch/lhcb-rpm/lbenv-kits/lhcb.cern.ch/lbenv-kit.stable.x86_64-centos7.347-4a87636109.tar.bz2
tar: Removing leading `/' from member names
2019-03-07T16:39:43+0100 Adding link /cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/testing/x86_64-centos7 -> ../347/stable/x86_64-centos7
`/cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/testing/x86_64-centos7' -> `../347/stable/x86_64-centos7'
2019-03-07T16:39:43+0100 Unpacking lbenv-kit.stable.x86_64-slc6.347-4a87636109.tar.bz2 ...
2019-03-07T16:39:43+0100 Downloading https://lhcb-rpm.web.cern.ch/lhcb-rpm/lbenv-kits/lhcb.cern.ch/lbenv-kit.stable.x86_64-slc6.347-4a87636109.tar.bz2
tar: Removing leading `/' from member names
2019-03-07T16:39:47+0100 Adding link /cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/testing/x86_64-slc6 -> ../347/stable/x86_64-slc6
`/cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/testing/x86_64-slc6' -> `../347/stable/x86_64-slc6'
2019-03-07T16:39:47+0100 Unpacking lbenv-kit.unstable.x86_64-centos7.347-68420333c0.tar.bz2 ...
2019-03-07T16:39:47+0100 Downloading https://lhcb-rpm.web.cern.ch/lhcb-rpm/lbenv-kits/lhcb.cern.ch/lbenv-kit.unstable.x86_64-centos7.347-68420333c0.tar.bz2
tar: Removing leading `/' from member names
2019-03-07T16:39:51+0100 Adding link /cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/unstable/x86_64-centos7 -> ../347/unstable/x86_64-centos7
`/cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/unstable/x86_64-centos7' -> `../347/unstable/x86_64-centos7'
2019-03-07T16:39:51+0100 Unpacking lbenv-kit.unstable.x86_64-slc6.347-68420333c0.tar.bz2 ...
2019-03-07T16:39:51+0100 Downloading https://lhcb-rpm.web.cern.ch/lhcb-rpm/lbenv-kits/lhcb.cern.ch/lbenv-kit.unstable.x86_64-slc6.347-68420333c0.tar.bz2
tar: Removing leading `/' from member names
2019-03-07T16:39:57+0100 Adding link /cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/unstable/x86_64-slc6 -> ../347/unstable/x86_64-slc6
`/cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/unstable/x86_64-slc6' -> `../347/unstable/x86_64-slc6'
`.installed_kits.txt.2019-03-07T16:39:37+0100' -> `/cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/.installed_kits.txt.2019-03-07T16:39:37+0100'
removed `.installed_kits.txt.2019-03-07T16:39:37+0100'
`/cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/.installed_kits.txt' -> `.installed_kits.txt.2019-03-07T16:39:37+0100'
Starting PUBLISH =====================================
gio  7 mar 2019, 16.39.57, CET
Using auto tag 'generic-2019-03-07T15:39:57Z'
Processing changes...
...........

Update stable version

When a new testing version of LbEnv is ready for production, the release shifter is asked to update the prod version, in which case he/she has to run from cvmfs-lhcb.cern.ch:

update_lbenv_stable.sh

WARNING: in case of failure the transaction is kept open for the shifter to fix the problem or abort it.

Special Instructions and troubleshooting

Installation on AFS

Note that only librarians will be able to change the content of AFS, so contact <lhcb-software-librarians@cern.ch>.

Deployment from Old Tarfiles

Removing Software (archival)

Projects

• Archive the project documentation (step to be defined, for the time being try python $LHCBDOC/scripts/gc_archive.py <project> <version>, noting that <project> must be in all lowercase). This must be done before removing the software!

• On cvmfs-lhcb, start a transaction in the usual way, then list the RPMs you want to remove using the cvmfslbinstall command:

cvmfslbinstall list URANIA_v6r2p1
URANIA_v6r2p1                            1.0.0   1    local
URANIA_v6r2p1_index                      1.0.0   1    local
URANIA_v6r2p1_x86_64_centos7_gcc62_dbg   1.0.0   1    local
URANIA_v6r2p1_x86_64_centos7_gcc62_do0   1.0.0   1    local
URANIA_v6r2p1_x86_64_centos7_gcc62_opt   1.0.0   1    local
URANIA_v6r2p1_x86_64_slc6_gcc49_dbg      1.0.0   1    local
URANIA_v6r2p1_x86_64_slc6_gcc49_do0      1.0.0   1    local
URANIA_v6r2p1_x86_64_slc6_gcc49_opt      1.0.0   1    local
URANIA_v6r2p1_x86_64_slc6_gcc62_dbg      1.0.0   1    local
URANIA_v6r2p1_x86_64_slc6_gcc62_do0      1.0.0   1    local
URANIA_v6r2p1_x86_64_slc6_gcc62_opt      1.0.0   1    local

Then you can remove the packages using cvmfslbinstall remove, passing it the list of files to be removed (it will order them correctly and check dependencies). Please note that lbinstall does NOT remove directories if they contain files that do not belong to the RPM packages (e.g. generated files). Therefore it is necessary to check the project top directory if still existing and check what is left and remove manually if necessary.

N.B. after removing the packages, the files created by hand that do not belong to any package are NOT removed. You may need to check the directory for those.

N.B. Be careful about the packages returned, cvmfslbinstall list URANIA_v6r2 returns the packages for v6r2 and v6r2p1...

Remember to run cvmfs_publish at the end, as usual, to close the transaction.

Data Packages

• got to the CVMFS publishing machine (cvmfs-lhcb) and uninstall

  cvmfs_transaction
  cvmfslbinstall remove <PROJECT>_<Hat>_<Package>_<version><Package> <version>
  cvmfs_publish
  

Troubleshooting (For Releasers)

If there is a quota problem

fs lq $LHCBTAR/*

afs_admin sq $LHCBTAR/<PROJECT> <quota>

In a project:

fs lq /afs/cern.ch/lhcb/software/releases/<PROJECT>/*

afs_admin sq /afs/cern.ch/lhcb/software/releases/<PROJECT>/<PROJECT>_<version>/*

In rpms... on 2/12/2016 Ben did (see JIRA):

for r in $LHCBTAR/rpm/lhcb/<PROJECT>_<version>_*rpm; do 
          for p in `rpm -qp --requires $r | grep LCG`; 
               do ./afslbinstall --disable-yum-check --just-db install  $p; 
           done; 
    done

In case of problems with incomplete installs

Sometimes the automatic installation may fail, for example because not all platforms were correctly built in the release build, or because some of the needed external dependencies are missing. Depending on the reason for the incomplete install, you may want to ignore the missing dependencies, and install anyway, or just install those platforms for which the dependencies are OK.

• First find the available packages:

/afs/cern.ch/lhcb/software/lbinstall/afslbinstall  query "${MyProject^^}_${MyVersion}"

• There are two useful options of afslbinstall to help with partial deployment of the packages returned by the above query:

/afs/cern.ch/lhcb/software/lbinstall/afslbinstall install --nodeps <package_name>

/afs/cern.ch/lhcb/software/lbinstall/afslbinstall install --no-strict <package_name>

For completeness, afslbinstall called without options installs the complete set of dependencies but stops with an error if a dependency is missing. It is what is called internally by lb-deployment-afs-install

/afs/cern.ch/lhcb/software/lbinstall/afslbinstall install <package_name>

• Hint: all the above can be condensed in a single command to install all platforms ignoring missing dependencies:

for r in $(/afs/cern.ch/lhcb/software/lbinstall/afslbinstall query "${MyProject^^}_${MyVersion}"); do
/afs/cern.ch/lhcb/software/lbinstall/afslbinstall install --no-strict $r
done

• Or, if you want to exclude a platform (e.g. because it has missing dependencies):

for r in $(/afs/cern.ch/lhcb/software/lbinstall/afslbinstall query "${MyProject^^}_${MyVersion}" | grep -v ThePlatformToExclude); do
/afs/cern.ch/lhcb/software/lbinstall/afslbinstall install $r
done

• Don't forget to do the same on cvmfs, where you replace /afs/cern.ch/lhcb/software/lbinstall/afslbinstall with cvmfslbinstall

Troubleshooting (For Librarians)

If the nightly builds do not work

• Go to an Centos7 machine with AFS (e.g. lxplus or a build machine), to a temporary working directory (e.g. /build/$USER)

• Prepare the configuration file to drive the build

  lbn-gen-release-config MyProject vXrY > lhcb-release.json
  

  or

  lbn-gen-release-config --cmt  --pack 'Hat/Package vXrY' > lhcb-release.json
  

  for data packages

• Checkout the sources and prepare the shared RPM

  lbn-checkout --verbose lhcb-release.json
  lbn-rpm --shared --builddir tmp/checkout --verbose lhcb-release.json
  

• For each required platform build, test and prepare the RPM (not needed for data packages)

  export CMTCONFIG=...
  lbn-build --verbose --clean --with-tests -j 4 lhcb-release.json
  lbn-rpm --verbose lhcb-release.json
  

• Check if the builds/tests are ok (not needed for data packages)
  • start the Python mini web server

    cd artifacts && python -m SimpleHTTPServer
    

  • with a web browser connect to the port 8000 of the build machine used (e.g. http://lxbuild00.cern.ch:8000/) and navigate to the
    • build log: summaries.<CMTCONFIG>/<Project>/build_log.html
    • test reports: summaries.<CMTCONFIG>/<Project>/html

• Publish the RPMs (from lxplus.cern.ch)

  cp -v -n artifacts/MYPROJECT*.rpm $LHCBTAR/rpm/lhcb
  

  then, from lxplus.cern.ch

  createrepo --workers=20 --update $LHCBTAR/rpm/lhcb
  

• If lxplus.cern.ch works, continue with the procedure to install to AFS, otherwise you should install on AFS from the generated tar files, then continue with the normal procedure

If lb-release-rpm prints messages like "error: not an rpm package"

This could happen during the copy, in which case it is enough to remove the corrupted files from $LHCBTAR/rpm/lhcb and call again lb-release-rpm.

If it still does not work, check that there are no permanent problems in $LHCBTAR/rpm/lhcb (e.q. quota) and that the files are copied correctly with something like (bash)

for f in /eos/project/l/lhcbwebsites/www/lhcb-nightlies-artifacts/release/lhcb-release/${build_id}/*.rpm ; do
  cmp $f $LHCBTAR/rpm/lhcb/$(basename $f)
done

If everything looks correct, try to rebuild the project via the rebuild button in the dashboard.

If everything has gone so wrong that you you want to sit in a corner and cry... or restart from scratch

THIS SHOULD NOT BE DONE, but I document it in any case.

To undo a release, so that it can be replayed from the beginning you should (not that it is a somehow reverse order to what done in the normal procedure):

• uninstall from CVMFS... you shouldn't have gone that far...
• revert the software database (should not be needed)
• remove the old-style tarballs and html files

  rm -i $LHCBTAR/html/${MyProject^^}_${MyProject^^}_${MyVersion}*
  rm -i $LHCBTAR/${MyProject^^}/${MyProject^^}_${MyProject^^}_${MyVersion}*
  

• remove web page links
• uninstall the RPMs
  • check that you know what you want to remove

    /afs/cern.ch/lhcb/software/lbinstall/lbinstall list ${MyProject^^}_${MyVersion}
    

  • actually remove them

    /afs/cern.ch/lhcb/software/lbinstall/lbinstall remove $(/afs/cern.ch/lhcb/software/rpmrel/afslbpkr list ${MyProject^^}_${MyVersion})
    

• remove the AFS volume in the release area

  afs_admin delete /afs/cern.ch/lhcb/software/releases/${MyProject^^}/${MyProject^^}_${MyVersion}
  

• remove the RPMs from the repository

  rm -i $LHCBTAR/rpm/lhcb/${MyProject^^}_${MyVersion}*.rpm
  createrepo --update --workers=20 $LHCBTAR/rpm/lhcb
  

For the Librarian

Releasing a new version of LCG externals

Until Gaudi v23r2

Until Gaudi v23r2, the dependencies to be included in LCGCMT were included in the Gaudi requirements files. The procedure to build the tarball was therefore the following:

For Gaudi, one more setup needs to be done: the generation of the tarball of its dependencies. This has to be done for each and every optimized >binary< (x86-64-slc6-gcc46-opt, x86-64-slc5-gcc46-opt, x86-64-slc5-gcc43-opt, i686-slc5-gcc43-opt)

          cd $Gaudi_release_area
          mkLCGCMTtar -n GAUDI_<version> -b <binary>

As from Gaudi v23r3...

The generation of the tarball of its dependencies. This has to be done for each and every optimized and debug >binary< (x86-64-slc6-gcc48-opt, x86-64-slc6-gcc48-dbg, x86-64-slc6-gcc49-opt, x86-64-slc6-gcc49-dbg).

The project LHCbExternals contains the list of dependencies to be included in LCGCMT. It is therefore necessary to:

1. Make sure that the dependency list is up to date
2. Tag a new version of LHCbExternals, with a version matching LCGCMT e.g. LCGCMT 63 -> LHCbExternals v63r0 LCGCMT 63a -> LHCbExternals v63r1 etc...

Release LHCbExternals to AFS:

          cd /afs/cern.ch/lhcb/software/releases/
          mkproject -p LHCbExternals -v vXrY -a ngc 
          mkproject -p LHCbExternals -v vXrY -a K

Now prepare the tar ball of LCGCMT, which has to be done differently depending on the version of LCGCMT:

* Up to LCGCMT 66:

          cd /afs/cern.ch/lhcb/software/releases/
          mkLCGCMTtar -n LHCBEXTERNALS_<version> -b <binary>

This script produces a log file in the local directory from where it has been run. Check it to see if everything was OK.

* As from LCGCMT 68:

          mkLCGCMTtarFromRPM LHCbExternals v69r0p1 $CMTCONFIG

As from LCG 86

We do not extract dependencies any more, the LHCBEXTERNALS project is abandoned.

To Generate the install project tar files LCGCMT for x86_64-slc6-gcc49-opt, just run (LbScript v8r7 at least is needed)

makeTarFromRPM LCGCMT v86r0 x86_64-slc6-gcc49-opt ./LHCBEXTERNALS_86.json 

Where LHCBEXTERNALS_86.json is a file of the form:

{
    "heptools": {
    "version": 86,
    "packages":[
        "AIDA",
        "Boost",
        "eigen",
   "CLHEP",
   "COOL",
   "CORAL",
   "CppUnit",
   "Frontier_Client",
   "GSL",
   "HepMC",
   "HepPDT",
   "Python",
   "QMtest",
   "Qt",
   "RELAX",
   "ROOT",
   "XercesC",
   "fastjet",
   "fftw",
   "graphviz",
   "libunwind",
   "neurobayes_expert",
   "oracle",
   "pyanalysis",
   "pygraphics",
   "pytools",
   "sqlite",
   "tcmalloc",
   "vdt",
   "xqilla",
   "xrootd",
   "tbb",
   "rangev3",
   "cppgsl",
   "ipython"]

    }
}

As from LCG 87

We now just need to pre-install the build dependencies so that the nightly builds can continue working. The RPM needed at install time are pulled in as needed.

The metadata is contained in files such as:

LHCBEXTERNALS_88.json:

{
    "heptools": {
    "version": 88,
    "packages":[
        "AIDA",
        "Boost",
        "eigen",
   "CLHEP",
   "COOL",
   "CORAL",
   "CppUnit",
   "Frontier_Client",
   "GSL",
   "HepMC",
   "HepPDT",
   "Python",
   "QMtest",
   "Qt",
   "RELAX",
   "ROOT",
   "XercesC",
   "fastjet",
   "fftw",
   "graphviz",
   "libunwind",
   "neurobayes_expert",
   "oracle",
   "pyanalysis",
   "pygraphics",
   "pytools",
   "sqlite",
   "tcmalloc",
   "vdt",
   "xqilla",
   "xrootd",
   "tbb",
   "rangev3",
   "cppgsl",
   "ipython"]

    }
}

The metadata files are kept and versioned in https://gitlab.cern.ch/lhcb-core/rpm-recipes (LHCBEXTERNALS sub directory).

Follow the instructions in the README.md to get the list of rpms to install, and to install them on CVMFS and AFS.

Then update the software configuration DB, e.g.

lb-sdb-addplatform LCG 91 x86_64-slc6-gcc62-opt
lb-sdb-addplatform LCG 91 x86_64-slc6-gcc62-dbg
lb-sdb-addplatform LCG 91 x86_64-centos7-gcc62-dbg
lb-sdb-addplatform LCG 91 x86_64-centos7-gcc62-opt
lb-sdb-addplatform LCG 91 x86_64-centos7-gcc7-opt
lb-sdb-addplatform LCG 91 x86_64-centos7-gcc7-dbg

Updating the Software Configuration DB for a new LCG or Gaudi

First import Gaudi without asking for a release:

lb-sdb-import --norelease Gaudi v28r3

If we are dealing with a new LCG, add the corresponding platforms:

lb-sdb-addplatform LCG 89 x86_64-slc6-gcc62-opt
lb-sdb-addplatform LCG 89 x86_64-slc6-gcc62-dbg
lb-sdb-addplatform LCG 89 x86_64-centos7-gcc62-dbg
lb-sdb-addplatform LCG 89 x86_64-centos7-gcc62-opt
lb-sdb-addplatform LCG 89 x86_64-centos7-gcc7-opt
lb-sdb-addplatform LCG 89 x86_64-centos7-gcc7-dbg

If Gaudi features more platforms than LCG, then specify the list of platforms to release:

lb-sdb-addplatform --release Gaudi v28r3 x86_64-slc6-gcc62-opt
[...]

Then schedule Gaudi for release:

lb-sdb-release Gaudi v28r3

Then check if everything is ok:

$ lb-sdb-query show Gaudi v28r3
Node 382601 Properties
------------------------------
project   : GAUDI
version   : v28r3

Node 382601 relationships
------------------------------
2683451:REQUIRES(O)     -> (ID:382602, project:LCG, version:89)
2683462:REQUESTED_PLATFORM(O) -> (ID:382306, platform:x86_64-centos7-gcc62-dbg)
2683463:REQUESTED_PLATFORM(O) -> (ID:382305, platform:x86_64-centos7-gcc62-opt)
2683464:REQUESTED_PLATFORM(O) -> (ID:382607, platform:x86_64-centos7-gcc7-dbg)
2683466:REQUESTED_PLATFORM(O) -> (ID:382608, platform:x86_64-centos7-gcc7-opt)
2683468:REQUESTED_PLATFORM(O) -> (ID:382303, platform:x86_64-slc6-gcc62-dbg)
2683469:REQUESTED_PLATFORM(O) -> (ID:382302, platform:x86_64-slc6-gcc62-opt)
2683470:REQUESTED_PLATFORM(O) -> (ID:382307, platform:x86_64-centos7-gcc62-do0)
2683471:REQUESTED_PLATFORM(O) -> (ID:382609, platform:x86_64-centos7-gcc7-do0)
2683473:REQUESTED_PLATFORM(O) -> (ID:382304, platform:x86_64-slc6-gcc62-do0)
2683449:PROJECT(I)      <- (ID:122379, project:GAUDI, sourceuri:gitlab-cern:gaudi/Gaudi)
2683474:RELEASEREQ(I)   <- (ID:122366, project:NONE, type:RELEASE, version:NONE)

Releasing a new version of LCG Grid

After LHCbDirac v8r0 (before then mkLCGCMTTar was necessary)

mkLCGCMTtarFromRPM  LHCbDirac vXrY $CMTCONFIG --nonatives

If some RPMs are missing from the middleware, it is possible to create them and to add them to the LHCb externals repository in $LHCBTAR/rpm/lcg The code necessary is in the LCGRPM repo. It can be used to create the spec.

git clone https://gitlab.cern.ch/lhcb-core/LCGRPM.git
cd LCGRPM/package 
./createExternalRPMSpec.py  -s Grid -o ext.spec voms 2.0.12-3 x86_64-slc6-gcc48-opt
rpmbuild -bb ext.spec

After that step, the file should be copied to the RPM repo and the RPM repo should be reindexed:

cp /tmp/rpmbuild/RPMS/noarch/voms_2.0.12_3_x86_64_slc6_gcc48_opt-1.0.0-1.noarch.rpm $LHCBTAR/rpm/lcg
 createrepo --workers=20 --update $LHCBTAR/rpm/lcg

N.B. /tmp/rpmbuild is a default of createExternalRPMSpec, and the -b option can be used to put the files to another directory e.g.

./createExternalRPMSpec.py -b /tmp/lben/toto -s Grid -o ext.spec voms 2.0.12-3 x86_64-slc6-gcc48-opt && rpmbuild -bb ext.spec

Creates the RPM: /tmp/lben/toto/rpmbuild/RPMS/noarch/voms_2.0.12_3_x86_64_slc6_gcc48_opt-1.0.0-1.noarch.rpm

Preparing the meta RPM with the software to be installed locally on the Online farm

First get a local version of LbNightlyTools:

mkdir -p /build/$USER
cd /build/$USER
git clone http://git.cern.ch/pub/LbNightlyTools
cd LbNightlyTools
. setup.sh

Now prepare the Meta RPM spec. There are two ways:

• By specifying explicitly the packages to be included:

  lbn-generate-metaspec -o ofm.spec OnlineFarmMeta 1.1.1 MOOREONLINE_v23r6_x86_64_slc6_gcc48_opt [...]
  

rpmbuild -bb ofm.spec

• Using the ONLINE tag in the software configuration DB:

First add the project/version/combination to the soft configuration DB and check that the update worked:

lb-sdb-tag -p x86_64-slc6-gcc48-opt MooreOnline v23r7p15 ONLINE
lb-sdb-query listTag ONLINE

Removal can be done with

lb-sdb-tag -r -p x86_64-slc6-gcc48-opt MooreOnline v23r7pq5 ONLINE
lb-sdb-query listTag ONLINE

Then produce the RPM :

. lb-sdb-env.sh
lbn-generate-metaspec -o ofm.spec OnlineFarmMeta 1.1.1 -t ONLINE

rpmbuild -bb ofm.spec

Now copying it to the RPM repository:

lb-release-rpm --rpm-dir=/eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2019 /tmp/rpmbuild/RPMS/noarch/
lb-release-rpm --rpm-dir=/eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2019 --copy /tmp/rpmbuild/RPMS/noarch/

Now install on plus:

cd /group/hltsw
./onlinelbpkr install OnlineFarmMeta

Releases of the nightly builds on /cvmfs/lhcbdev.cern.ch

The releases should be completely automated, but here are the details needed to pcheck the system, perform manual installations/add slots etc...

To connect:

ssh cvmfs-lhcbdev.cern.ch 
sudo -i -u cvlhcbdev

Then you can start a transaction and copy files to /cvmfs/lhcbdev.cern.ch

Preparing the RPM and releasing a new version of CMake

LHCb uses the Linux_x86_64 binaries from Kitware:

https://cmake.org/download/

The tools to repqckage the RPM can be found in:

https://gitlab.cern.ch/lhcb-core/rpm-recipes

In the cmake folder.

The RPM prepared just needs to be copied to the yum repository and the metadata rebuilt.

Installing new generators

The simulation team tests generators on /cvmfs/lhcbdev.cern.ch. Once validated they need to be installed on the production cvmfs.

The list will be expressed as a YAML file such as:

packages:   
- name: 'lhapdf'     
  LCG: [ '88']     
  versions: [ '6.1.6.cxxstd' ]
- name: 'pythia8'
  LCG: [ '88' ]
  versions: [ '230' ]   
- name: 'rivet'
  LCG: [ '88' ]
  versions: [ '2.5.2' ]   
- name: 'thepeg'
  LCG: [ '88' ]
  versions: [ '2.0.3' ]   

To perform the installation, it is first necessary to derive the list of RPM packages to install. This can be done in the following manner:

virtualenv yamlToPackages
. ./yamlToPackages/bin/activate
pip install --extra-index-url https://lbmultipython03.cern.ch/simple/ --trusted-host lbmultipython03.cern.ch lbinstall lbdevmanager

lbGetPackagesFromYAML <YAML FILE>

This will print out a list of packages. It is then necessary to login to cvmfs-lhcb and install them using the cvmfslbinstall command, e.g.

cvmfs_transaction
cvmfslbinstall install <packages>
cvmfs_publish

CVMFS Appendix

IT CVMFS Information

All information from IT Department can be found at https://twiki.cern.ch/twiki/bin/view/CvmFS/Installers

LHCB Software on cvmfs-lhcb

Scripts

All updates to software deployed on cvmfs should be done under account "cvlhcb". "${HOME}/bin" contains the necessary scripts to add/remove software:

• cvmfs_transaction: To start a transaction to deploy software on the server

• install_software.sh: To call install_project to add new projects/versions

• remove_software.sh: To Remove software versions

• cvmfs_publish: Synchronize the CVMFS file system

Some other scripts are also available (same name as above but prefixed with std_ perform the same action without logging)

The output of those command is automatically logged to "${HOME}/logs/install.log

Sometimes a package is not known to the install_software.sh script. In that case

cvmfslbinstall install DBASE_HAT_PKG_vXrY

CRON

The local CRON is running on this host to automatically update SQLDDDB every hour. The logs for this cron is in: ${HOME}/logs/update_sqldddb.log

A backup of the CRONTAB is available in ${HOME}/conf.crontab.backup

-- MarcoClemencic - 30 Jun 2014

-- MarcoCattaneo - 3 Jun 2017

-- MarcoClemencic - 2018-01-22