(r28) LbScriptsRelease < LHCb

LHCb Web>LHCbComputing>ProjectRelease>LbScriptsRelease (revision 28) (raw view)~~EditAttachPDF~~
---+ Preparing the release

---++ Updating/Tagging the sources

The GIT repository used for the sources is:

ssh://git@gitlab.cern.ch:7999/lhcb-core/LbScripts.git

To work on the clone, fork it in Gitlab and create Merge requests with new features.
Once done, just tag the code as for any GIT project and push the tag to the main repo.

To generate the release notes:
[... sync you branch with origin master here]
cd LbScriptsSys/doc
./getreleasenotes.sh > release.notes
git add release.notes
git commit -m "Added release notes"
git push origin master

---++ Building

   * Trigger by hand the release Jenkins job:
      * go to https://jenkins-lhcb-nightlies.web.cern.ch/job/nightly-builds/job/release/build
      * add LbScripts <version> in the field =projects_list= 
      * click on the =Build= button (leave all the others fields empty)

   * Once the build is completed, check the checkout log
      * go to https://lhcb-nightlies.cern.ch/release/ and check the status of the *LbScripts* build requested
      * if there are problems, ask the project manager to fix them, and restart the build


---++ Releasing the files

The files stored on the build machine need to be copied to the RPM repo.
   * Login to lhcb-archive
   * Identify the build_id from the nightlies release page
   * try to copy the RPMs to the repository <verbatim>
build_id=1234
lb-release-rpm --rpm-dir=/eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2017 /data/archive/artifacts/release/lhcb-release/$build_id
</verbatim>
      * if everything is fine, do the real copy and update the repository info <verbatim>
build_id=XXX
lb-release-rpm --copy --rpm-dir=/eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2017 /data/archive/artifacts/release/lhcb-release/$build_id
</verbatim>

BE CAREFUL: Since 20141010, the release procedure automatically bumps up the release number for packages for which there is already a version in the RPM repo.
If your slot contains more projects that the ones requested for release, you need to *ONLY copy the needed ones* to the RPM repo.
In that case the *--rpm-regex* option of lb-release-rpm should be used.

Copy the tar files
 * try to copy the RPMs to the repository <verbatim>
build_id=1234
lb-release-oldtar /data/archive/artifacts/release/lhcb-release/$build_id/olddist
</verbatim>
      * if everything is fine, do the real copy and update the repository info <verbatim>
build_id=XXX
lb-release-oldtar --copy /data/archive/artifacts/release/lhcb-release/$build_id/olddist
</verbatim>

---++ Install on AFS

This has to be done from an SLC6 node.

   * install on AFS 
      * First, set some environment variables that define the project name and version to be installed.
        Used in the following commands (so you can simply cut and paste them directly). Works for bash shells, so recommend switching to this if it isn't your default.<verbatim>
MyProject=<ProjectName>
MyVersion=vXrY
</verbatim> To check the following commands will work for you, now run <verbatim>
echo ${MyProject^^}
</verbatim> which should return the project name, in upper case.
      * prepare the AFS volume <verbatim>
lb-project-manage-volume -c $MyProject $MyVersion
</verbatim>
      * install the new project <verbatim>
 /afs/cern.ch/lhcb/software/lbinstall/afslbinstall install ${MyProject^^}_${MyVersion}
</verbatim>

or <verbatim>
/afs/cern.ch/lhcb/software/rpmrel/rpmextractor.py -v ${LHCBTAR}/rpm/lhcb/${MyProject^^}_${MyVersion}[-_]*
</verbatim>

      * Rebuild the project to get the proper hardoded paths in LbLogin.sh and such files. <verbatim>
cd /afs/cern.ch/lhcb/software/releases/LBSCRIPTS/LBSCRIPTS_${MyVersion}/LbScriptsSys/cmt
cmt br cmt make
</verbatim>


   * %X% Don't forget to edit/fix the environment cache files

Two cache files have to be edited, for both *sh* and *csh*.

This means:
  * Removing the COMPILER_PATH env variable
  * Adjusting the PATH and LD_LIBRARY_PATH

A quickcheck of the consistency of the two files can be done using:

<verbatim>
EnvCompare.py `envcache v7r5 sh` `envcache v7r6 sh`
EnvCompare.py `envcache v7r5 csh` `envcache v7r6 csh`
</verbatim>


   * release and lock the AFS volume <verbatim>
lb-project-manage-volume -r $MyProject $MyVersion
lb-project-manage-volume -l $MyProject $MyVersion
</verbatim>
   * prepare the web pages links <verbatim>
$LHCBDOC/scripts/addrel.py $MyProject $MyVersion
</verbatim>


---+ Creating the links RPMs

You need get a copy of the NightlyTools repository:
<verbatim>
mkdir -p $TMPDIR/build
cd  $TMPDIR/build
git clone https://gitlab.cern.ch/lhcb-core/LbNightlyTools.git
cd LbNightlyTools
. setup.sh
</verbatim>

Then you build the RPMs like so:

To create the meta RPM to put that version as the prod one:
<verbatim>
lbn-generate-lbscriptsspec ${MyVersion} --prod -o lb.spec -b $TMPDIR/toto && rpmbuild -bb lb.spec
</verbatim>


To create the meta RPM to put that version as the dev one:
<verbatim>
lbn-generate-lbscriptsspec ${MyVersion} --dev -o lb.spec -b $TMPDIR/toto && rpmbuild -bb lb.spec
</verbatim>

Then copy the files to the RPM repository:
<verbatim>
lb-release-rpm --rpm-dir=/eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2017 $TMPDIR/toto/rpmbuild/RPMS/noarch/
lb-release-rpm --copy --rpm-dir=/eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2017 $TMPDIR/toto/rpmbuild/RPMS/noarch/
</verbatim>

And install the RPM in the RPM database (just the metadata) e.g.
<verbatim>

$LHCBHOME/software/lbinstall/afslbinstall update LBSCRIPTSDEV
</verbatim>


---+ Setting the new release as the "dev" version

Once the release of LbScripts vXrYpZ has been deployed to the the release area, it is necessary to change the link $LHCBRELEASES/LBSCRIPTS/dev: 

<verbatim>
cd $LHCBRELEASES/LBSCRIPTS 
ln -nsf LBSCRIPTS_vXrYpZ dev
</verbatim>

---+ Switching the production release to a new version

A "prod" symbolic link exists on AFS that points to the production version of LbScripts.

%X% When putting a release in production, the link has to be updated by doing:

<verbatim>
cd $LHCBRELEASES/LBSCRIPTS 
ln -nsf LBSCRIPTS_vXrYpZ prod
</verbatim>

The links should be updated on CVMFS and on local install by install project itself.

Upon going to production, you need to create the prod meta RPM and install the packages with the -justdb option in lppkr e.g.
<verbatim>
cd /afs/cern.ch/lhcb/software/rpmrel
./afslbpkr rpm -- -ivh --justdb tmp/LBSCRIPTS-X.Y.Z-1.rpm
</verbatim>




















==============================================================================================================

---+ For LBSCRIPTS in SVN - Valid until LbScripts v8r4x - Obsolete afterwards

==============================================================================================================


---+++ Release tools
LbScripts can be released with the same tools as for all other projects, but some extra steps are needed for install project.

%X% a cached version of the environment is generated, so it is necessary to make sure that you do not have LbScripts in your User_release_area when performing the release. 

---+++ LbConfiguration
The LbConfiguration package generates the LbLogin script whichs contains the name of the released version.

%X% Therefore, LbConfiguration must ALWAYS be released with the same version as LbScripts itself !



---+++ LbLegacy / install_projects specific steps

When releasing LbLegacy, install_project.py must be updated at each new release of LbScripts in order to make sure that the auto-update mechanism of install_project and LbScripts works correctly.
In this scripts the script_version variable must be set to the day of the release with the following pattern: YYMMDD, e.g.:

<verbatim>
script_version = '120206'
</verbatim>

It should increase monotonously, as install_project uses this variable to check wherther a new version is available.

The LbScripts version should be set to the version of LbScripts to be released as this is used to choose the version to e downloaded from the repository e.g

<verbatim>
lbscripts_version = "v6r6p4"
</verbatim>


Once all packages have been tagged individually, call:
<verbatim>
tag_package -P LbScripts version
</verbatim>



---+ Building the release

<verbatim>
cd $LHCBRELEASES
mkproject -p LbScripts -v vXrYpZ --not-parallel
</verbatim>


%X% Don't forget to edit/fix the environment cache files

Two cache files have to be edited, for both *sh* and *csh*.

This means:
  * Removing the COMPILER_PATH env variable
  * Adjusting the PATH and LD_LIBRARY_PATH

A quickcheck of the consistency of the two files can be done using:

<verbatim>
EnvCompare.py `envcache v7r5 sh` `envcache v7r6 sh`
EnvCompare.py `envcache v7r5 csh` `envcache v7r6 csh`

</verbatim>


The release also needs to be locked:
<verbatim>
mkproject -p LbScripts -v vXrYpZ -a K 
</verbatim>


---+ Creating the RPMs

You need get a copy of the NightlyTools repository:
<verbatim>
mkdir -p $TMPDIR/build
cd  $TMPDIR/build
git clone https://gitlab.cern.ch/lhcb-core/LbNightlyTools.git
cd LbNightlyTools
. setup.sh
</verbatim>

Then you build the RPM like so:

<verbatim>
lbn-generate-lbscriptsspec v8r1p1  -o lb.spec -b $TMPDIR/toto && rpmbuild -bb lb.spec
</verbatim>

To create the meta RPM to put that version as the prod one:
<verbatim>
lbn-generate-lbscriptsspec v8r1p1 --prod -o lb.spec -b $TMPDIR/toto && rpmbuild -bb lb.spec
</verbatim>


To create the meta RPM to put that version as the dev one:
<verbatim>
lbn-generate-lbscriptsspec v8r1p1 --dev -o lb.spec -b $TMPDIR/toto && rpmbuild -bb lb.spec
</verbatim>

Then copy the files to the RPM repository:
<verbatim>
lb-release-rpm --copy --rpm-dir=/eos/project/l/lhcbwebsites/www/lhcb-rpm/lhcb2017 $TMPDIR/toto/rpmbuild/RPMS/noarch/
</verbatim>

And install the RPM in the RPM database (just the metadata) e.g.
<verbatim>
$LHCBHOME/software/rpmrel/afslbpkr rpm -- -ivh --justdb $LHCBTAR/rpm/lhcb/LBSCRIPTS_v8r4-1.0.0-1.noarch.rpm 
$LHCBHOME/software/rpmrel/afslbpkr rpm -- -Uvh --justdb $LHCBTAR/rpm/lhcb/LBSCRIPTSDEV-8.4.0-1.noarch.rpm

</verbatim>


---+ Setting the new release as the "dev" version

Once the release of LbScripts vXrYpZ has been deployed to the the release area, it is necessary to change the link $LHCBRELEASES/LBSCRIPTS/dev: 

<verbatim>
cd $LHCBRELEASES/LBSCRIPTS 
rm dev && ln -s LBSCRIPTS_vXrYpZ dev
</verbatim>

or 
<verbatim>
cd $LHCBRELEASES/LBSCRIPTS 
ln -nsf LBSCRIPTS_vXrYpZ dev
</verbatim>

---+ Switching the production release to a new version

A "prod" symbolic link exists on AFS that points to the production version of LbScripts.

%X% When putting a release in production, the link has to be updated by doing:

<verbatim>
cd $LHCBRELEASES/LBSCRIPTS 
rm prod && ln -s LBSCRIPTS_vXrYpZ prod
</verbatim>

The links should be updated on CVMFS and on local install by install project itself.

Upon going to production, you need to create the prod meta RPM and install the packages with the -justdb option in lppkr e.g.
<verbatim>
cd /afs/cern.ch/lhcb/software/rpmrel
./afslbpkr rpm -- -ivh --justdb tmp/LBSCRIPTS-8.3.1-1.rpm
./afslbpkr rpm -- -Uvh --justdb tmp/LBSCRIPTS-8.3.1-1.rpm
</verbatim>





---+ Test uses cases

This section provides some (incomplete) information about the scenarios that should be tested when releasing a new version of LbScripts.

Main types of installations:
* Single repository: Used for normal installations on a mahine without AFS or CVMFS
* Chained repositories: Used for grid jobs which use software from a shared area, but also install software in their local disk space

Main use case:
* Install some new software
* Add versions/new projects to an existing area
* Check whether a project is installed
* Check whether the SQLDDDB data package is installed (this is a special case)

---+++ Prerequisites

Make sure that the account used for testing does not have the LHCb environment already set up !
The variables CMTSITE and CMTPROJECTPATH especially !
For accounts linked to group z5, touch the file ".nogrouplogin" to avoid invoking the group scripts automatically !


A version of LbScripts is needed. For the production version:
<verbatim>
wget http://lhcbproject.web.cern.ch/lhcbproject/dist/install_project.py
</verbatim>
For the dev version:
<verbatim>
wget http://lhcbproject.web.cern.ch/lhcbproject/dist/devel/install_project.py
</verbatim>

If a version of install_project is already there, it is also possible to use the --dev-install option to switch to the "dev" version of install project.

MYSITEROOT and CMTCONFIG must be defined, e.g.
<verbatim>
export MYSITEROOT=/opt/install/lib
export CMTCONFIG=x86_64-slc5-gcc43-opt
</verbatim>

N.B. In order to test LbScripts before the release of a new version, it is necessary to invoke LbLogin with the following options:
<verbatim>
LbLogin --user-area-scripts --scripts-version=""
</verbatim>


---+++ Single area

* Install a version of Gaudi and check logs and presence of files

<verbatim>
python install_project.py -b Gaudi v23r4
</verbatim>

* Use install project to check that the software is installed

<verbatim>
python install_project.py -b --check Gaudi v23r14
</verbatim>

* Check group permissions on repositories (XXX check what they should be)

* Check that --check on SQLDDDB is always false
<verbatim>
python install_project.py --check SQLDDDB
</verbatim>

* install Brunel
<verbatim>
python install_project.py -b Brunel v44r0
</verbatim>

* Setup the Brunel environment and run the SAM test
<verbatim>
. ./LbLogin.sh
SetupProject Brunel 
cd $BRUNELSYSROOT/cmt
cmt TestPackage sam
</verbatim>

* Try SetupProject with --use
<verbatim>
python install_project.py AppConfig v3r147
SetupProject --debug --use="AppConfig v3r147"  Brunel v44r0
</verbatim>

---+++ Chained area

In this configuration, the MYSITEROOT contains several repositories  (2 normally) separated by a colon like a PATH.
<verbatim>
export MYSITEROOT=/opt/install/joblocal:/opt/install/shared
</verbatim>
In this configuration, for the tests to be more realistics, it is necessary to make sure that the test cannot modify files in the shared area (e.g. chaining with cvmfs with give that effect). e.g. using two different accounts on lxplus (with /tmp/lben/siteroot containing the previous installation).

<verbatim>
setenv MYSITEROOT '/tmp/benlhcb/LocalArea:/tmp/lben/siteroot'
cd /tmp/benlhcb/LocalArea
python ./install_project.py AppConfig v3r146
source ./LbLogin.csh
</verbatim>

Check that AppConfig v3r146 is in /tmp/benlhcb/LocalArea/lhcb/EXTRAPACKAGES/
<verbatim>
SetupProject --debug --use="AppConfig v3r146"  Brunel v44r0
</verbatim>

Run Brunel with a few events
<verbatim>
gaudirun.py $APPCONFIGOPTS/Brunel/DataType-2012.py $BRUNELROOT/options/COLLISION12-Beam4000GeV-VeloClosed-MagDown.py --option="from Configurables import Brunel; Brunel().EvtMax=10"

</verbatim>


The same tests as for a single area should be re-run.

InstallProjectWithRPM

-- Main.BenjaminCouturier - 13-Feb-2012
Topic revision: r28 - 2017-10-18 - BenjaminCouturier
LHCb
Public webs
Welcome Guest
- Cern Search
- TWiki Search
- Google Search
LHCb All webs