Difference: LHCbNightlies (1 vs. 25)

Revision 252019-10-08 - RosenMatev

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 73 to 73
 getNightlyRefs lhcb-sim09 Mon Gauss x86_64-slc6-gcc48-opt %ENDSYNTAX%
Added:
>
>

Platform specific references

To have different references depending on the platform you need to suffix the filename with the platform, e.g. hlt1_reco_baseline.ref.x86_64+avx2+fma-centos7-gcc8-opt. You can have a single reference match multiple platforms by omitting some of the tokens split by -. For instance, hlt1_reco_baseline.ref.x86_64 would match all but the avx2 platform and hlt1_reco_baseline.ref.x86_64+avx2+fma would match only the avx2 platform. The general logic is that the platform where you run is split into tokens with - as a separator and the same is done with the last part of the ref filename. The filename where the most tokens are a subset of the platform's tokens is the reference used for the validation, see here.
 

Nightly tests input data

When nightly tests require event data as input, they should used files stored in the CERN-SWTEST storage element, and described in the Test Files database. See the TestFileDB TWiki for details

Revision 242017-03-21 - MarcoCattaneo

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 9 to 9
 A technical description of the system can be found in LHCbNightliesImplementation and hints on how to fix some problems in LHCbNightliesTroubleshooting.

Status of nightlies

Changed:
<
<
The current status of the LHCb nightly builds is available at https://lhcb-nightlies.cern.ch. See also the status of the LCG nightlies
>
>
The current status of the LHCb nightly builds is available at http://lhcb-nightlies.cern.ch. See also the status of the LCG nightlies. The older version of the status page can be reached at https://lhcb-nightlies-old.cern.ch/
  RSS feeds reporting about error/warning details for slots, projects or specific platforms can be defined on the summary webpage:
  • use "Custom RSS feed" button in the top of the summary webpage

Revision 232017-03-16 - MarcoCattaneo

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 23 to 23
  For the platforms we use the strings that identify the SupportedPlatforms.
Deleted:
<
<
Note: The format of the configuration files might change in the future (see LBCORE-110), but the old format is described in LHCbNightliesOldConfiguration#Configuration.
 

Regular builds

Changed:
<
<
Every night we start the build of several slots, whose configurations are stored on Gitlab (LHCbNightlyConf).
>
>
Every night we start the build of several slots, whose configurations are stored on Gitlab (LHCbNightlyConf). Some of the slots are still using the old XML format described in LHCbNightliesOldConfiguration#Configuration, these are slowly being migrated to a new python description that was first discussed in LBCORE-110). An editor exists for the XML configuration file, it can be accessed from the "Configuration Editor" button of the Nightly Builds status page
 

Monitoring

%TODO%

Revision 222017-01-24 - MarcoCattaneo

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 59 to 59
 

Nightly tests reference files

The testing infrastructure is described in GaudiTestingInfrastructure wiki. The nightlies execute in all slots all the tests that have been defined for packages being build in that slot. The tests that fail because of a mismatch between the output and the reference file produce a special file with extension '.new'.
Changed:
<
<
%TODO% Path below is no longer valid, to be updated (info in JIRA task is also largely obsolete) These references are saved in the directory
<!-- SyntaxHighlightingPlugin -->
https://lhcb-archive.cern.ch/artifacts/nightly/<slot>/<day>/newrefs.<platform>/...
<!-- end SyntaxHighlightingPlugin -->
See LBCORE-164
>
>
If you want to copy the nightly reference files to commit them as replacements without needing to re-run the tests yourself, you can use getNightlyRefs, a script that adds the .new files to some local checkout. For example, to update the Brunel references with those from today's lhcb-head nightly, for the currently defined CMTCONFIG platform:
<!-- SyntaxHighlightingPlugin -->
lb-dev —nightly lhcb-head Brunel HEAD
cd BrunelDev_HEAD
git lb-use Brunel
git lb-checkout Brunel/master Rec/Brunel
git lb-checkout Brunel/master BrunelSys
getNightlyRefs lhcb-head
<!-- end SyntaxHighlightingPlugin -->
Then replace the existing *.ref files with the uploaded *.ref.new, commit and push to a new branch to make a merge request

You can simply upload all the new references for a given application/nightly slot/platform combination to the local directory, without need for a local checkout:

<!-- SyntaxHighlightingPlugin -->
getNightlyRefs lhcb-sim09 Mon Gauss x86_64-slc6-gcc48-opt
<!-- end SyntaxHighlightingPlugin -->
 
Deleted:
<
<
If you want to copy the nightly reference files to simply upload them as replacements without needing to re-run the tests yourself, take a look at getNightlyRefs, a script that adds the .new files to some local checkout.
 

Nightly tests input data

When nightly tests require event data as input, they should used files stored in the CERN-SWTEST storage element, and described in the Test Files database. See the TestFileDB TWiki for details
Line: 100 to 110
  -- MarcoClemencic - 01-Oct-2013 -- RosenMatev - 2015-10-12
Added:
>
>
-- MarcoCattaneo - 2017-01-24
 \ No newline at end of file

Revision 212017-01-24 - MarcoCattaneo

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 36 to 36
 

Running from the nightlies

The following instructions are for DaVinci, but they apply to any other project; instructions for Gauss are here. The first thing is to decide which slot. Usually one slot builds DaVinci on the latest LHCb, or the LHCb release candidate, and the other uses the head of Gaudi. Which one to pick is up to what you want to do. See https://lhcb-nightlies.cern.ch for the definitions. Then decide on the day. Make sure that the version you picked actually compiles. Now you have a slot, say lhcb-head and a day, say last night.
Deleted:
<
<
%TODO% This changes with CMake
 First do
<!-- SyntaxHighlightingPlugin -->
lb-dev --nightly lhcb-head [ day ] DaVinci HEAD
<!-- end SyntaxHighlightingPlugin -->
Changed:
<
<
where day is optional. The default is Today, or pick up a day like Mon, Tue... This builds you a directory ./DaVinciDev_HEAD/. In there getpack what you need
<!-- SyntaxHighlightingPlugin -->
getpack Phys/DaVinci head
<!-- end SyntaxHighlightingPlugin -->
and the needed packages. Then do
>
>
where day is optional. The default is Today, or pick up a day like Mon, Tue... This builds you a directory ./DaVinciDev_HEAD/. In there check out what you need
<!-- SyntaxHighlightingPlugin -->
cd DavinciDev_HEAD
git lb-use DaVinci
git lb-checkout DaVinci/master Phys/DaVinci 
<!-- end SyntaxHighlightingPlugin -->
and any other needed packages (see Git4LHCb on how to work with LHCb software under Git). Then do
 
<!-- SyntaxHighlightingPlugin -->
./run gaudirun.py <options>
<!-- end SyntaxHighlightingPlugin -->
This will execute gaudirun.py in the environment of your local project. You can also use
<!-- SyntaxHighlightingPlugin -->
./run bash --norc
<!-- end SyntaxHighlightingPlugin -->
Line: 52 to 53
  See also GaudiCMakeConfiguration#Building_with_CMake.
Changed:
<
<
You can also run from ganga (not on the grid, only at CERN) using the setupProjectOptions field to pass the right options to SetupProject.

See also the SetupProject wiki.

>
>
You can also run from ganga %TODO% instructions to be updated, previous instructions for SetupProject no longer valid
 

Nightly tests reference files

Changed:
<
<
The testing infrastructure is described in GaudiTestingInfrastructure wiki. The nightlies execute in all slots all the tests that have been defined for packages being build in that slot. The tests that fail because of a mismatch between the output and the reference file produce a special file with extension '.new'. These references are saved in the directory
>
>
The testing infrastructure is described in GaudiTestingInfrastructure wiki. The nightlies execute in all slots all the tests that have been defined for packages being build in that slot. The tests that fail because of a mismatch between the output and the reference file produce a special file with extension '.new'.

%TODO% Path below is no longer valid, to be updated (info in JIRA task is also largely obsolete) These references are saved in the directory

 
<!-- SyntaxHighlightingPlugin -->
https://lhcb-archive.cern.ch/artifacts/nightly/<slot>/<day>/newrefs.<platform>/...
<!-- end SyntaxHighlightingPlugin -->
See LBCORE-164

Revision 202016-12-11 - MarcoCattaneo

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 19 to 19
  A slot is a consistent set of inter-related software projects that should be built and tested together. The configuration of a slot includes, in addition to the list of configured projects, the list of platforms the slot should be built for and some extra metadata for the fine tuning of the build.
Changed:
<
<
Software projects are defined in the slot configuration with enough detail to be able to check-out the code from the software repositories. The special version HEAD means master plus all non-WIP merge requests targeting master. Any other version maps directly to Git (as in git checkout version). Special configurations can be defined on an ad hoc basis (for example projects in 2016-patches check out 2016-patches plus all non-WIP merge requests targeting 2016-patches branch
>
>
Software projects are defined in the slot configuration with enough detail to be able to check-out the code from the software repositories. The special version HEAD means master plus all non-WIP merge requests targeting master. Any other version maps directly to Git (as in git checkout version). Special configurations can be defined on an ad hoc basis (for example projects in 2016-patches check out 2016-patches plus all non-WIP merge requests targeting 2016-patches branch). It is also possible to target a WIP merge request to a specific nightly slot by giving the slot as a label - see LBCORE-1156
  For the platforms we use the strings that identify the SupportedPlatforms.
Line: 61 to 61
 
<!-- SyntaxHighlightingPlugin -->
https://lhcb-archive.cern.ch/artifacts/nightly/<slot>/<day>/newrefs.<platform>/...
<!-- end SyntaxHighlightingPlugin -->
See LBCORE-164
Changed:
<
<
If you want to copy the nightly reference files to simply upload them as replacements without needing to re-run the tests yourself, take a look at getNightlyRefs a script recently added to Lbscripts which uses rsynch to add the .new files to some local checkout. It has its own help but doesn't work outside of the CERN firewall.
>
>
If you want to copy the nightly reference files to simply upload them as replacements without needing to re-run the tests yourself, take a look at getNightlyRefs, a script that adds the .new files to some local checkout.
 

Nightly tests input data

When nightly tests require event data as input, they should used files stored in the CERN-SWTEST storage element, and described in the Test Files database. See the TestFileDB TWiki for details

Revision 192016-11-23 - MarcoCattaneo

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 19 to 19
  A slot is a consistent set of inter-related software projects that should be built and tested together. The configuration of a slot includes, in addition to the list of configured projects, the list of platforms the slot should be built for and some extra metadata for the fine tuning of the build.
Changed:
<
<
Software projects are defined in the slot configuration with enough detail to be able to check-out the code from the software repositories. We can use an existing tag, the latest version (trunk/HEAD) or fine tune the check-out overriding the versions of the packages constituting the projects.
>
>
Software projects are defined in the slot configuration with enough detail to be able to check-out the code from the software repositories. The special version HEAD means master plus all non-WIP merge requests targeting master. Any other version maps directly to Git (as in git checkout version). Special configurations can be defined on an ad hoc basis (for example projects in 2016-patches check out 2016-patches plus all non-WIP merge requests targeting 2016-patches branch
  For the platforms we use the strings that identify the SupportedPlatforms.

Revision 182016-08-04 - MarcoCattaneo

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 26 to 26
 Note: The format of the configuration files might change in the future (see LBCORE-110), but the old format is described in LHCbNightliesOldConfiguration#Configuration.

Regular builds

Changed:
<
<
Every night we start the build several slots, which configurations are stored on SVN (LHCbNightlyConf).
>
>
Every night we start the build of several slots, whose configurations are stored on Gitlab (LHCbNightlyConf).
 

Monitoring

%TODO%
Added:
>
>
KIbana monitoring of the LHCb build machines can be found here
 

Running from the nightlies

The following instructions are for DaVinci, but they apply to any other project; instructions for Gauss are here. The first thing is to decide which slot. Usually one slot builds DaVinci on the latest LHCb, or the LHCb release candidate, and the other uses the head of Gaudi. Which one to pick is up to what you want to do. See https://lhcb-nightlies.cern.ch for the definitions. Then decide on the day. Make sure that the version you picked actually compiles. Now you have a slot, say lhcb-head and a day, say last night.

Revision 172016-07-19 - MarcoCattaneo

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 84 to 84
 %ENDSYNTAX% If you want to run the test manually (without make), obtain the command from make test ARGS="-R cosmics -V -N" by removing "--report" "ctest".
Added:
>
>
For more options for selecting which tests to run, see the CMake FAQ
 

Documentation

Revision 162016-02-24 - MarcoCattaneo

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 62 to 62
 If you want to copy the nightly reference files to simply upload them as replacements without needing to re-run the tests yourself, take a look at getNightlyRefs a script recently added to Lbscripts which uses rsynch to add the .new files to some local checkout. It has its own help but doesn't work outside of the CERN firewall.

Nightly tests input data

Changed:
<
<
When nightly tests require event data as input, they should used files stored in the CERN-EOS-SWTEST storage element, and described in the Test Files database. See the TestFileDB TWiki for details
>
>
When nightly tests require event data as input, they should used files stored in the CERN-SWTEST storage element, and described in the Test Files database. See the TestFileDB TWiki for details
 

Reproducing nightly tests locally

To re-run the nightly tests locally for a certain project (e.g. Moore), you should do the following steps (in bash).

Revision 152015-11-06 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 56 to 56
 

Nightly tests reference files

The testing infrastructure is described in GaudiTestingInfrastructure wiki. The nightlies execute in all slots all the tests that have been defined for packages being build in that slot. The tests that fail because of a mismatch between the output and the reference file produce a special file with extension '.new'. These references are saved in the directory
Changed:
<
<
<!-- SyntaxHighlightingPlugin -->
https://buildlhcb.cern.ch/artifacts/nightly/<slot>/<day>/newrefs.<platform>/...
<!-- end SyntaxHighlightingPlugin -->
>
>
<!-- SyntaxHighlightingPlugin -->
https://lhcb-archive.cern.ch/artifacts/nightly/<slot>/<day>/newrefs.<platform>/...
<!-- end SyntaxHighlightingPlugin -->
 See LBCORE-164

If you want to copy the nightly reference files to simply upload them as replacements without needing to re-run the tests yourself, take a look at getNightlyRefs a script recently added to Lbscripts which uses rsynch to add the .new files to some local checkout. It has its own help but doesn't work outside of the CERN firewall.

Revision 142015-10-20 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 9 to 9
 A technical description of the system can be found in LHCbNightliesImplementation and hints on how to fix some problems in LHCbNightliesTroubleshooting.

Status of nightlies

Changed:
<
<
The current status of the LHCb nightly builds is available at https://buildlhcb.cern.ch/nightlies/. See also the status of the LCG nightlies
>
>
The current status of the LHCb nightly builds is available at https://lhcb-nightlies.cern.ch. See also the status of the LCG nightlies
 
Changed:
<
<
RSS feeds reporting about error/warning details for slots, projects or specific platforms can be defined on the summary webpage:
>
>
RSS feeds reporting about error/warning details for slots, projects or specific platforms can be defined on the summary webpage:
 
  • use "Custom RSS feed" button in the top of the summary webpage

Configuration

Line: 28 to 28
 

Regular builds

Every night we start the build several slots, which configurations are stored on SVN (LHCbNightlyConf).
Deleted:
<
<
Which slots are built (and when they are started) is defined in the configuration of the Jenkins server.
 

Monitoring

%TODO%

Running from the nightlies

Changed:
<
<
The following instructions are for DaVinci, but they apply to any other project; instructions for Gauss are here. The first thing is to decide which slot. Usually one slot builds DaVinci on the latest LHCb, or the LHCb release candidate, and the other uses the head of Gaudi. Which one to pick is up to what you want to do. See https://buildlhcb.cern.ch/nightlies/ for the definitions. Then decide on the day. Make sure that the version you picked actually compiles. Now you have a slot, say lhcb-head and a day, say last night.
>
>
The following instructions are for DaVinci, but they apply to any other project; instructions for Gauss are here. The first thing is to decide which slot. Usually one slot builds DaVinci on the latest LHCb, or the LHCb release candidate, and the other uses the head of Gaudi. Which one to pick is up to what you want to do. See https://lhcb-nightlies.cern.ch for the definitions. Then decide on the day. Make sure that the version you picked actually compiles. Now you have a slot, say lhcb-head and a day, say last night.
  %TODO% This changes with CMake

First do

Changed:
<
<
<!-- SyntaxHighlightingPlugin -->
SetupProject DaVinci HEAD --build-env --nightly lhcb-head [ day ]
<!-- end SyntaxHighlightingPlugin -->
where day is optional. The default is today, or pick up a day like Mon, Tue... This builds you a directory ~/cmtuser/DaVinci_HEAD/. In there getpack what you need
>
>
<!-- SyntaxHighlightingPlugin -->
lb-dev --nightly lhcb-head [ day ] DaVinci HEAD
<!-- end SyntaxHighlightingPlugin -->
where day is optional. The default is Today, or pick up a day like Mon, Tue... This builds you a directory ./DaVinciDev_HEAD/. In there getpack what you need
 
<!-- SyntaxHighlightingPlugin -->
getpack Phys/DaVinci head
<!-- end SyntaxHighlightingPlugin -->
and the needed packages. Then do
Changed:
<
<
<!-- SyntaxHighlightingPlugin -->
SetupProject DaVinci HEAD --nightly lhcb-head [ day ]
<!-- end SyntaxHighlightingPlugin -->
i.e. the same as above without the --build-env. Now $DAVINCIROOT should point to your getpacked DaVinci and you can run.
>
>
<!-- SyntaxHighlightingPlugin -->
./run gaudirun.py <options>
<!-- end SyntaxHighlightingPlugin -->
This will execute gaudirun.py in the environment of your local project. You can also use
<!-- SyntaxHighlightingPlugin -->
./run bash --norc
<!-- end SyntaxHighlightingPlugin -->
or
<!-- SyntaxHighlightingPlugin -->
./run tcsh -f
<!-- end SyntaxHighlightingPlugin -->
to start a new subshell in the modified environment.

See also GaudiCMakeConfiguration#Building_with_CMake.

  You can also run from ganga (not on the grid, only at CERN) using the setupProjectOptions field to pass the right options to SetupProject.
Line: 52 to 56
 

Nightly tests reference files

The testing infrastructure is described in GaudiTestingInfrastructure wiki. The nightlies execute in all slots all the tests that have been defined for packages being build in that slot. The tests that fail because of a mismatch between the output and the reference file produce a special file with extension '.new'. These references are saved in the directory
Changed:
<
<
<!-- SyntaxHighlightingPlugin -->
https://buildlhcb.cern.ch/artifacts/<slot>/<day>/newrefs.<platform>/...
<!-- end SyntaxHighlightingPlugin -->
>
>
<!-- SyntaxHighlightingPlugin -->
https://buildlhcb.cern.ch/artifacts/nightly/<slot>/<day>/newrefs.<platform>/...
<!-- end SyntaxHighlightingPlugin -->
 See LBCORE-164

If you want to copy the nightly reference files to simply upload them as replacements without needing to re-run the tests yourself, take a look at getNightlyRefs a script recently added to Lbscripts which uses rsynch to add the .new files to some local checkout. It has its own help but doesn't work outside of the CERN firewall.

Revision 132015-10-12 - RosenMatev

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 60 to 60
 

Nightly tests input data

When nightly tests require event data as input, they should used files stored in the CERN-EOS-SWTEST storage element, and described in the Test Files database. See the TestFileDB TWiki for details
Added:
>
>

Reproducing nightly tests locally

To re-run the nightly tests locally for a certain project (e.g. Moore), you should do the following steps (in bash). First, copy the full project from the nightlies:
<!-- SyntaxHighlightingPlugin -->
cp -r $LHCBNIGHTLIES/lhcb-head/Today/MOORE/MOORE_HEAD .
cd MOORE_HEAD
<!-- end SyntaxHighlightingPlugin -->
Then, setup the environment and configure your local copy:
<!-- SyntaxHighlightingPlugin -->
. /afs/cern.ch/lhcb/software/nightlies/lhcb-head/Today/setupSearchPath.sh
export USE_CMAKE=1
make configure
<!-- end SyntaxHighlightingPlugin -->

You can now run a single test with

<!-- SyntaxHighlightingPlugin -->
make test ARGS="-R cosmics -V"
<!-- end SyntaxHighlightingPlugin -->
If you want to run the test manually (without make), obtain the command from make test ARGS="-R cosmics -V -N" by removing "--report" "ctest".
 

Documentation

Line: 68 to 88
 

-- MarcoClemencic - 01-Oct-2013 \ No newline at end of file

Added:
>
>
-- RosenMatev - 2015-10-12
 \ No newline at end of file

Revision 122015-07-06 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 6 to 6
 

The System

The LHCb Nightly Build System is a collection of tools and scripts that allow automation of build and test tasks for LHCb software.
Changed:
<
<
A technical description of the system can be found in LHCbNightliesImplementation.
>
>
A technical description of the system can be found in LHCbNightliesImplementation and hints on how to fix some problems in LHCbNightliesTroubleshooting.
 

Status of nightlies

The current status of the LHCb nightly builds is available at https://buildlhcb.cern.ch/nightlies/. See also the status of the LCG nightlies

Revision 102015-07-01 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 6 to 6
 

The System

The LHCb Nightly Build System is a collection of tools and scripts that allow automation of build and test tasks for LHCb software.
Added:
>
>
A technical description of the system can be found in LHCbNightliesImplementation.
 

Status of nightlies

The current status of the LHCb nightly builds is available at https://buildlhcb.cern.ch/nightlies/. See also the status of the LCG nightlies

Revision 92015-01-28 - PaulSeyfert

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 34 to 34
 

Running from the nightlies

The following instructions are for DaVinci, but they apply to any other project; instructions for Gauss are here. The first thing is to decide which slot. Usually one slot builds DaVinci on the latest LHCb, or the LHCb release candidate, and the other uses the head of Gaudi. Which one to pick is up to what you want to do. See https://buildlhcb.cern.ch/nightlies/ for the definitions. Then decide on the day. Make sure that the version you picked actually compiles. Now you have a slot, say lhcb-head and a day, say last night.
Added:
>
>
%TODO% This changes with CMake
 First do
<!-- SyntaxHighlightingPlugin -->
SetupProject DaVinci HEAD --build-env --nightly lhcb-head [ day ]
<!-- end SyntaxHighlightingPlugin -->
where day is optional. The default is today, or pick up a day like Mon, Tue... This builds you a directory ~/cmtuser/DaVinci_HEAD/. In there getpack what you need

Revision 82014-04-22 - MarcoCattaneo

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 9 to 9
 

Status of nightlies

The current status of the LHCb nightly builds is available at https://buildlhcb.cern.ch/nightlies/. See also the status of the LCG nightlies
Added:
>
>
RSS feeds reporting about error/warning details for slots, projects or specific platforms can be defined on the summary webpage:
  • use "Custom RSS feed" button in the top of the summary webpage
 

Configuration

The Nightly Build System configuration is defined in terms of slots, projects and platforms.

Revision 72014-04-22 - RobLambert

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 48 to 48
 
<!-- SyntaxHighlightingPlugin -->
https://buildlhcb.cern.ch/artifacts/<slot>/<day>/newrefs.<platform>/...
<!-- end SyntaxHighlightingPlugin -->
See LBCORE-164
Added:
>
>
If you want to copy the nightly reference files to simply upload them as replacements without needing to re-run the tests yourself, take a look at getNightlyRefs a script recently added to Lbscripts which uses rsynch to add the .new files to some local checkout. It has its own help but doesn't work outside of the CERN firewall.
 

Nightly tests input data

When nightly tests require event data as input, they should used files stored in the CERN-EOS-SWTEST storage element, and described in the Test Files database. See the TestFileDB TWiki for details

Revision 62014-03-04 - MarcoCattaneo

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 43 to 43
  See also the SetupProject wiki.
Changed:
<
<

Nightlies tests reference files

>
>

Nightly tests reference files

 The testing infrastructure is described in GaudiTestingInfrastructure wiki. The nightlies execute in all slots all the tests that have been defined for packages being build in that slot. The tests that fail because of a mismatch between the output and the reference file produce a special file with extension '.new'. These references are saved in the directory
<!-- SyntaxHighlightingPlugin -->
https://buildlhcb.cern.ch/artifacts/<slot>/<day>/newrefs.<platform>/...
<!-- end SyntaxHighlightingPlugin -->
See LBCORE-164
Added:
>
>

Nightly tests input data

When nightly tests require event data as input, they should used files stored in the CERN-EOS-SWTEST storage element, and described in the Test Files database. See the TestFileDB TWiki for details
 

Documentation

Revision 52014-02-12 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 18 to 18
  For the platforms we use the strings that identify the SupportedPlatforms.
Changed:
<
<
Note: The format of the configuration files might change in the future (see LBCORE-110), but the old format is described in LHCbNightliesConfiguration#Configuration.
>
>
Note: The format of the configuration files might change in the future (see LBCORE-110), but the old format is described in LHCbNightliesOldConfiguration#Configuration.
 

Regular builds

Every night we start the build several slots, which configurations are stored on SVN (LHCbNightlyConf).

Revision 42014-01-06 - MarcoCattaneo

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 23 to 23
 

Regular builds

Every night we start the build several slots, which configurations are stored on SVN (LHCbNightlyConf).
Changed:
<
<
Which slots are built (and when they are started) is defined in the configuration of the https://buildlhcb.cern.ch/jenkins/view/Nightly%20Builds/.
>
>
Which slots are built (and when they are started) is defined in the configuration of the Jenkins server.
 

Monitoring

%TODO%
Line: 43 to 43
  See also the SetupProject wiki.
Added:
>
>

Nightlies tests reference files

The testing infrastructure is described in GaudiTestingInfrastructure wiki. The nightlies execute in all slots all the tests that have been defined for packages being build in that slot. The tests that fail because of a mismatch between the output and the reference file produce a special file with extension '.new'. These references are saved in the directory
<!-- SyntaxHighlightingPlugin -->
https://buildlhcb.cern.ch/artifacts/<slot>/<day>/newrefs.<platform>/...
<!-- end SyntaxHighlightingPlugin -->
See LBCORE-164
 

Documentation

Revision 32013-12-02 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 21 to 21
 Note: The format of the configuration files might change in the future (see LBCORE-110), but the old format is described in LHCbNightliesConfiguration#Configuration.

Regular builds

Changed:
<
<
%TODO% Starting from the configuration stored on SVN, every night we trigger a
>
>
Every night we start the build several slots, which configurations are stored on SVN (LHCbNightlyConf).

Which slots are built (and when they are started) is defined in the configuration of the https://buildlhcb.cern.ch/jenkins/view/Nightly%20Builds/.

 

Monitoring

%TODO%

Revision 22013-10-23 - MarcoCattaneo

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

Line: 6 to 6
 

The System

The LHCb Nightly Build System is a collection of tools and scripts that allow automation of build and test tasks for LHCb software.
Added:
>
>

Status of nightlies

The current status of the LHCb nightly builds is available at https://buildlhcb.cern.ch/nightlies/. See also the status of the LCG nightlies
 

Configuration

The Nightly Build System configuration is defined in terms of slots, projects and platforms.
Line: 18 to 21
 Note: The format of the configuration files might change in the future (see LBCORE-110), but the old format is described in LHCbNightliesConfiguration#Configuration.

Regular builds

Changed:
<
<
Starting from the configuration stored on SVN, every night we trigger a
>
>
%TODO% Starting from the configuration stored on SVN, every night we trigger a
 

Monitoring

%TODO%
Added:
>
>

Running from the nightlies

The following instructions are for DaVinci, but they apply to any other project; instructions for Gauss are here. The first thing is to decide which slot. Usually one slot builds DaVinci on the latest LHCb, or the LHCb release candidate, and the other uses the head of Gaudi. Which one to pick is up to what you want to do. See https://buildlhcb.cern.ch/nightlies/ for the definitions. Then decide on the day. Make sure that the version you picked actually compiles. Now you have a slot, say lhcb-head and a day, say last night.

First do

<!-- SyntaxHighlightingPlugin -->
SetupProject DaVinci HEAD --build-env --nightly lhcb-head [ day ]
<!-- end SyntaxHighlightingPlugin -->
where day is optional. The default is today, or pick up a day like Mon, Tue... This builds you a directory ~/cmtuser/DaVinci_HEAD/. In there getpack what you need
<!-- SyntaxHighlightingPlugin -->
getpack Phys/DaVinci head
<!-- end SyntaxHighlightingPlugin -->
and the needed packages. Then do
<!-- SyntaxHighlightingPlugin -->
SetupProject DaVinci HEAD --nightly lhcb-head [ day ]
<!-- end SyntaxHighlightingPlugin -->
i.e. the same as above without the --build-env. Now $DAVINCIROOT should point to your getpacked DaVinci and you can run.

You can also run from ganga (not on the grid, only at CERN) using the setupProjectOptions field to pass the right options to SetupProject.

See also the SetupProject wiki.

Documentation

  • A New Nightly Build System for LHCb (Marco Clemencic and Ben Couturier) LHCb-INT-2013-006, Poster-013-328
  • Nightly build and test system supports LHC experiments (Stefan Roiser, Ana Gaspar, Yves Perrin, Victor Diez and Karol Kruzelecki) CERN Computer Newsletter April-June 2009
  • The nightly build and test system for LCG AA and LHCb software (Karol Kruzelecki, Stefan Roiser and Hubert Degaudenzi) Computing in High Energy and Nuclear Physics (CHEP 2009) Prague, Czech Republic, March 21-27, 2009 LHCb-CONF-2009-007, presented as a poster.
  -- MarcoClemencic - 01-Oct-2013 \ No newline at end of file

Revision 12013-10-01 - MarcoClemencic

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="LHCbComputing"

LHCb Nightly Build System

The System

The LHCb Nightly Build System is a collection of tools and scripts that allow automation of build and test tasks for LHCb software.

Configuration

The Nightly Build System configuration is defined in terms of slots, projects and platforms.

A slot is a consistent set of inter-related software projects that should be built and tested together. The configuration of a slot includes, in addition to the list of configured projects, the list of platforms the slot should be built for and some extra metadata for the fine tuning of the build.

Software projects are defined in the slot configuration with enough detail to be able to check-out the code from the software repositories. We can use an existing tag, the latest version (trunk/HEAD) or fine tune the check-out overriding the versions of the packages constituting the projects.

For the platforms we use the strings that identify the SupportedPlatforms.

Note: The format of the configuration files might change in the future (see LBCORE-110), but the old format is described in LHCbNightliesConfiguration#Configuration.

Regular builds

Starting from the configuration stored on SVN, every night we trigger a

Monitoring

%TODO%

-- MarcoClemencic - 01-Oct-2013

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback