Difference: LHCbNightliesOldConfiguration (1 vs. 26)

Revision 262014-02-12 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbNightliesOld"

LHCb Nightly builds setup and configuration

Line: 651 to 651
 
    • checkout/prepare acrontab
    • checkout sensors
Changed:
<
<
META TOPICMOVED by="KarolKruzelecki" date="1247756095" from="Main.LHCbNightliesConfiguration" to="LHCb.LHCbNightliesConfiguration"
>
>
META TOPICMOVED by="clemenci" date="1392203670" from="LHCb.LHCbNightliesConfiguration" to="LHCb.LHCbNightliesOldConfiguration"

Revision 252013-09-27 - MarcoClemencic

Line: 1 to 1
Changed:
<
<
META TOPICPARENT name="LHCbNightlies"
>
>
META TOPICPARENT name="LHCbNightliesOld"
 

LHCb Nightly builds setup and configuration

Line: 520 to 520
 

File structure

  • nightlies related files
Changed:
<
<
    • Nightlies: scripts necessary for running the nightlies, e.g. LHCbNightlies.py...
>
>
    • Nightlies: scripts necessary for running the nightlies, e.g. LHCbNightliesOld.py...
 
      • bin: helper programs around the Nightlies, e.g. nightliesClient.sh...
    • Webpage: scripts for displaying the results, e.g. LHCbNightliesWebpage.py,...
      • webpage helpers/cgi-scripts: nightlies.py,... (reside in /afs/cern.ch/lhcb/software/nightlies/www/cgi-bin)
Line: 539 to 539
 
        • /afs/cern.ch/lhcb/software/nightlies/SLOTS/WEEKDAYS
      • output/web
        • /afs/cern.ch/lhcb/software/nightlies/www/
Changed:
<
<
          • summaries index-LHCb-cache.html and index-LHCb-cache.xml (generated regularly by acron:=LHCbNightlies=)
>
>
          • summaries index-LHCb-cache.html and index-LHCb-cache.xml (generated regularly by acron:=LHCbNightliesOld=)
 
            • refer to job-outputs
Changed:
<
<
              • merged from LHCbNightlies from local dirs to /afs/cern.ch/lhcb/software/nightlies/www/logs/
>
>
              • merged from LHCbNightliesOld from local dirs to /afs/cern.ch/lhcb/software/nightlies/www/logs/
 
              • SLOT.DAY_PROJECT_PLATFORM{.log,-log.html,-log.summary,-qmtest.log,-qmtest (DIR) }
          • tests index-LHCb-cache.html and index-LHCb-cache.xml (generated by hand)
          • acron outputs: /afs/cern.ch/lhcb/software/nightlies/www/logs/lient.DAY_mainlog_MACHINE.cern.ch.txt
Line: 564 to 564
 
        • on each work node buildlhcb0?: working directory /build/nightlies/lhcb-newslot (the base directory should be writable, in case the system has to create it)
          • corresponding parameters in the slot config:
              <path value="%BUILDROOT%/nightlies/%SLOT%/%DAY%/%CMTCONFIG%" name="builddir"/>
            <path value="%BUILDROOT%/builders/%SLOT%" name="buildersdir"/>
Changed:
<
<
          • the directories for each day are created by LHCbNightlies.py when running on a node /build/nightlies/lhcb-newslot/Mon, /build/nightlies/lhcb-newslot/Tue, ...
>
>
          • the directories for each day are created by LHCbNightliesOld.py when running on a node /build/nightlies/lhcb-newslot/Mon, /build/nightlies/lhcb-newslot/Tue, ...
 
      • release directory:
        • the directory/AFS volume containing the build outputs
          • the parameter in the slot's configuration for setting the directory looks like:
             <path value="%AFSROOT%/cern.ch/lhcb/software/nightlies/%SLOT%/%DAY%" name="releasedir"/>
Line: 589 to 589
 

setting up a test system

Changed:
<
<
  • check out the main system LHCbNightlies.py and LHCbNightliesWebpage.py
>
>
  • check out the main system LHCbNightliesOld.py and LHCbNightliesWebpage.py
 
Line: 636 to 636
 

test system

  • checkout
    • base system (trunk/tag)
Changed:
<
<
      • LHCbNightlies.py
>
>
      • LHCbNightliesOld.py
 
    • helper scripts (trunk/tag)
    • configuration
    • LCG nightlies

Revision 242012-12-19 - ThomasHartmann

Line: 1 to 1
 
META TOPICPARENT name="LHCbNightlies"

LHCb Nightly builds setup and configuration

Line: 559 to 559
 
    • if configuration-parameters are set via environment variables, check if they are getting set properly, i.e. $BUILDROOT, $AFSROOT,...
      • environment variables are exported by LbLogin, nightliesClient.sh
    • depending on the slot configuration -- build and output pathes have to be set and created accordingly (or vice versa)
Added:
>
>
    • 'standard' directories are
 
    • build directory:
      • on each work node buildlhcb0?: working directory /build/nightlies/lhcb-newslot (the base directory should be writable, in case the system has to create it)
        • corresponding parameters in the slot config:
            <path value="%BUILDROOT%/nightlies/%SLOT%/%DAY%/%CMTCONFIG%" name="builddir"/>
          <path value="%BUILDROOT%/builders/%SLOT%" name="buildersdir"/>
        • the directories for each day are created by LHCbNightlies.py when running on a node /build/nightlies/lhcb-newslot/Mon, /build/nightlies/lhcb-newslot/Tue, ...
    • release directory:
Changed:
<
<
      • directory/AFS volume, e.g. in $LHCBNIGHTLIES/lhcb-newslot aka /afs/cern.ch/lhcb/software/nightlies/lhcb-newslot
        • for smaller slots one volume for all days is sufficient
          • /afs/cern.ch/lhcb/software/nightlies/lhcb-newslot/Mon,...
>
>
        • the directory/AFS volume containing the build outputs
          • the parameter in the slot's configuration for setting the directory looks like:
             <path value="%AFSROOT%/cern.ch/lhcb/software/nightlies/%SLOT%/%DAY%" name="releasedir"/>
        • due to the size limits of AFS volumes it is advisable to create small general directory/volume for the slot and add for each day an own AFS volume with enough AFS space allocated
          • $LHCBNIGHTLIES/lhcb-newslot aka /afs/cern.ch/lhcb/software/nightlies/lhcb-newslot
          • for smaller slots one volume for all days is sufficient and one can add the days as subdirectories
            • volume/mount point creation: afs_admin create -q 20000 /afs/cern.ch/lhcb/software/nightlies/lhcb-newslot q.lhcb.nght_new
            • subdirectories for each day mkdir /afs/cern.ch/lhcb/software/nightlies/lhcb-newslot/Mon,...
 
        • for larger projects it is reasonable to create a directory in the AFS-base directory for the slot and create new AFS-volumes for each day
          • e.g. create a volume called q.lhcb.nght_new_Mon and mount it to the directory Mon
            • afs_admin create -q 80000000 /afs/cern.ch/lhcb/software/nightlies/lhcb-newslot/Mon q.lhcb.nght_new_Mon
Deleted:
<
<
        • parameter in the slot config:
           <path value="%AFSROOT%/cern.ch/lhcb/software/nightlies/%SLOT%/%DAY%" name="releasedir"/>
 
    • www directory:
Added:
>
>
        • the general www directory is located at $LHCBNIGHTLIES/www
          • the log files themselfes are located in $LHCBNIGHTLIES/www/logs
 
      • if the path is on AFS check if it is sufficiently large for the log-files
      • corresponding parameter in the slot config:
    • for production add new jobs to acron if necessary

Revision 232012-10-22 - ThomasHartmann

Line: 1 to 1
 
META TOPICPARENT name="LHCbNightlies"

LHCb Nightly builds setup and configuration

Line: 17 to 17
 Configuration editor:
https://cern.ch/lhcb-nightlies/editor.html
Added:
>
>

Setting up a new machine

How to set up a new machine as build machine

 

Server

Updated version of LHCb Nightlies run in client-server architecture. Server reads configuration file once, while it's started - we start it every night, just after midnight. Configuration is automatically checked out from SVN repository. To be able to start server on the same port the following day, we also kill the server just before the midnight:

Changed:
<
<
58 23 * * * lxbuild135 ~kkruzele/public/killtree.py kill `ps xf | grep "nightliesServer.sh" | grep -v "grep" | awk '{print $1}'` 20 0 * * * lxbuild135 ~/bin/nightliesServer.sh
>
>
58 23 * * * buildlhcb07 ~/bin/killtree.py kill `ps xf | grep "nightliesServer.sh" | grep -v "grep" | awk '{print $1}'` > /build/nightlies/OldLogs/KillServer.log 2>&1 20 0 * * * buildlhcb07 ~/bin/nightliesServer.sh > /afs/cern.ch/lhcb/software/nightlies/www/logs/nightliesServer_`date +\%a`_buildlhcb07.txt 2>&1
 
Added:
>
>
The server is set up to use buildlhcb07, aka lxbuild171. The server name is hard coded in the start-up scripts nightliesServer.ch and nightliesClient.sh, when moving the server the corresponding scripts and the acron settings have to be updated

restart/test mode

As standard the server picks the configuration.xml present at midnight of the new day to have a normed environment. As standard port 61007 is used (check for new machines, if the firewall settings allow access to this ports). To use the most current version of configuration.xml in SVN, the server can be started with a restart flag, i.e. ~/bin/nightliesServer.sh restart. Port 61009 is used for serving this configuration; clients have to be started with the same flag to access the restart server.

As test environment, the server can be started with the test flag; it pushes the most recent configuration-test.xml as configuration to the clients, that were started with test as well. Here, port 61008 is used. Be aware, that in the startup scripts the pathes to the nightlies files are switched as well to the development versions, when setting the test flag.

 

Clients

Line: 35 to 47
 10 1 * * * volhcb27 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh x86_64-slc5-gcc43-opt | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild157_1_`date | awk '{print $1}'`_mainlog.txt
Added:
>
>
Clients can be started for a specific slot and platform with
 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh PLATFORM SLOT

client restart

As the server clients can be started with the restart flag to use the most current configuration.xml (server started with restart flag set has to be running, i.e. the most current version of configuration.xml is the one present when starting the server, that serves it to the clients)

If the restart of a client is intended to supersede a already job, that has been running, several steps have to be considered. Most of the steps are taken care of, when using the ~/bin/restart.py --slot SLOT --platform PLATFORM script. The individual steps are:

  • a running job creates a control file $LHCBNIGHTLIES/SLOTNAME/DAY/isRunning-PLATFORM
    • if the file is present for a given slot&platform restart, the job will not start as to avoid collision with its predecessor
      • if the predecessor is known to have died, the file can be deleted
      • else the predecessor has to be killed manually on its build machine; to kill all daughter processes the killtree script should be used with something like: ~/bin/killtree.py kill `ps xf | grep "PLATFORM" | grep "SLOT"  | grep -v "grep" | awk '{print $1}'`
  • a finished job creates a control file $LHCBNIGHTLIES/SLOTNAME/DAY/isDone-PLATFORM -- it does not influences a restart
  • a predecessor's build files have to be purged locally (if running on the same build machine), the copied files on AFS are overwritten when the new job copies its files.
  • the same holds for the status and test files accessible from the status webpage. These files are located at $LHCBNIGHTLIES/www/logs/SLOTNAME/SLOTNAME.DAY_PROJECT_HEAD-PLATFORM(/.log/-log.html/-log.summary/.qmtest/qmtest.log)
    • if not (re)moved at the start of a restart, the new status files overwrite the old ones one by one -- which can cause confusion, when two status versions are mixed up

client test

To run a test job, the client can be started with the test flag, i.e.

/afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh PLATFORM SLOT test
 

Summary web page generation

Added:
>
>
The webpage script has to run on the same build machine, that is used for building Coverity since it accesses its files locally (if the webpage script is run on another machine, the Coverity related informations will be missing)
 
Changed:
<
<
*/15 * * * * lxplus export AFSROOT=/afs && /afs/cern.ch/user/l/lhcbsoft/PRODUCTION/prod-LCG/LHCbGenNightlies.py /afs/cern.ch/lhcb/software/nightlies/www/index-LHCb-cache.html
>
>
*/15 * * * * lhcb-coverity export AFSROOT=/afs && export PYTHONPATH=/afs/cern.ch/user/l/lhcbsoft/PRODUCTION/prod-LCG:/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/python:$PYTHONPATH && /afs/cern.ch/user/l/lhcbsoft/PRODUCTION/prod-LHCb/LbRelease/LHCbNightliesWebpage.py /afs/cern.ch/lhcb/software/nightlies/www/index-LHCb-cache.html > /afs/cern.ch/lhcb/software/nightlies/www/index-LHCb-cache.out 2>&1
 
Added:
>
>
A html page index-LHCb-cache.html and a corresponding xml file index-LHCb-cache.xml are created. When AFS is lagging, the webpage script can be running for quite some time, since it accesses all status files. To avopid collisions with each other, a webpage instance is checking, if aanother one is still running. The check can be bypassed with the force flag, i.e. LHCbNightliesWebpage.py OUTPUTNAME.html force
 

Cleaning steps

AFS

Added:
>
>
Build files copied to AFS
 
Changed:
<
<
5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-branches/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-patches/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-prerelease/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-head/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-gaudi-head/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-lcg-head/`date | awk '{print $1}'`/*
>
>
5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/SLOTNAME/`date | awk '{print $1}'`/**
 
Added:
>
>
15 0 * * * lxplus find /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-* -iname '*' -mtime +6 -print -exec rm -rf '{}' \; > /afs/cern.ch/lhcb/software/nightlies/www/logs/nightliesClient/removedFiles.log
 

Log files

Added:
>
>
Log files in $LHCBNIGHTLIES/www/logs/SLOTNAME/, picked up by the webpage scripts
 
Changed:
<
<
10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-gaudi-head.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-head-windows.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-lcg-head.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-branches.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-patches.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-prerelease.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-head.`date | awk '{print $1}'`_*
>
>
10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/SLOTNAME/SLOTNAME.`date | awk '{print $1}'`_*
 

/build directories

Changed:
<
<
>
>
local build directories
 
Changed:
<
<
10 0 * * * lxbuild111 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'` 10 0 * * * lxbuild120 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'` 10 0 * * * lxbuild135 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'` 10 0 * * * lxbuild156 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'` 10 0 * * * lxbuild157 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'`
>
>
10 0 * * * BUILDMACHINE rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'`
 
Added:
>
>

Monitoring

SLS sensors have been set up to monitor different aspects of the nightlies system
  • the local build machines disk usages
  • the AFS volumes usages, i.e. for each mounted volume below /afs/cern.ch/lhcb/software/nightlies
  • the webpage update intervalls
  • the RSS feed status, i.e. the sqlite database sometimes got corrupted during AFS errors
  • the Coverity processe running on =lhcb-coverity=s, i.e. the database status, the web interface status and the general Coverity status
    • currently the monitor shows an error on Mondays, since the LCG nightlies are not available. Since as input LCG's nightlies build each Monday are used as input, the corresponding files are missing on Monday's while they are build
 

Configuration

current configuration file is taken from SVN repository (not anymore from AFS):

Revision 222012-09-14 - ThomasHartmann

Line: 1 to 1
 
META TOPICPARENT name="LHCbNightlies"

LHCb Nightly builds setup and configuration

Line: 588 to 588
 
  • to do

Development

Added:
>
>
See LHCbNightliesDevelopment
 

test system

  • checkout
    • base system (trunk/tag)

Revision 212012-06-26 - MarcoCattaneo

Line: 1 to 1
 
META TOPICPARENT name="LHCbNightlies"

LHCb Nightly builds setup and configuration

Changed:
<
<
The system expects to find build machines (both server and clients) configured in the way described here: LHCbNightliesPrerequirements.
>
>
The system expects to find build machines (both server and clients) configured in the way described here: LHCbNightliesPrerequirements. This page lists also the currently available machines and their status
 

System bootstrap

Revision 202012-03-13 - ThomasHartmann

Line: 1 to 1
 
META TOPICPARENT name="LHCbNightlies"

LHCb Nightly builds setup and configuration

Line: 481 to 481
 

Structure of LHCb Nightlies

File structure

Added:
>
>
 
  • nightlies related files
    • Nightlies: scripts necessary for running the nightlies, e.g. LHCbNightlies.py...
      • bin: helper programs around the Nightlies, e.g. nightliesClient.sh...
Line: 489 to 490
 
 
      • javascript=/=css helpers -->
reside in /afs/cern.ch/lhcb/software/nightlies/www/js/ /afs/cern.ch/lhcb/software/nightlies/www/css/
  • rss db (resides in /afs/cern.ch/lhcb/software/nightlies/db not in .../www/... !!!)
  • acrontab: configuration of the acrontab
  • Added:
    >
    >
    • sensors: sensor scripts monitoring the [[https://sls.cern.ch/sls/service.php?id=LHCb_Nightlies][system]
     

    Relations

    • working/output directories
    Line: 513 to 515
     
    • testing
      • no relations to acron-starts: are run by hand
    Changed:
    <
    <

    >
    >

    Guides

    adding a new slot

    • for new slot with name lhcb-newslot
      • if configuration-parameters are set via environment variables, check if they are getting set properly, i.e. $BUILDROOT, $AFSROOT,...
        • environment variables are exported by LbLogin, nightliesClient.sh
      • depending on the slot configuration -- build and output pathes have to be set and created accordingly (or vice versa)
      • build directory:
        • on each work node buildlhcb0?: working directory /build/nightlies/lhcb-newslot (the base directory should be writable, in case the system has to create it)
          • corresponding parameters in the slot config:
              <path value="%BUILDROOT%/nightlies/%SLOT%/%DAY%/%CMTCONFIG%" name="builddir"/>
            <path value="%BUILDROOT%/builders/%SLOT%" name="buildersdir"/>
          • the directories for each day are created by LHCbNightlies.py when running on a node /build/nightlies/lhcb-newslot/Mon, /build/nightlies/lhcb-newslot/Tue, ...
      • release directory:
        • directory/AFS volume, e.g. in $LHCBNIGHTLIES/lhcb-newslot aka /afs/cern.ch/lhcb/software/nightlies/lhcb-newslot
          • for smaller slots one volume for all days is sufficient
            • /afs/cern.ch/lhcb/software/nightlies/lhcb-newslot/Mon,...
          • for larger projects it is reasonable to create a directory in the AFS-base directory for the slot and create new AFS-volumes for each day
            • e.g. create a volume called q.lhcb.nght_new_Mon and mount it to the directory Mon
              • afs_admin create -q 80000000 /afs/cern.ch/lhcb/software/nightlies/lhcb-newslot/Mon q.lhcb.nght_new_Mon
          • parameter in the slot config:
             <path value="%AFSROOT%/cern.ch/lhcb/software/nightlies/%SLOT%/%DAY%" name="releasedir"/>
      • www directory:
        • if the path is on AFS check if it is sufficiently large for the log-files
        • corresponding parameter in the slot config:
      • for production add new jobs to acron if necessary
      • add cleaning jobs to acrontab
        • 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-newslot.`date +\%a`_*
        • 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-newslot/`date +\%a`/*

    setting up a test system

    • check out the main system LHCbNightlies.py and LHCbNightliesWebpage.py
      • including helper NightliesXML.py
      • depend on LCG nightlies
      • depend on LbScripts: LbConfiguration, LbUtils
        • e.g. /afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/python/
    • checkout scripts for clients and server
      • nightliesClient.sh & nightliesServer.sh
        • setup in both scripts the pathes, which are exported and used in the system
    • prepare the base directories
    • start server with nightliesServer.sh
    • set in nightliesClient.sh the machine where the server is running
      • (have to use a parameter for that!)
      • or start LCG-client directly (check for environment variables)

    setting up a production system

    • basic steps as for a test system
    • check out helpers as killTree.sh
    • (semi-)automatization
      • checkout the acrontab
        • adapt the server and client starts as necessary
      • or set up a new acrontab
        • kill the server from the previous day and start a new one on a specific machine
          58 23 * * * lxbuild135 ~/bin/killtree.py kill `ps xf | grep "nightliesServer.sh" | grep -v "grep" | awk '{print $1}'` 2>&1
          20 0 * * * lxbuild135 ~/bin/nightliesServer.sh > /afs/cern.ch/lhcb/software/nightlies/www/logs/nightliesServer_`date +\%a`_lxbuild135.txt 2>&1
        • kill on each client machine any running nightlies
          3 0 * * * buildlhcb01 ~/bin/killTree.sh nightlies `date --date='1 day ago' +\%a` 2>&1
        • add starts for clients
          01 01 * * * buildlhcb01 ~/bin/nightliesClient.sh 686-slc5-gcc43-opt > /afs/cern.ch/lhcb/software/nightlies/www/logs/nightliesClient_`date +\%a`_1.txt 2>&1
        • update the webpage
          */15 * * * * lhcb-coverity export AFSROOT=/afs && export PYTHONPATH=/afs/cern.ch/user/l/lhcbsoft/PRODUCTION/prod-LCG:/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/python:$PYTHONPATH && /afs/cern.ch/user/l/lhcbsoft/PRODUCTION/prod-LHCb/LbRelease/LHCbNightliesWebpage.py /afs/cern.ch/lhcb/software/nightlies/www/index-LHCb-cache.html SVN > /afs/cern.ch/lhcb/software/nightlies/www/index-LHCb-cache.out 2>&1
          • if coverity is used, the webpage script has to run on the same machine as coverity
          • adapt pathes accordingly
        • add cleaning job on each node
          2 0 * * * buildlhcb01 rm -rf /build/nightlies/lhcb-*/`date --date='1 days ago' +\%a`/*
    • checkout sensor scripts and add starts to acrontab in sync (more or less) with the settings in SDB
      • parameters should hopefully be all parseble now
        */25 * * * * buildlhcb06 export AFSROOT=/afs && export PYTHONPATH=/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/python:$PYTHONPATH && /afs/cern.ch/user/l/lhcbsoft/PRODUCTION/prod-LHCb/LbRelease/NightliesSensors/WebpageSensor.py --ft 90 --fw /afs/cern.ch/lhcb/software/nightlies/www/LHCb_Nightlies_Webpage.xml --verbose /build/nightlies/OldLogs/SensorWebpageLog > /build/nightlies/OldLogs/SensorWebpage.out 2>&1
        */25 * * * * buildlhcb06 export AFSROOT=/afs && export PYTHONPATH=/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/python:$PYTHONPATH && /afs/cern.ch/user/l/lhcbsoft/PRODUCTION/prod-LHCb/LbRelease/NightliesSensors/RSSStatus.py --rw /afs/cern.ch/lhcb/software/nightlies/www/LHCb_Nightlies_RSS.xml --ri /afs/cern.ch/lhcb/software/nightlies/db/nightlies.results --verbose /build/nightlies/OldLogs/SensorRSSLog > /build/nightlies/OldLogs/SensorRSS.out 2>&1
        */25 * * * * buildlhcb06 export AFSROOT=/afs && export PYTHONPATH=/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/python:$PYTHONPATH && /afs/cern.ch/user/l/lhcbsoft/PRODUCTION/prod-LHCb/LbRelease/NightliesSensors/AFSSensor.py --aw /afs/cern.ch/lhcb/software/nightlies/www/LHCb_Nightlies_AFS.xml > /build/nightlies/OldLogs/AFSSensor.out 2>&1
        # sensor for coverity needs to run on lxbuild161 aka lhcb-coverity
        */25 * * * * lhcb-coverity export AFSROOT=/afs && export PYTHONPATH=/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/python:$PYTHONPATH && /afs/cern.ch/user/l/lhcbsoft/PRODUCTION/prod-LHCb/LbRelease/NightliesSensors/CoveritySensor.py --cw /afs/cern.ch/lhcb/software/nightlies/www/LHCb_Nightlies_Coverity.xml --verbose /build/nightlies/OldLogs/SensorCoverityLog > /build/nightlies/OldLogs/SensorCoverity.out 2>&1

    setting up Coverity

    • to do

    Development

    test system

    • checkout
      • base system (trunk/tag)
        • LHCbNightlies.py
      • helper scripts (trunk/tag)
      • configuration
      • LCG nightlies
      • LbScripts
    • create test slot
    • apply changes
      • start test server
      • run test client for the test slot
    • ...

    production system

    • in addition to a test system
      • checkout/prepare acrontab
      • checkout sensors
     
    META TOPICMOVED by="KarolKruzelecki" date="1247756095" from="Main.LHCbNightliesConfiguration" to="LHCb.LHCbNightliesConfiguration"

    Revision 192012-03-13 - ThomasHartmann

    Line: 1 to 1
     
    META TOPICPARENT name="LHCbNightlies"

    LHCb Nightly builds setup and configuration

    Line: 478 to 478
     
    • cerntslhcblhcb04: i686-winxp-vc9-dbg lhcb-gaudi-head
    • cerntslhcblhcb05: i686-winxp-vc9-dbg lhcb-prerelease
    Added:
    >
    >

    Structure of LHCb Nightlies

    File structure

    • nightlies related files
      • Nightlies: scripts necessary for running the nightlies, e.g. LHCbNightlies.py...
        • bin: helper programs around the Nightlies, e.g. nightliesClient.sh...
      • Webpage: scripts for displaying the results, e.g. LHCbNightliesWebpage.py,...
        • webpage helpers/cgi-scripts: nightlies.py,... (reside in /afs/cern.ch/lhcb/software/nightlies/www/cgi-bin)
        • javascript=/=css helpers -->
    reside in /afs/cern.ch/lhcb/software/nightlies/www/js/ /afs/cern.ch/lhcb/software/nightlies/www/css/
  • rss db (resides in /afs/cern.ch/lhcb/software/nightlies/db not in .../www/... !!!)
  • acrontab: configuration of the acrontab
  • Relations

    • working/output directories
      • local
        • build directories
          • /build/nightlies/SLOTS/WEEKDAYS
      • afs ($LHCBNIGHTLIES --> /afs/cern.ch/lhcb/software/nightlies/)
        • build directories
          • /afs/cern.ch/lhcb/software/nightlies/SLOTS/WEEKDAYS
        • output/web
          • /afs/cern.ch/lhcb/software/nightlies/www/
            • summaries index-LHCb-cache.html and index-LHCb-cache.xml (generated regularly by acron:=LHCbNightlies=)
              • refer to job-outputs
                • merged from LHCbNightlies from local dirs to /afs/cern.ch/lhcb/software/nightlies/www/logs/
                • SLOT.DAY_PROJECT_PLATFORM{.log,-log.html,-log.summary,-qmtest.log,-qmtest (DIR) }
            • tests index-LHCb-cache.html and index-LHCb-cache.xml (generated by hand)
            • acron outputs: /afs/cern.ch/lhcb/software/nightlies/www/logs/lient.DAY_mainlog_MACHINE.cern.ch.txt

    Stable/Testing

    • stable
      • started by acrontab daily
    • testing
      • no relations to acron-starts: are run by hand

     
    META TOPICMOVED by="KarolKruzelecki" date="1247756095" from="Main.LHCbNightliesConfiguration" to="LHCb.LHCbNightliesConfiguration"

    Revision 182011-11-16 - ThomasHartmann

    Line: 1 to 1
     
    META TOPICPARENT name="LHCbNightlies"

    LHCb Nightly builds setup and configuration

    Line: 407 to 407
     
      • it takes the slot and platform as argument, i.e.
        restart.py --slot SLOT--platform PLATFORM

    Restart procedure by hand

    Added:
    >
    >
    • a nightlies slot produces for a specific platform several files and directories, that have to be cleaned by hand before a restart -- to avoid collisions with the fresh restart
      • on the build machine the build directory /build/nightlies/SLOT/DAY/PLATFORM
      • the copied log-files on AFS of the form $LHCBNIGHTLIES/www/logs/SLOT.DAY_PROJECTs_PLATFORM plus the qmtest-directories with the same naming scheme
      • the empty files indicating if a build has been started or has finished
        • a finished build touches a file in $LHCBNIGHTLIES/SLOT/isDone-PLATFORM
        • if a build is still running a file $LHCBNIGHTLIES/SLOT/isStarted-PLATFORM exists
          • in this case one has to find the build on one of the build machines buildlhcb0[1-6] and kill it before restarting the build
     the following steps are necessary when restarting a slot/platform by hand:

    only specific (platform, slot) pair

    Line: 455 to 464
     

    Windows Nightlies

    Added:
    >
    >
    The support for Windows has been dropped

    Legacy information

     The nightlies for Windows have to be run on their on. The builds are started by a windows-port of crontab
    • the build-partition is F:\ with the build directory containing the crontab-dir and the actual build-dir in nightlies and scripts containing the Windows scripts
    • the AFS token is (in principle) renewed every hour with the script C:\Program Files\OpenAFS\Client\Program\afscreds.exe
    • everything is build locally on the machines, so the necessary files are (re)copied from AFS by python F:\build\scripts\cleanBuildDirs.py
    • the interface to the actual nightlies-scripts is done with f:\build\scripts\nightliesClient9.bat
    Changed:
    <
    <

    Setup

    >
    >

    Setup

     the builds for the different slots/architectures are distributed over the windows machines
    • cerntslhcblhcb03: i686-winxp-vc9-dbg lhcb-head
    • cerntslhcblhcb04: i686-winxp-vc9-dbg lhcb-gaudi-head

    Revision 172011-11-14 - ThomasHartmann

    Line: 1 to 1
     
    META TOPICPARENT name="LHCbNightlies"

    LHCb Nightly builds setup and configuration

    Line: 398 to 398
     

    How to restart the Nightlies

    Production slots of the Nightlies run from lhcbsoft account. One has to login as lhcbsoft to be able to restart the Nightlies.

    Changed:
    <
    <

    only specific (platform, slot) pair

    >
    >

    Restart by script

    • for restart a nightlies restart a running server instance is necessary
      • start on machine lxbuild135 a server with ~/bin/nightliesServer.sh restart
      • it will pick up the most recent version of the configuration
    • on a build node a slot on a specific platform can be restarted with the script /afs/cern.ch/user/l/lhcbsoft/bin/restart.py
      • it takes the slot and platform as argument, i.e.
        restart.py --slot SLOT--platform PLATFORM

    Restart procedure by hand

    the following steps are necessary when restarting a slot/platform by hand:

    only specific (platform, slot) pair

      If you skip the optional steps, old log files will be still visible on the summary webpage and they will be overwritten one by one, while the process go on. It may confuse people checking the webpage.
    • the platform you want to restart may not be still running (the build must be stopped in that case before going on - you may use killtree LHCb script)
    Line: 406 to 416
     
    • (optional) remove the isDone file for the slot/platform from AFS slot directory (for example: /afs/cern.ch/lhcb/software/nightlies/lhcb-head/Wed/isDone-x86_64-slc5-gcc43-opt)
    • remove local build files for the specific slot/platform (for example: /build/nightlies/lhcb-head/Wed/x86_64-slc5-gcc43-opt)
    • start (you may consider using screen for that): ~/bin/nightliesClient.sh x86_64-slc5-gcc43-opt lhcb-head. You should check if you are using the build server (slc4/slc5) you really intend to use.
    Changed:
    <
    <

    all platforms in one slot

    >
    >

    all platforms in one slot

     
    • for the moment, the only solution is to repeat the procedure for "only specific (platform, slot) pair" from above
    Changed:
    <
    <

    all platforms in all slots

    >
    >

    all platforms in all slots

     
    • make sure that all the nightliesClient s on all build servers are stoped
    • stop the nightliesServer (usually running on lxbuild135)
    • start a new nightliesServer

    Revision 162011-10-21 - ThomasHartmann

    Line: 1 to 1
     
    META TOPICPARENT name="LHCbNightlies"

    LHCb Nightly builds setup and configuration

    Line: 131 to 131
     
    Added:
    >
    >
     
    Line: 358 to 359
     
    Added:
    >
    >
      The following can be set additionally for each of the projects in the slot:
    Changed:
    <
    <
    • dependence - modify the project.cmt file to use different project version (for projects one depends on). IMPORTANT: for the time being, while setting the dependence, lowercase project names should be used. Usage of the real project names must be tested.
    • change - change the specified package version; it overwrites headofeverything ("headofeverything but...").
    >
    >
    • dependence - modify the project.cmt file to use different project version (for projects one depends on). IMPORTANT: for the time being, while setting the dependence, lowercase project names should be used. Usage of the real project names must be tested.
    • change - change the specified package version; it overwrites headofeverything ("headofeverything but...").
    • addon - adds a package which is not present in the requirements of the project
      Schema constraints:
    • dependence - 'project' and 'tag' are obligatory
    • change - 'package' and 'value' are obligatory
    Added:
    >
    >
    • addon - 'package' and 'value' are obligatory
     
    • project - order is dependences first and then changes

    Troubleshooting

    Revision 152011-06-07 - ThomasHartmann

    Line: 1 to 1
     
    META TOPICPARENT name="LHCbNightlies"

    LHCb Nightly builds setup and configuration

    Line: 449 to 449
     

    Setup

    the builds for the different slots/architectures are distributed over the windows machines
    Added:
    >
    >
    • cerntslhcblhcb03: i686-winxp-vc9-dbg lhcb-head
     
    • cerntslhcblhcb04: i686-winxp-vc9-dbg lhcb-gaudi-head
    Changed:
    <
    <
    • cerntslhcblhcb05: i686-winxp-vc9-dbg lhcb-head, i686-winxp-vc9-dbg lhcb-prerelease
    >
    >
    • cerntslhcblhcb05: i686-winxp-vc9-dbg lhcb-prerelease
     
    META TOPICMOVED by="KarolKruzelecki" date="1247756095" from="Main.LHCbNightliesConfiguration" to="LHCb.LHCbNightliesConfiguration"

    Revision 142011-05-17 - ThomasHartmann

    Line: 1 to 1
     
    META TOPICPARENT name="LHCbNightlies"

    LHCb Nightly builds setup and configuration

    Line: 411 to 411
     
    • start a new nightliesServer
    • start one or more nightliesClient s
    Changed:
    <
    <
    nightliiesServer by default takes configuration from SVN, the last one from yesterday. If you want to restart the Nightlies because of the chacnge in configuration - you have to force the server to take another (for example recent) configuration revision.
    >
    >
    nightliesServer by default takes configuration from SVN, the last one from yesterday. If you want to restart the Nightlies because of the chacnge in configuration - you have to force the server to take another (for example recent) configuration revision.
     

    how to use not default (last from yesterday SVN) configuration

    Configuration is handled by nightliesServer. If you want to use other configuration source:

    Line: 423 to 423
     you need another nightliesServer instance (another port number) and dedicated nightliesClients to connect to.

    For the moment the easiest solution is to use nightliesServer-restart script, and nightliesClient-restart script, modifying all necessary parameters (source of configuraton) in the script itself.

    Added:
    >
    >

    Restarting

    For a restart or a start with a different configuration several points have to be taken into account
    • on which port the server is listening
    • on which port the client for a specific slot is listening on
    • if there are still clients running a slot and writing to the slot directories
    • if there is a client idle waiting for work

    A normal start of the server nightliesServer.py will pick up yesterdays configuration.xml to have a well-defined configuration and will listen on port 61008 (started without an argument the server and client are using port 61007). If nightliesServer.py is started with the restart argument it will pick up the most recent version of configuration.xml for all slots etc. Starting the server with the argument test it will use on port 61008 and use the most recent version of configuration-test.xml from svn (configuration-test.xml has to be edited via svn). Anyway, one has to take care, that for (re)starting a nightlies slot are all slots:

    • that no other client is still running for the same slot/architecture, i.e. if necessary use killtree.py or ps xf | grep lhcb-what2kill | grep -v whatnot2kill | awk '{print $1}' | xargs kill
    • the work directory of the slot(s) to run are clean, else it will probably pick up old stuff
    • that the server is running/started on the desired port with the desired configuration
    • the client is started with the desired slot and platform on the right port
      • if the client is run without a specific platform/slot it will run waiting for work -- and will probably pick up an undesired slot/platform from the server
     

    Windows Nightlies

    Revision 132011-05-16 - ThomasHartmann

    Line: 1 to 1
     
    META TOPICPARENT name="LHCbNightlies"

    LHCb Nightly builds setup and configuration

    Revision 122011-05-16 - ThomasHartmann

    Line: 1 to 1
     
    META TOPICPARENT name="LHCbNightlies"

    LHCb Nightly builds setup and configuration

    Line: 434 to 434
     

    Setup

    the builds for the different slots/architectures are distributed over the windows machines
    Changed:
    <
    <
    • cerntslhcblhcb03: i686-winxp-vc9-dbg lhcb-gaudi-head (?!)
    • cerntslhcblhcb04: i686-winxp-vc9-dbg lhcb-gaudi-head (?!)
    >
    >
    • cerntslhcblhcb04: i686-winxp-vc9-dbg lhcb-gaudi-head
     
    • cerntslhcblhcb05: i686-winxp-vc9-dbg lhcb-head, i686-winxp-vc9-dbg lhcb-prerelease

    META TOPICMOVED by="KarolKruzelecki" date="1247756095" from="Main.LHCbNightliesConfiguration" to="LHCb.LHCbNightliesConfiguration"

    Revision 112011-05-16 - ThomasHartmann

    Line: 1 to 1
     
    META TOPICPARENT name="LHCbNightlies"

    LHCb Nightly builds setup and configuration

    Line: 424 to 424
      For the moment the easiest solution is to use nightliesServer-restart script, and nightliesClient-restart script, modifying all necessary parameters (source of configuraton) in the script itself.
    Added:
    >
    >

    Windows Nightlies

    The nightlies for Windows have to be run on their on. The builds are started by a windows-port of crontab

    • the build-partition is F:\ with the build directory containing the crontab-dir and the actual build-dir in nightlies and scripts containing the Windows scripts
    • the AFS token is (in principle) renewed every hour with the script C:\Program Files\OpenAFS\Client\Program\afscreds.exe
    • everything is build locally on the machines, so the necessary files are (re)copied from AFS by python F:\build\scripts\cleanBuildDirs.py
    • the interface to the actual nightlies-scripts is done with f:\build\scripts\nightliesClient9.bat

    Setup

    the builds for the different slots/architectures are distributed over the windows machines
    • cerntslhcblhcb03: i686-winxp-vc9-dbg lhcb-gaudi-head (?!)
    • cerntslhcblhcb04: i686-winxp-vc9-dbg lhcb-gaudi-head (?!)
    • cerntslhcblhcb05: i686-winxp-vc9-dbg lhcb-head, i686-winxp-vc9-dbg lhcb-prerelease
     
    META TOPICMOVED by="KarolKruzelecki" date="1247756095" from="Main.LHCbNightliesConfiguration" to="LHCb.LHCbNightliesConfiguration"

    Revision 102011-04-15 - unknown

    Line: 1 to 1
     
    META TOPICPARENT name="LHCbNightlies"

    LHCb Nightly builds setup and configuration

    Line: 9 to 9
     

    System bootstrap

    Nighty builds are started every night with use of acrontab of lhcbsoft user. Configuration is kept in SVN repository:

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    https://svnweb.cern.ch/trac/lhcb/browser/LHCbNightlyConf/trunk/configuration.xml 
    <!-- end SyntaxHighlightingPlugin -->
    >
    >
    https://svnweb.cern.ch/trac/lhcb/browser/LHCbNightlyConf/trunk/configuration.xml
      Nightlies summary webpage:
    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    http://cern.ch/lhcb-nightlies 
    <!-- end SyntaxHighlightingPlugin -->
    >
    >
    http://cern.ch/lhcb-nightlies
      Configuration editor:
    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    https://cern.ch/lhcb-nightlies/editor.html 
    <!-- end SyntaxHighlightingPlugin -->
    >
    >
    https://cern.ch/lhcb-nightlies/editor.html
     

    Server

    Updated version of LHCb Nightlies run in client-server architecture. Server reads configuration file once, while it's started - we start it every night, just after midnight. Configuration is automatically checked out from SVN repository. To be able to start server on the same port the following day, we also kill the server just before the midnight:

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    58 23 * * * lxbuild135 ~kkruzele/public/killtree.py kill `ps xf | grep "nightliesServer.sh" | grep -v "grep" | awk '{print $1}'` 20 0 * * * lxbuild135 ~/bin/nightliesServer.sh 
    <!-- end SyntaxHighlightingPlugin -->
    >
    >
    58 23 * * * lxbuild135 ~kkruzele/public/killtree.py kill `ps xf | grep "nightliesServer.sh" | grep -v "grep" | awk '{print $1}'`
    20 0 * * * lxbuild135 ~/bin/nightliesServer.sh 
    
     

    Clients

    Client must be given at least one parameter on the command line - it's the CMTCONFIG which it supposed to be running with. After startup, each client connects to the server and gets one (slot,platform) pair which matches given CMTCONFIG (dbg platforms can be built with opt cmtconfigs...). When it finished the build, it takes another (slot,platform) pair as long as there is still one ready to built. If not, the client quits the loop. Some platforms are waiting for specific conditions to be fulfilled - if so, they may not be taken by the client when the client is ready. In that case, it may happen that the client finishes its work, and there are still platforms to be built. Therefore there should be more than one lients started during the night - some of them maybe even in the afternoon.

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    7 1 * * * lxbuild111 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh slc4_amd64_gcc34 | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild111_1_`date | awk '{print $1}'`_mainlog.txt 8 1 * * * lxbuild135 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh x86_64-slc5-gcc43-opt | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild135_1_`date | awk '{print $1}'`_mainlog.txt 9 1 * * * lxbuild156 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh slc4_ia32_gcc34 | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild156_1_`date | awk '{print $1}'`_mainlog.txt 10 1 * * * volhcb27 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh x86_64-slc5-gcc43-opt | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild157_1_`date | awk '{print $1}'`_mainlog.txt 
    <!-- end SyntaxHighlightingPlugin -->
    >
    >
    7 1 * * * lxbuild111 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh slc4_amd64_gcc34 | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild111_1_`date | awk '{print $1}'`_mainlog.txt
    8 1 * * * lxbuild135 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh x86_64-slc5-gcc43-opt | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild135_1_`date | awk '{print $1}'`_mainlog.txt
    9 1 * * * lxbuild156 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh slc4_ia32_gcc34 | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild156_1_`date | awk '{print $1}'`_mainlog.txt
    10 1 * * * volhcb27 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh x86_64-slc5-gcc43-opt | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild157_1_`date | awk '{print $1}'`_mainlog.txt 
    
     

    Summary web page generation

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    */15 * * * * lxplus export AFSROOT=/afs && /afs/cern.ch/user/l/lhcbsoft/PRODUCTION/prod-LCG/LHCbGenNightlies.py /afs/cern.ch/lhcb/software/nightlies/www/index-LHCb-cache.html 
    <!-- end SyntaxHighlightingPlugin -->
    >
    >
    */15 * * * * lxplus export AFSROOT=/afs && /afs/cern.ch/user/l/lhcbsoft/PRODUCTION/prod-LCG/LHCbGenNightlies.py /afs/cern.ch/lhcb/software/nightlies/www/index-LHCb-cache.html 
    
     

    Cleaning steps

    AFS

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-branches/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-patches/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-prerelease/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-head/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-gaudi-head/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-lcg-head/`date | awk '{print $1}'`/* 
    <!-- end SyntaxHighlightingPlugin -->
    >
    >
    5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-branches/`date | awk '{print $1}'`/*
    5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-patches/`date | awk '{print $1}'`/*
    5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-prerelease/`date | awk '{print $1}'`/*
    5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-head/`date | awk '{print $1}'`/*
    5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-gaudi-head/`date | awk '{print $1}'`/*
    5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-lcg-head/`date | awk '{print $1}'`/* 
    
     

    Log files

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-gaudi-head.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-head-windows.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-lcg-head.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-branches.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-patches.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-prerelease.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-head.`date | awk '{print $1}'`_* 
    <!-- end SyntaxHighlightingPlugin -->
    >
    >
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-gaudi-head.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-head-windows.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-lcg-head.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-branches.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-patches.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-prerelease.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-head.`date | awk '{print $1}'`_* 
    
     

    /build directories

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    10 0 * * * lxbuild111 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'` 10 0 * * * lxbuild120 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'` 10 0 * * * lxbuild135 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'` 10 0 * * * lxbuild156 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'` 10 0 * * * lxbuild157 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'` 
    <!-- end SyntaxHighlightingPlugin -->
    >
    >
    10 0 * * * lxbuild111 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'`
    10 0 * * * lxbuild120 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'`
    10 0 * * * lxbuild135 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'`
    10 0 * * * lxbuild156 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'`
    10 0 * * * lxbuild157 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'` 
    
     

    Configuration

    current configuration file is taken from SVN repository (not anymore from AFS):

    unless you are sure what you are doing, to edit configuration use only the web editor:

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    https://cern.ch/lhcb-nightlies/editor.html 
    <!-- end SyntaxHighlightingPlugin -->
    >
    >
     https://cern.ch/lhcb-nightlies/editor.html 
    
     

    xml file format:

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <configuration> <general> <parameters> <parameter name="" value="" /> &lt; ... &gt; </parameters> <ignore> <error value="" /> <error value="" type="" /> <warning value="" /> <warning value="" type="" /> </ignore> <mailto> <project name="lhcb"> <mail address="marco.cattaneo@cern.ch" edit="false" parent="false" /> <repository path="isscvs.cern.ch:/local/reps/lhcb" type="cvs"/> </project> </mailto> </general> <slot name="slot1"> <paths> <path name="builddir" value="/build/nightlies/%SLOT%/%DAY%/%CMTCONFIG%" /> <path name="buildersdir" value="/build/builders/%SLOT%" /> <path name="releasedir" value="/afs/cern.ch/lhcb/software/nightlies/%SLOT%/%DAY%" /> <path name="wwwdir" value="/afs/cern.ch/lhcb/software/nightlies/www/logs" /> </paths> <cmtprojectpath> <path value="/afs/cern.ch/lhcb/software/DEV/nightlies" /> </cmtprojectpath> <platforms> <platform name="slc4_amd64_gcc34" /> <platform name="x86_64-slc5-gcc43-opt" /> </platforms> <waitfor flag="/afs/cern.ch/sw/lcg/app/nightlies/dev1/%DAY%/isDone-%PLATFORM%" /> <cmtextratags value="use-distcc,no-pyzip" /> <lblogin linux="echo hello" /> <runafter linux="" /> <days mon="true" tue="true" wed="true" thu="true" fri="true" sat="true" sun="true" /> <projects> <project name="Lbcom" tag="LBCOM_v9r5p1" /> <project name="Phys" tag="PHYS_HEAD" headofeverything="true" /> <project name="Online" tag="ONLINE_v4r43"> <dependence project="Gaudi" tag="GAUDI_v21r10p1" /> <change package="Online/Presenter" value="v0r21" /> </project> </projects> </slot> <slot name="slot2"> <!-- slot contents --> </slot> &lt; ... &gt; </configuration> 
    <!-- end SyntaxHighlightingPlugin -->
    >
    >
    <configuration>
       <general>
          <parameters>
             <parameter name="" value="" />
             &lt; ... &gt;
          </parameters>
          <ignore>
             <error value="" />
             <error value="" type="" />
             <warning value="" />
             <warning value="" type="" />
          </ignore>
          <mailto>
             <project name="lhcb">
                <mail address="marco.cattaneo@cern.ch" edit="false" parent="false" />
                <repository path="isscvs.cern.ch:/local/reps/lhcb" type="cvs" />
             </project>
          </mailto>
       </general>
       <slot name="slot1">
          <paths>
             <path name="builddir" value="/build/nightlies/%SLOT%/%DAY%/%CMTCONFIG%" />
             <path name="buildersdir" value="/build/builders/%SLOT%" />
             <path name="releasedir" value="/afs/cern.ch/lhcb/software/nightlies/%SLOT%/%DAY%" />
             <path name="wwwdir" value="/afs/cern.ch/lhcb/software/nightlies/www/logs" />
          </paths>
          <cmtprojectpath>
             <path value="/afs/cern.ch/lhcb/software/DEV/nightlies" />
          </cmtprojectpath>
          <platforms>
             <platform name="slc4_amd64_gcc34" />
             <platform name="x86_64-slc5-gcc43-opt" />
          </platforms>
          <waitfor
             flag="/afs/cern.ch/sw/lcg/app/nightlies/dev1/%DAY%/isDone-%PLATFORM%" />
          <cmtextratags value="use-distcc,no-pyzip" />
          <lblogin linux="echo hello" />
          <runafter linux="" />
          <days mon="true" tue="true" wed="true" thu="true" fri="true" sat="true"
             sun="true" />
          <projects>
             <project name="Lbcom" tag="LBCOM_v9r5p1" />
             <project name="Phys" tag="PHYS_HEAD" headofeverything="true" />
             <project name="Online" tag="ONLINE_v4r43">
                <dependence project="Gaudi" tag="GAUDI_v21r10p1" />
                <change package="Online/Presenter" value="v0r21" />
             </project>
          </projects>
       </slot>
       <slot name="slot2"> <!-- slot contents -->
       </slot>
       &lt; ... &gt;
    </configuration>
    
     

    configuration/general/parameters currently available:

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <parameter name="wwwurl" value="http://cern.ch/lhcb-nightlies" /> <parameter name="logurl" value="http://cern.ch/lhcb-nightlies/logs" /> <parameter name="rrdurl" value="http://cern.ch/lhcb-nightlies/rrd" /> <parameter name="wwwtitle" value="Summaries of nightly builds for LHCb" /> <parameter name="mailfrom" value="karol.kruzelecki@cern.ch" /> <parameter name="mailsubjectprefix" value="[LHCb Nightlies]" /> <parameter name="shownotfinishedplatforms" value="true" /> 
    <!-- end SyntaxHighlightingPlugin -->
    >
    >
    <parameter name="wwwurl" value="http://cern.ch/lhcb-nightlies" />
    <parameter name="logurl" value="http://cern.ch/lhcb-nightlies/logs" />
    <parameter name="rrdurl" value="http://cern.ch/lhcb-nightlies/rrd" />
    <parameter name="wwwtitle" value="Summaries of nightly builds for LHCb" />
    <parameter name="mailfrom" value="karol.kruzelecki@cern.ch" />
    <parameter name="mailsubjectprefix" value="[LHCb Nightlies]" />
    <parameter name="shownotfinishedplatforms" value="true" /> 
    
     
    • wwwurl, logurl, rrdurl - URLs to find the files (index.html, log files, test statistic graphs)
    • wwwtitle - header of the Nightlies summary page
    • mailfrom - (cern) e-mail address - sender of the build/test results
    Line: 82 to 164
     

    configuration/general/ignore

    Each line of the log files is compared to all of the given error/warning statements and if any of error/warning values can be found in the line, the line is not marked as one containing an error/warning. Errors and warings are considered separately, but the values are common for all of the slots. Regular expressions (type="regex") or fnmat/glob expressions (type="fnmatch") can be used here. In the case of regex - remember about .* in the beginning and the end of the expression (unless you are sure what you are doing).

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <error value="" /> <error value="" type="" /> <warning value="" /> <warning type="regex" value=".*example.*" /> 
    <!-- end SyntaxHighlightingPlugin -->
    >
    >
    <error value="" />
    <error value="" type="" />
    <warning value="" />
    <warning type="regex" value=".*example.*" /> 
    
     Schema constraints:
    • error - 'value' is obligatory, 'type' is optional
    • warning - 'value' is obligatory, 'type' is optional
    Line: 93 to 178
     

    configuration/general/mailto

    All of the projects which are used in one or more slots must have a section here as follows:

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <project name="lhcb"> <mail address="marco.cattaneo@cern.ch" edit="false" parent="false" /> <repository path="isscvs.cern.ch:/local/reps/lhcb" type="cvs"/> </project> 
    <!-- end SyntaxHighlightingPlugin -->
    Name of the project should be given as they are (LHCb and DaVinci, not LHCB, DAVINCI). mail tag is used to send summary of the builds to the responsible person after the build is finished. The section can contain none, one, or more mail tags. edit and parent attributes are not in use at the moment. repository tag is not in use at the moment.
    >
    >
    <project name="lhcb">
       <mail address="marco.cattaneo@cern.ch" edit="false" parent="false" />
       <repository path="isscvs.cern.ch:/local/reps/lhcb" type="cvs" />
    </project> 
    
    Name of the project should be given as they are (LHCb and DaVinci, not LHCB, DAVINCI). mail tag is used to send summary of the builds to the responsible person after the build is finished. The section can contain none, one, or more mail tags. edit and parent attributes are not in use at the moment. repository tag is not in use at the moment.
      Schema constraints:
    • mail - 'address' is obligatory
    Line: 104 to 194
     

    configuration/general/builders

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <builders> <project name="Geant4" path="%LHCBRELEASES%/../nightlies/plugins/Geant4" /> </builders> 
    <!-- end SyntaxHighlightingPlugin -->
    In this section it can be configured which builders should be copied from the given location instead of being generated automatically (environment variables can be used as shown - case sensitive).
    >
    >
    <builders>
       <project name="Geant4" path="%LHCBRELEASES%/../nightlies/plugins/Geant4" />
    </builders>
    
    In this section it can be configured which builders should be copied from the given location instead of being generated automatically (environment variables can be used as shown - case sensitive).
      Schema constraints:
    • project - 'path' and 'name' are obligatory
    Line: 121 to 215
     

    main slot settings

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <slot name="lhcb2" description="lhcb 2 HEAD of everything on top of GAUDI_v21r0 and LCG_56" mails="true" hidden="false" computedependencies="false" renice="+19"> 
    <!-- end SyntaxHighlightingPlugin -->
    >
    >
    
    <slot name="lhcb2"
       description="lhcb 2 HEAD of everything on top of GAUDI_v21r0 and LCG_56"
       mails="true" hidden="false" computedependencies="false" renice="+19">  
    
     
    • name - to create a new slot AFS space has to be allocated for releasedir. lhcb[1-6] are production slots, lhcb-test[1-2] are test slots started from different (not lhcbsoft) account.
    • description - extra information about the slot
    • mails - if set to false no summary e-mails will be send after the each of the projects in this slot is finished
    Line: 152 to 250
     

    paths configuration

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <paths> <path name="builddir" value="/build/nightlies/%SLOT%/%DAY%" /> <path name="buildersdir" value="/build/builders/%SLOT%" /> <path name="releasedir" value="/afs/cern.ch/lhcb/software/nightlies/%SLOT%/%DAY%" /> <path name="wwwdir" value="/afs/cern.ch/lhcb/software/nightlies/www/logs" /> </paths> 
    <!-- end SyntaxHighlightingPlugin -->
    all paths in this section are obligatory
    >
    >
    <paths>
       <path name="builddir" value="/build/nightlies/%SLOT%/%DAY%" />
       <path name="buildersdir" value="/build/builders/%SLOT%" />
       <path name="releasedir" value="/afs/cern.ch/lhcb/software/nightlies/%SLOT%/%DAY%" />
       <path name="wwwdir" value="/afs/cern.ch/lhcb/software/nightlies/www/logs" />
    </paths> 
    
    all paths in this section are obligatory
     
    • builddir - local build directory (must exist in filesystem and write permission has to be set for lhcbsoft user)
    • buildersdir - local directory for storing building scripts (plugins which in case of LHCb software are common, generated automatically)
    Line: 165 to 269
     

    CMTPROJECTPATH

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <cmtprojectpath> <path value="/afs/cern.ch/user/m/marcocle/public" /> <path value="/afs/cern.ch/sw/Gaudi/releases" /> <path value="/afs/cern.ch/sw/lcg/app/nightlies/dev1/%DAY%" /> <path value="/afs/cern.ch/sw/lcg/app/releases" /> <path value="/afs/cern.ch/lhcb/software/releases" /> </cmtprojectpath> 
    <!-- end SyntaxHighlightingPlugin -->
    CMTPROJECTPATH variable, order is important, local build directory is prepended.
    >
    >
    <cmtprojectpath>
       <path value="/afs/cern.ch/user/m/marcocle/public" />
       <path value="/afs/cern.ch/sw/Gaudi/releases" />
       <path value="/afs/cern.ch/sw/lcg/app/nightlies/dev1/%DAY%" />
       <path value="/afs/cern.ch/sw/lcg/app/releases" />
       <path value="/afs/cern.ch/lhcb/software/releases" />
    </cmtprojectpath> 
    
    CMTPROJECTPATH variable, order is important, local build directory is prepended.
      Schema constraints:
    • path - 'value' is obligatory

    CMTEXTRATAGS

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <cmtextratags value="use-distcc" /> 
    <!-- end SyntaxHighlightingPlugin -->
    Setup of CMTEXTRATAGS value.
    >
    >
     <cmtextratags value="use-distcc" /> 
    
    Setup of CMTEXTRATAGS value.
      Schema constraints:
    • cmtextratags - 'value' is obligatory

    "wait for" flag

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <waitfor flag="/afs/cern.ch/sw/lcg/app/nightlies/dev1/%DAY%/isDone-%PLATFORM%" /> 
    <!-- end SyntaxHighlightingPlugin -->
    If waitfor flag is configured, compilation of each platform (see below) is not started until the file is available in given location.
    <!-- SyntaxHighlightingPlugin -->
    <platforms> <platform name="slc4_amd64_gcc34_dbg" /> <platform name="x86_64-slc5-gcc34-dbg" /> </platforms> 
    <!-- end SyntaxHighlightingPlugin -->
    This section defines platforms to be build in the slot. If wait for is used, this is the order the files (isDone files) are looked for. The one which is found (if they are different for each of the platforms) makes the system start checkout and build stages for the platform it describes. If wait for is not used - this is the order of platforms to be build. if any platform is specified as a parameter for the starting script (lhcbNightlies-????.sh), only the platforms found there will be taken into consideration for the machine.
    >
    >
    <waitfor flag="/afs/cern.ch/sw/lcg/app/nightlies/dev1/%DAY%/isDone-%PLATFORM%" /> 
    
    If waitfor flag is configured, compilation of each platform (see below) is not started until the file is available in given location.
    <platforms>
       <platform name="slc4_amd64_gcc34_dbg" />
       <platform name="x86_64-slc5-gcc34-dbg" />
    </platforms> 
    
    This section defines platforms to be build in the slot. If wait for is used, this is the order the files (isDone files) are looked for. The one which is found (if they are different for each of the platforms) makes the system start checkout and build stages for the platform it describes. If wait for is not used - this is the order of platforms to be build. if any platform is specified as a parameter for the starting script (nightliesClient.sh), only the platforms found there will be taken into consideration for the machine.
      Schema constraints:
    • waitfor - 'flag' is obligatory
    • platform - 'name' is obligatory
    • platforms - must contain at least one platform tag
    Deleted:
    <
    <

    LbLogin settings

    <!-- SyntaxHighlightingPlugin -->
    <lblogin linux="source /afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/scripts/LbLogin.sh --no-cache --quiet --cmtvers=v1r20p20090520 ; export PATH=$PATH:/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/LBSCRIPTS_v4r1/InstallArea/scripts ; export VERBOSE=1" /> <lblogin win="call %HOMEDRIVE%/local/lib/LbLogin.bat" /> 
    <!-- end SyntaxHighlightingPlugin -->
    Should be used only for setting up different login environment. The given string is prepanded to the make cmt action inside builders, and it's launched just before the cmt action (in the same command).

    Schema constraints:

    • lblogin - no constraints, but should define at least 'linux' or 'win' or 'mac'

    runbefore/runafter

    <!-- SyntaxHighlightingPlugin -->
    <runbefore linux="any bash command && sequence of commands" win="any windows shell command" /> <runafter linux="any bash command && sequence of commands" win="any windows shell command"/> 
    <!-- end SyntaxHighlightingPlugin -->
    Commands to run before/after the run action is started.

    Schema constraints:

    • runbefore/runafter - no constraints, but should define at least 'linux' or 'win' or 'mac'
     

    days

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <days mon="true" tue="true" wed="true" thu="true" fri="true" sat="true" sun="true" /> 
    <!-- end SyntaxHighlightingPlugin -->
    Setting any of the weekdays to false will make the Nightlies not run this day.
    >
    >
    <days mon="true" tue="true" wed="true" thu="true" fri="true" sat="true" sun="true" /> 
    
    Setting any of the weekdays to false will make the Nightlies not run this day.
      Schema constraints:
    • days - 'mon', 'tue', 'wed', 'thu', 'fri', 'sat', 'sun' are obligatory

    list of projects

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <projects> <project name="Gaudi" tag="GAUDI_v21r0"> <dependence project="lcgcmt" tag="LCGCMT_56" /> </project> <project name="LHCb" tag="LHCB_HEAD" headofeverything="true"> </project> <project name="Lbcom" tag="LBCOM_HEAD" headofeverything="true" /> <project name="Gauss" tag="GAUSS_HEAD" headofeverything="true"> <dependence project="geant4" tag="GEANT4_v91r2p1" /> </project> <project name="Panoptes" tag="PANOPTES_HEAD" headofeverything="true" /> </projects> 
    <!-- end SyntaxHighlightingPlugin -->
    List of projects to be build in the slot; order is important. For each of the projects the following keys are obligatory:
    >
    >
    <projects>
       <project name="Gaudi" tag="GAUDI_v21r0">
          <dependence project="lcgcmt" tag="LCGCMT_56" />
       </project>
       <project name="LHCb" tag="LHCB_HEAD" headofeverything="true">
       </project>
       <project name="Lbcom" tag="LBCOM_HEAD" headofeverything="true" />
       <project name="Gauss" tag="GAUSS_HEAD" headofeverything="true">
          <dependence project="geant4" tag="GEANT4_v91r2p1" />
       </project>
       <project name="Panoptes" tag="PANOPTES_HEAD" headofeverything="true" />
    </projects> 
    
    List of projects to be build in the slot; order is important. For each of the projects the following keys are obligatory:
     
    • name - real name of the project (LHCb, DaVinci, not: LHCB, davinci)
    • tag
    Line: 232 to 350
     

    tuning the project configuration

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <project name="Gauss" tag="GAUSS_HEAD" headofeverything="true"> <dependence project="geant4" tag="GEANT4_v91r2p1" /> </project> <project name="LHCb" tag="LHCB_HEAD" headofeverything="true"> <dependence project="gaudi" tag="GAUDI_HEAD" /> <change package="Phys/LoKiArrayFunctors" value="vanya_20090424"/> <change package="Det/DetDescSvc" value="v2r2"/> </project> 
    <!-- end SyntaxHighlightingPlugin -->
    The following can be set additionally for each of the projects in the slot:
    >
    >
    <project name="Gauss" tag="GAUSS_HEAD" headofeverything="true">
       <dependence project="geant4" tag="GEANT4_v91r2p1" />
    </project>
    <project name="LHCb" tag="LHCB_HEAD" headofeverything="true">
       <dependence project="gaudi" tag="GAUDI_HEAD" />
       <change package="Phys/LoKiArrayFunctors" value="vanya_20090424" />
       <change package="Det/DetDescSvc" value="v2r2" />
    </project> 
    
    The following can be set additionally for each of the projects in the slot:
     
    • dependence - modify the project.cmt file to use different project version (for projects one depends on). IMPORTANT: for the time being, while setting the dependence, lowercase project names should be used. Usage of the real project names must be tested.
    • change - change the specified package version; it overwrites headofeverything ("headofeverything but...").
    Line: 245 to 372
     

    Troubleshooting

    Logs

    • each acrontab job output is sent to the lhcbsoft e-mail account unless output is empty. In the case of LHCb Nightlies there should be an e-mail for every job because some diagnostic information is always printed. This log contains checkout log which is not stored in any separate file. Build output is not printed to this log.
    Changed:
    <
    <
    • build log can be found for each project on the summary webpage and on build node local disk (files are kept on local discs for one day only!):
      /build/nightlies/<slot>/<day>/<project>/<tag>/logs/<platform>.log
    • test log can be found for each project on the build node local disk only (files are kept on local discs for one day only!):
      /build/nightlies/<slot>/<day>/<project>/<tag>/logs/<platform>-tests.log
    • test log summary can be found for each project on the summary webpage and on build node local disk (files are kept on local discs for one day only!):
      /build/nightlies/<slot>/<day>/<project>/<tag>/logs/<platform>-qmtest.log
    >
    >
    • build log can be found for each project on the summary webpage and on build node local disk (files are kept on local discs for one day only!):
    /build/nightlies/<slot>/<day>/<project>/<tag>/logs/<platform>.log
    • test log can be found for each project on the build node local disk only (files are kept on local discs for one day only!):
    /build/nightlies/<slot>/<day>/<project>/<tag>/logs/<platform>-tests.log
    • test log summary can be found for each project on the summary webpage and on build node local disk (files are kept on local discs for one day only!):
    /build/nightlies/<slot>/<day>/<project>/<tag>/logs/<platform>-qmtest.log
     

    Possible problems

    Revision 92010-11-04 - unknown

    Line: 1 to 1
     
    META TOPICPARENT name="LHCbNightlies"

    LHCb Nightly builds setup and configuration

    Added:
    >
    >
     

    The system expects to find build machines (both server and clients) configured in the way described here: LHCbNightliesPrerequirements.

    Line: 6 to 7
     The system expects to find build machines (both server and clients) configured in the way described here: LHCbNightliesPrerequirements.

    System bootstrap

    Changed:
    <
    <
    Nighty builds are started every night with use of acrontab of lhcbsoft user. Configuration is kept in SVN repository: %SYNTAX{ syntax="python"}% https://svnweb.cern.ch/trac/lhcb/browser/LHCbNightlyConf/trunk/configuration.xml
    >
    >
    Nighty builds are started every night with use of acrontab of lhcbsoft user. Configuration is kept in SVN repository: %SYNTAX{ syntax="python"}% https://svnweb.cern.ch/trac/lhcb/browser/LHCbNightlyConf/trunk/configuration.xml
     %ENDSYNTAX%

    Nightlies summary webpage:

    Changed:
    <
    <
    %SYNTAX{ syntax="python"}% http://cern.ch/lhcb-nightlies
    >
    >
    %SYNTAX{ syntax="python"}% http://cern.ch/lhcb-nightlies
     %ENDSYNTAX%

    Configuration editor:

    Changed:
    <
    <
    %SYNTAX{ syntax="python"}% https://cern.ch/lhcb-nightlies/editor.html
    >
    >
    %SYNTAX{ syntax="python"}% https://cern.ch/lhcb-nightlies/editor.html
     %ENDSYNTAX%

    Server

    Line: 23 to 21
     %ENDSYNTAX%

    Server

    Added:
    >
    >
     Updated version of LHCb Nightlies run in client-server architecture. Server reads configuration file once, while it's started - we start it every night, just after midnight. Configuration is automatically checked out from SVN repository. To be able to start server on the same port the following day, we also kill the server just before the midnight:
    Changed:
    <
    <
    %SYNTAX{ syntax="python"}% 58 23 * * * lxbuild135 ~kkruzele/public/killtree.py kill `ps xf | grep "nightliesServer.sh" | grep -v "grep" | awk '{print $1}'` 20 0 * * * lxbuild135 ~/bin/nightliesServer.sh
    >
    >
    %SYNTAX{ syntax="python"}% 58 23 * * * lxbuild135 ~kkruzele/public/killtree.py kill `ps xf | grep "nightliesServer.sh" | grep -v "grep" | awk '{print $1}'` 20 0 * * * lxbuild135 ~/bin/nightliesServer.sh
     %ENDSYNTAX%

    Clients

    Changed:
    <
    <
    Client must be given at least one parameter on the command line - it's the CMTCONFIG which it supposed to be running with. After startup, each client connects to the server and gets one (slot,platform) pair which matches given CMTCONFIG (dbg platforms can be built with opt cmtconfigs...). When it finished the build, it takes another (slot,platform) pair as long as there is still one ready to built. If not, the client quits the loop. Some platforms are waiting for specific conditions to be fulfilled - if so, they may not be taken by the client when the client is ready. In that case, it may happen that the client finishes its work, and there are still platforms to be built. Therefore there should be more than one lients started during the night - some of them maybe even in the afternoon. %SYNTAX{ syntax="python"}% 7 1 * * * lxbuild111 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh slc4_amd64_gcc34 | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild111_1_`date | awk '{print $1}'`_mainlog.txt 8 1 * * * lxbuild135 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh x86_64-slc5-gcc43-opt | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild135_1_`date | awk '{print $1}'`_mainlog.txt 9 1 * * * lxbuild156 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh slc4_ia32_gcc34 | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild156_1_`date | awk '{print $1}'`_mainlog.txt 10 1 * * * volhcb27 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh x86_64-slc5-gcc43-opt | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild157_1_`date | awk '{print $1}'`_mainlog.txt
    >
    >
    Client must be given at least one parameter on the command line - it's the CMTCONFIG which it supposed to be running with. After startup, each client connects to the server and gets one (slot,platform) pair which matches given CMTCONFIG (dbg platforms can be built with opt cmtconfigs...). When it finished the build, it takes another (slot,platform) pair as long as there is still one ready to built. If not, the client quits the loop. Some platforms are waiting for specific conditions to be fulfilled - if so, they may not be taken by the client when the client is ready. In that case, it may happen that the client finishes its work, and there are still platforms to be built. Therefore there should be more than one lients started during the night - some of them maybe even in the afternoon. %SYNTAX{ syntax="python"}% 7 1 * * * lxbuild111 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh slc4_amd64_gcc34 | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild111_1_`date | awk '{print $1}'`_mainlog.txt 8 1 * * * lxbuild135 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh x86_64-slc5-gcc43-opt | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild135_1_`date | awk '{print $1}'`_mainlog.txt 9 1 * * * lxbuild156 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh slc4_ia32_gcc34 | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild156_1_`date | awk '{print $1}'`_mainlog.txt 10 1 * * * volhcb27 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh x86_64-slc5-gcc43-opt | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild157_1_`date | awk '{print $1}'`_mainlog.txt
     %ENDSYNTAX%

    Summary web page generation

    Changed:
    <
    <
    %SYNTAX{ syntax="python"}% */15 * * * * lxplus export AFSROOT=/afs && /afs/cern.ch/user/l/lhcbsoft/PRODUCTION/prod-LCG/LHCbGenNightlies.py /afs/cern.ch/lhcb/software/nightlies/www/index-LHCb-cache.html
    >
    >
    %SYNTAX{ syntax="python"}% */15 * * * * lxplus export AFSROOT=/afs && /afs/cern.ch/user/l/lhcbsoft/PRODUCTION/prod-LCG/LHCbGenNightlies.py /afs/cern.ch/lhcb/software/nightlies/www/index-LHCb-cache.html
     %ENDSYNTAX%

    Cleaning steps

    AFS

    Changed:
    <
    <
    %SYNTAX{ syntax="python"}% 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-branches/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-patches/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-prerelease/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-head/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-gaudi-head/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-lcg-head/`date | awk '{print $1}'`/*
    >
    >
    %SYNTAX{ syntax="python"}% 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-branches/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-patches/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-prerelease/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-head/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-gaudi-head/`date | awk '{print $1}'`/* 5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-lcg-head/`date | awk '{print $1}'`/*
     %ENDSYNTAX%

    Log files

    Changed:
    <
    <
    %SYNTAX{ syntax="python"}% 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-gaudi-head.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-head-windows.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-lcg-head.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-branches.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-patches.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-prerelease.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-head.`date | awk '{print $1}'`_*
    >
    >
    %SYNTAX{ syntax="python"}% 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-gaudi-head.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-head-windows.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-lcg-head.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-branches.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-patches.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-prerelease.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-head.`date | awk '{print $1}'`_*
     %ENDSYNTAX%

    /build directories

    Changed:
    <
    <
    %SYNTAX{ syntax="python"}% 10 0 * * * lxbuild111 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'` 10 0 * * * lxbuild120 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'` 10 0 * * * lxbuild135 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'` 10 0 * * * lxbuild156 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'` 10 0 * * * lxbuild157 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'`
    >
    >
    %SYNTAX{ syntax="python"}% 10 0 * * * lxbuild111 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'` 10 0 * * * lxbuild120 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'` 10 0 * * * lxbuild135 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'` 10 0 * * * lxbuild156 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'` 10 0 * * * lxbuild157 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'`
     %ENDSYNTAX%

    Configuration

    current configuration file is taken from SVN repository (not anymore from AFS):

    unless you are sure what you are doing, to edit configuration use only the web editor:

    Changed:
    <
    <
    %SYNTAX{ syntax="python"}% https://cern.ch/lhcb-nightlies/editor.html
    >
    >
    %SYNTAX{ syntax="python"}% https://cern.ch/lhcb-nightlies/editor.html
     %ENDSYNTAX%

    xml file format:

    Changed:
    <
    <
    %SYNTAX{ syntax="python"}% < ... >
    <!-- slot contents -->
    < ... >
    >
    >
    %SYNTAX{ syntax="python"}% < ... >
    <!-- slot contents -->
    < ... >
     %ENDSYNTAX%

    configuration/general/parameters currently available:

    Changed:
    <
    <
    %SYNTAX{ syntax="python"}%
    >
    >
    %SYNTAX{ syntax="python"}%
     %ENDSYNTAX%
    • wwwurl, logurl, rrdurl - URLs to find the files (index.html, log files, test statistic graphs)
    • wwwtitle - header of the Nightlies summary page
    Line: 159 to 79
     
    • parameter - 'name' and 'value' are obligatory
    • parameters - The tag 'parameters' must exist but may be empty
    Deleted:
    <
    <
     

    configuration/general/ignore

    Added:
    >
    >
     Each line of the log files is compared to all of the given error/warning statements and if any of error/warning values can be found in the line, the line is not marked as one containing an error/warning. Errors and warings are considered separately, but the values are common for all of the slots. Regular expressions (type="regex") or fnmat/glob expressions (type="fnmatch") can be used here. In the case of regex - remember about .* in the beginning and the end of the expression (unless you are sure what you are doing).
    Changed:
    <
    <
    %SYNTAX{ syntax="python"}%
    >
    >
    %SYNTAX{ syntax="python"}%
     %ENDSYNTAX%

    Schema constraints:

    Line: 175 to 91
     
    • ignore - The tag must exist but may be empty, order is errors first and then warnings

    configuration/general/mailto

    Added:
    >
    >
     All of the projects which are used in one or more slots must have a section here as follows:
    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <project name="lhcb">
        <mail address="marco.cattaneo@cern.ch" edit="false" parent="false" />
        <repository path="isscvs.cern.ch:/local/reps/lhcb" type="cvs"/>
    </project>
    <!-- end SyntaxHighlightingPlugin -->
    Name of the project should be given as they are (LHCb and DaVinci, not LHCB, DAVINCI). mail tag is used to send summary of the builds to the responsible person after the build is finished. The section can contain none, one, or more mail tags. edit and parent attributes are not in use at the moment. repository tag is not in use at the moment.
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    <project name="lhcb"> <mail address="marco.cattaneo@cern.ch" edit="false" parent="false" /> <repository path="isscvs.cern.ch:/local/reps/lhcb" type="cvs"/> </project> 
    <!-- end SyntaxHighlightingPlugin -->
    Name of the project should be given as they are (LHCb and DaVinci, not LHCB, DAVINCI). mail tag is used to send summary of the builds to the responsible person after the build is finished. The section can contain none, one, or more mail tags. edit and parent attributes are not in use at the moment. repository tag is not in use at the moment.
      Schema constraints:
    • mail - 'address' is obligatory
    Line: 190 to 102
     
    • project - 'name' is obligatory, order is mails first and then repositories. Must be configured for every project which is used in one or more of the slots, can be empty (no mail, repository tags)
    • mailto - this tag must exist
    Deleted:
    <
    <
     

    configuration/general/builders

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <builders>
        <project name="Geant4" path="%LHCBRELEASES%/../nightlies/plugins/Geant4" />
    </builders>
    <!-- end SyntaxHighlightingPlugin -->
    In this section it can be configured which builders should be copied from the given location instead of being generated automatically (environment variables can be used as shown - case sensitive).
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    <builders> <project name="Geant4" path="%LHCBRELEASES%/../nightlies/plugins/Geant4" /> </builders> 
    <!-- end SyntaxHighlightingPlugin -->
    In this section it can be configured which builders should be copied from the given location instead of being generated automatically (environment variables can be used as shown - case sensitive).
      Schema constraints:
    • project - 'path' and 'name' are obligatory
    Line: 212 to 120
     
    • Environment variable can be used in the same way - between two percent characters (case sensitive)

    main slot settings

    Changed:
    <
    <
    %SYNTAX{ syntax="python"}%
    >
    >
    %SYNTAX{ syntax="python"}%
     %ENDSYNTAX%
    • name - to create a new slot AFS space has to be allocated for releasedir. lhcb[1-6] are production slots, lhcb-test[1-2] are test slots started from different (not lhcbsoft) account.
    • description - extra information about the slot
    Line: 242 to 151
     
    • projects obligatory

    paths configuration

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <paths>
        <path name="builddir" value="/build/nightlies/%SLOT%/%DAY%" />
        <path name="buildersdir" value="/build/builders/%SLOT%" />
        <path name="releasedir" value="/afs/cern.ch/lhcb/software/nightlies/%SLOT%/%DAY%" />
        <path name="wwwdir" value="/afs/cern.ch/lhcb/software/nightlies/www/logs" />
    </paths>
    <!-- end SyntaxHighlightingPlugin -->
    all paths in this section are obligatory
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    <paths> <path name="builddir" value="/build/nightlies/%SLOT%/%DAY%" /> <path name="buildersdir" value="/build/builders/%SLOT%" /> <path name="releasedir" value="/afs/cern.ch/lhcb/software/nightlies/%SLOT%/%DAY%" /> <path name="wwwdir" value="/afs/cern.ch/lhcb/software/nightlies/www/logs" /> </paths> 
    <!-- end SyntaxHighlightingPlugin -->
    all paths in this section are obligatory
     
    • builddir - local build directory (must exist in filesystem and write permission has to be set for lhcbsoft user)
    • buildersdir - local directory for storing building scripts (plugins which in case of LHCb software are common, generated automatically)
    Line: 261 to 164
     
    • path - 'value' and 'name' are obligatory

    CMTPROJECTPATH

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <cmtprojectpath>
        <path value="/afs/cern.ch/user/m/marcocle/public" />
        <path value="/afs/cern.ch/sw/Gaudi/releases" />
        <path value="/afs/cern.ch/sw/lcg/app/nightlies/dev1/%DAY%" />
        <path value="/afs/cern.ch/sw/lcg/app/releases" />
        <path value="/afs/cern.ch/lhcb/software/releases" />
    </cmtprojectpath>
    <!-- end SyntaxHighlightingPlugin -->
    CMTPROJECTPATH variable, order is important, local build directory is prepended.
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    <cmtprojectpath> <path value="/afs/cern.ch/user/m/marcocle/public" /> <path value="/afs/cern.ch/sw/Gaudi/releases" /> <path value="/afs/cern.ch/sw/lcg/app/nightlies/dev1/%DAY%" /> <path value="/afs/cern.ch/sw/lcg/app/releases" /> <path value="/afs/cern.ch/lhcb/software/releases" /> </cmtprojectpath> 
    <!-- end SyntaxHighlightingPlugin -->
    CMTPROJECTPATH variable, order is important, local build directory is prepended.
      Schema constraints:
    • path - 'value' is obligatory

    CMTEXTRATAGS

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <cmtextratags value="use-distcc" />
    <!-- end SyntaxHighlightingPlugin -->
    Setup of CMTEXTRATAGS value.
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    <cmtextratags value="use-distcc" /> 
    <!-- end SyntaxHighlightingPlugin -->
    Setup of CMTEXTRATAGS value.
      Schema constraints:
    • cmtextratags - 'value' is obligatory

    "wait for" flag

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <waitfor flag="/afs/cern.ch/sw/lcg/app/nightlies/dev1/%DAY%/isDone-%PLATFORM%" />
    <!-- end SyntaxHighlightingPlugin -->
    If waitfor flag is configured, compilation of each platform (see below) is not started until the file is available in given location.
    <!-- SyntaxHighlightingPlugin -->
    <platforms>
        <platform name="slc4_amd64_gcc34_dbg" />
        <platform name="x86_64-slc5-gcc34-dbg" />
    </platforms>
    <!-- end SyntaxHighlightingPlugin -->
    This section defines platforms to be build in the slot. If wait for is used, this is the order the files (isDone files) are looked for. The one which is found (if they are different for each of the platforms) makes the system start checkout and build stages for the platform it describes. If wait for is not used - this is the order of platforms to be build. if any platform is specified as a parameter for the starting script (lhcbNightlies-????.sh), only the platforms found there will be taken into consideration for the machine.
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    <waitfor flag="/afs/cern.ch/sw/lcg/app/nightlies/dev1/%DAY%/isDone-%PLATFORM%" /> 
    <!-- end SyntaxHighlightingPlugin -->
    If waitfor flag is configured, compilation of each platform (see below) is not started until the file is available in given location.
    <!-- SyntaxHighlightingPlugin -->
    <platforms> <platform name="slc4_amd64_gcc34_dbg" /> <platform name="x86_64-slc5-gcc34-dbg" /> </platforms> 
    <!-- end SyntaxHighlightingPlugin -->
    This section defines platforms to be build in the slot. If wait for is used, this is the order the files (isDone files) are looked for. The one which is found (if they are different for each of the platforms) makes the system start checkout and build stages for the platform it describes. If wait for is not used - this is the order of platforms to be build. if any platform is specified as a parameter for the starting script (lhcbNightlies-????.sh), only the platforms found there will be taken into consideration for the machine.
      Schema constraints:
    • waitfor - 'flag' is obligatory
    Line: 303 to 192
     
    • platforms - must contain at least one platform tag

    LbLogin settings

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <lblogin linux="source /afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/scripts/LbLogin.sh --no-cache --quiet --cmtvers=v1r20p20090520 ; export PATH=$PATH:/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/LBSCRIPTS_v4r1/InstallArea/scripts ; export VERBOSE=1" />
    <lblogin win="call %HOMEDRIVE%/local/lib/LbLogin.bat" />
    <!-- end SyntaxHighlightingPlugin -->
    Should be used only for setting up different login environment. The given string is prepanded to the make cmt action inside builders, and it's launched just before the cmt action (in the same command).
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    <lblogin linux="source /afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/scripts/LbLogin.sh --no-cache --quiet --cmtvers=v1r20p20090520 ; export PATH=$PATH:/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/LBSCRIPTS_v4r1/InstallArea/scripts ; export VERBOSE=1" /> <lblogin win="call %HOMEDRIVE%/local/lib/LbLogin.bat" /> 
    <!-- end SyntaxHighlightingPlugin -->
    Should be used only for setting up different login environment. The given string is prepanded to the make cmt action inside builders, and it's launched just before the cmt action (in the same command).
      Schema constraints:
    • lblogin - no constraints, but should define at least 'linux' or 'win' or 'mac'

    runbefore/runafter

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <runbefore linux="any bash command && sequence of commands" win="any windows shell command" />
    <runafter linux="any bash command && sequence of commands" win="any windows shell command"/>
    <!-- end SyntaxHighlightingPlugin -->
    Commands to run before/after the run action is started.
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    <runbefore linux="any bash command && sequence of commands" win="any windows shell command" /> <runafter linux="any bash command && sequence of commands" win="any windows shell command"/> 
    <!-- end SyntaxHighlightingPlugin -->
    Commands to run before/after the run action is started.
      Schema constraints:
    • runbefore/runafter - no constraints, but should define at least 'linux' or 'win' or 'mac'
    Line: 322 to 207
     Schema constraints:
    • runbefore/runafter - no constraints, but should define at least 'linux' or 'win' or 'mac'
    Deleted:
    <
    <
     

    days

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <days mon="true" tue="true" wed="true" thu="true" fri="true" sat="true" sun="true" />
    <!-- end SyntaxHighlightingPlugin -->
    Setting any of the weekdays to false will make the Nightlies not run this day.
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    <days mon="true" tue="true" wed="true" thu="true" fri="true" sat="true" sun="true" /> 
    <!-- end SyntaxHighlightingPlugin -->
    Setting any of the weekdays to false will make the Nightlies not run this day.
      Schema constraints:
    • days - 'mon', 'tue', 'wed', 'thu', 'fri', 'sat', 'sun' are obligatory

    list of projects

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <projects>
        <project name="Gaudi" tag="GAUDI_v21r0">
            <dependence project="lcgcmt" tag="LCGCMT_56" />
        </project>
        <project name="LHCb" tag="LHCB_HEAD" headofeverything="true">
        </project>
        <project name="Lbcom" tag="LBCOM_HEAD" headofeverything="true" />
        <project name="Gauss" tag="GAUSS_HEAD" headofeverything="true">
            <dependence project="geant4" tag="GEANT4_v91r2p1" />
        </project>
        <project name="Panoptes" tag="PANOPTES_HEAD" headofeverything="true" />
    </projects>
    <!-- end SyntaxHighlightingPlugin -->
    List of projects to be build in the slot; order is important. For each of the projects the following keys are obligatory:
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    <projects> <project name="Gaudi" tag="GAUDI_v21r0"> <dependence project="lcgcmt" tag="LCGCMT_56" /> </project> <project name="LHCb" tag="LHCB_HEAD" headofeverything="true"> </project> <project name="Lbcom" tag="LBCOM_HEAD" headofeverything="true" /> <project name="Gauss" tag="GAUSS_HEAD" headofeverything="true"> <dependence project="geant4" tag="GEANT4_v91r2p1" /> </project> <project name="Panoptes" tag="PANOPTES_HEAD" headofeverything="true" /> </projects> 
    <!-- end SyntaxHighlightingPlugin -->
    List of projects to be build in the slot; order is important. For each of the projects the following keys are obligatory:
     
    • name - real name of the project (LHCb, DaVinci, not: LHCB, davinci)
    • tag
    Added:
    >
    >
     Additional keys (not obligatory):
    • headofeverything - if set to true version of all the packages from project container requirements file will be changed to head.
    • docs - Doxygen documentation will be generated after build of the project, if set to true
    Line: 359 to 231
     
    • projects - May be empty (no project tags)

    tuning the project configuration

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    <project name="Gauss" tag="GAUSS_HEAD" headofeverything="true">
            <dependence project="geant4" tag="GEANT4_v91r2p1" />
        </project>
        <project name="LHCb" tag="LHCB_HEAD" headofeverything="true">
            <dependence project="gaudi" tag="GAUDI_HEAD" />
            <change package="Phys/LoKiArrayFunctors" value="vanya_20090424"/>
            <change package="Det/DetDescSvc" value="v2r2"/>
        </project>
    <!-- end SyntaxHighlightingPlugin -->
    The following can be set additionally for each of the projects in the slot:
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    <project name="Gauss" tag="GAUSS_HEAD" headofeverything="true"> <dependence project="geant4" tag="GEANT4_v91r2p1" /> </project> <project name="LHCb" tag="LHCB_HEAD" headofeverything="true"> <dependence project="gaudi" tag="GAUDI_HEAD" /> <change package="Phys/LoKiArrayFunctors" value="vanya_20090424"/> <change package="Det/DetDescSvc" value="v2r2"/> </project> 
    <!-- end SyntaxHighlightingPlugin -->
    The following can be set additionally for each of the projects in the slot:
     
    • dependence - modify the project.cmt file to use different project version (for projects one depends on). IMPORTANT: for the time being, while setting the dependence, lowercase project names should be used. Usage of the real project names must be tested.
    • change - change the specified package version; it overwrites headofeverything ("headofeverything but...").
    Line: 398 to 262
     
    • if only some platforms were broken, check the machines they where started on (lxbuild111/120/135) (for example consulting acrontab) - the process can be restarted only for the selected machines by performing: clean, and run actions for the given slot on the build server. *do not launch cleanAFS action unless you want to restart the slot on all platforms. IMPORTANT: if you do not clean AFS release area, the configuration.xml file from the night will be taken as a configuration for the restarted platforms.

    How to restart the Nightlies

    Changed:
    <
    <
    Production slots of the Nightlies run from lhcbsoft account. One has to login as lhcbsoft to be able to restart the Nightlies.
    >
    >
    Production slots of the Nightlies run from lhcbsoft account. One has to login as lhcbsoft to be able to restart the Nightlies.
     

    only specific (platform, slot) pair

    Added:
    >
    >
     If you skip the optional steps, old log files will be still visible on the summary webpage and they will be overwritten one by one, while the process go on. It may confuse people checking the webpage.
    • the platform you want to restart may not be still running (the build must be stopped in that case before going on - you may use killtree LHCb script)
    • (optional) copy and remove the old log files for a specific platform/slot from /afs/cern.ch/lhcb/software/nightlies/www/logs/ For example these files: lhcb-head.Wed_*x86_64-slc5-gcc43-opt*
    Line: 406 to 272
     
    • (optional) remove the isDone file for the slot/platform from AFS slot directory (for example: /afs/cern.ch/lhcb/software/nightlies/lhcb-head/Wed/isDone-x86_64-slc5-gcc43-opt)
    • remove local build files for the specific slot/platform (for example: /build/nightlies/lhcb-head/Wed/x86_64-slc5-gcc43-opt)
    • start (you may consider using screen for that): ~/bin/nightliesClient.sh x86_64-slc5-gcc43-opt lhcb-head. You should check if you are using the build server (slc4/slc5) you really intend to use.
    Added:
    >
    >

    all platforms in one slot

     
    Added:
    >
    >
    • for the moment, the only solution is to repeat the procedure for "only specific (platform, slot) pair" from above
     

    all platforms in all slots

    • make sure that all the nightliesClient s on all build servers are stoped
    • stop the nightliesServer (usually running on lxbuild135)
    Line: 412 to 280
     
    • stop the nightliesServer (usually running on lxbuild135)
    • start a new nightliesServer
    • start one or more nightliesClient s
    Added:
    >
    >
     nightliiesServer by default takes configuration from SVN, the last one from yesterday. If you want to restart the Nightlies because of the chacnge in configuration - you have to force the server to take another (for example recent) configuration revision.
    Added:
    >
    >

    how to use not default (last from yesterday SVN) configuration

    Configuration is handled by nightliesServer. If you want to use other configuration source:

    • local file
    • URL
    • SVN, but not last from yesterday

    you need another nightliesServer instance (another port number) and dedicated nightliesClients to connect to.

     
    Added:
    >
    >
    For the moment the easiest solution is to use nightliesServer-restart script, and nightliesClient-restart script, modifying all necessary parameters (source of configuraton) in the script itself.
     
    META TOPICMOVED by="KarolKruzelecki" date="1247756095" from="Main.LHCbNightliesConfiguration" to="LHCb.LHCbNightliesConfiguration"

    Revision 82010-11-03 - unknown

    Line: 1 to 1
     
    META TOPICPARENT name="LHCbNightlies"

    LHCb Nightly builds setup and configuration

    Line: 78 to 78
     

    Configuration

    current configuration file is taken from SVN repository (not anymore from AFS):

    Changed:
    <
    <

    unless you are sure what you are doing, to edit configuration use only the web editor:

    >
    >
    unless you are sure what you are doing, to edit configuration use only the web editor:
     
    <!-- SyntaxHighlightingPlugin -->
    https://cern.ch/lhcb-nightlies/editor.html
    <!-- end SyntaxHighlightingPlugin -->
    Line: 397 to 397
     

    action

    • if only some platforms were broken, check the machines they where started on (lxbuild111/120/135) (for example consulting acrontab) - the process can be restarted only for the selected machines by performing: clean, and run actions for the given slot on the build server. *do not launch cleanAFS action unless you want to restart the slot on all platforms. IMPORTANT: if you do not clean AFS release area, the configuration.xml file from the night will be taken as a configuration for the restarted platforms.
    Changed:
    <
    <

    How to restart the Nightlies (obsolete!)

    >
    >

    How to restart the Nightlies

     Production slots of the Nightlies run from lhcbsoft account. One has to login as lhcbsoft to be able to restart the Nightlies.
    Changed:
    <
    <

    Linux

    • check if the Nightlies for the slot you want to restart are still running (on any of the lxbuild machines) and stop them if so
    • if you want to restart the whole slot (all platforms), run:
      ~/bin/lhcbNightlies-prod.sh cleanAFS <slot>
      This will remove the contents of the slot's AFS directory. If you want to restart only some of the platforms, skip this step.
    • clean the local build directories and make the system checkout all the files again when started.
      ~/bin/lhcbNightlies-prod.sh clean <slot>
      It must be done on each machine you are going to use, once per slot and per machine.
    • start the Nightlies for each machine/slot (it's recommended to use acrontab job, or screen):
      ~/bin/lhcbNightlies-prod.sh run <slot> <-j> <-l> <list_of_platforms_to_build_on_this_machine>
      (details in the top of this page)

    Windows

    • check if the Nightlies for the slot you want to restart are still running (cmd window) and stop them if so (close the window with X)
    • remove the contents of the slot directory on AFS (from Windows, or from Linux):
      /afs/cern.ch/lhcb/software/nightlies/<slot>/<day>
      You can use:
      ~/bin/lhcbNightlies-prod.sh cleanAFS <slot>
      from Linux.
    • check if Kerberberos token is valid, or obtain the new one (Network Identity Manager application)
    • go to: Start -> Control Panel -> Scheduled Tasks
    • run (right click on it, and then Run): "[lhcbsoft] Nightlies - lhcb5" task in case of lhcb5 slot
    • if you restart them during the day: it's better to choose the task with "(10am)" in the name if you start it about 10 - These tasks are the copies of the main one with limited working time to stop them before midnight of the next day (when the new builds should start)
    >
    >

    only specific (platform, slot) pair

    If you skip the optional steps, old log files will be still visible on the summary webpage and they will be overwritten one by one, while the process go on. It may confuse people checking the webpage.
    • the platform you want to restart may not be still running (the build must be stopped in that case before going on - you may use killtree LHCb script)
    • (optional) copy and remove the old log files for a specific platform/slot from /afs/cern.ch/lhcb/software/nightlies/www/logs/ For example these files: lhcb-head.Wed_*x86_64-slc5-gcc43-opt*
    • (optional) remove the isDone file for the slot/platform from AFS slot directory (for example: /afs/cern.ch/lhcb/software/nightlies/lhcb-head/Wed/isDone-x86_64-slc5-gcc43-opt)
    • remove local build files for the specific slot/platform (for example: /build/nightlies/lhcb-head/Wed/x86_64-slc5-gcc43-opt)
    • start (you may consider using screen for that): ~/bin/nightliesClient.sh x86_64-slc5-gcc43-opt lhcb-head. You should check if you are using the build server (slc4/slc5) you really intend to use.

    all platforms in all slots

    • make sure that all the nightliesClient s on all build servers are stoped
    • stop the nightliesServer (usually running on lxbuild135)
    • start a new nightliesServer
    • start one or more nightliesClient s
    nightliiesServer by default takes configuration from SVN, the last one from yesterday. If you want to restart the Nightlies because of the chacnge in configuration - you have to force the server to take another (for example recent) configuration revision.
     

    META TOPICMOVED by="KarolKruzelecki" date="1247756095" from="Main.LHCbNightliesConfiguration" to="LHCb.LHCbNightliesConfiguration"

    Revision 72010-10-05 - unknown

    Line: 1 to 1
     
    META TOPICPARENT name="LHCbNightlies"

    LHCb Nightly builds setup and configuration

    Added:
    >
    >
    The system expects to find build machines (both server and clients) configured in the way described here: LHCbNightliesPrerequirements.
     

    System bootstrap

    Changed:
    <
    <
    Nighty builds are started every night with use of acrontab of lhcbsoft user. Actions are separated, because some of them have to be done once per slot and the others once per slot and per machine.

    main steps

    clean AFS release area of slot (only once per slot)

    <!-- SyntaxHighlightingPlugin -->
    45 0 * * * lxplus.cern.ch ~/bin/lhcbNightlies-prod.sh cleanAFS lhcb2
    <!-- end SyntaxHighlightingPlugin -->

    clean build directories (once per machine, per slot)

    <!-- SyntaxHighlightingPlugin -->
    0 2 * * * lxbuild111 ~/bin/lhcbNightlies-prod.sh clean lhcb2
    0 2 * * * lxbuild120 ~/bin/lhcbNightlies-prod.sh clean lhcb2
    0 2 * * * lxbuild135 ~/bin/lhcbNightlies-prod.sh clean lhcb2
    <!-- end SyntaxHighlightingPlugin -->

    start Nightlies (once per machine, per slot)

    <!-- SyntaxHighlightingPlugin -->
    30 2 * * * lxbuild111 ~/bin/lhcbNightlies-prod.sh run lhcb2 50 4 slc4_ia32_gcc34 slc4_ia32_gcc34_dbg slc4_amd64_gcc43_dbg
    40 2 * * * lxbuild120 ~/bin/lhcbNightlies-prod.sh run lhcb2 50 4 slc4_amd64_gcc34 slc4_amd64_gcc43 slc4_amd64_gcc34_dbg
    10 2 * * * lxbuild135 ~/bin/lhcbNightlies-prod.sh run lhcb2 50 8 x86_64-slc5-gcc34 x86_64-slc5-gcc43 x86_64-slc5-gcc43-opt x86_64-slc5-gcc34-opt x86_64-slc5-gcc43-dbg x86_64-slc5-gcc34-dbg #i686-slc5-gcc34-dbg i686-slc5-gcc34-dbg
    <!-- end SyntaxHighlightingPlugin -->

    Starting points

    • ~/bin/lhcbNightlies-prod.sh - production version of LbScripts (/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod)
    • ~/bin/lhcbNightlies-dev.sh - developement version of LbScripts (/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/dev)
    • ~/bin/lhcbNightlies-user.sh - (lhcbsoft) user version of LbScripts (/afs/cern.ch/user/l/lhcbsoft/cmtuser/LBSCRIPTS/LBSCRIPTS_v99r0)

    "run" action parameters

    • ~/bin/lhcbNightlies-prod.sh run SLOT J_VALUE L_VALUE [PLAT1 [PLAT2 [...]]]
    • SLOT - name of the slot,
    • J_VALUE - -j value for make,
    • L_VALUE - -l value for make,
    • PLAT1 PLAT2 ... - limit the platforms to be build on the given machine to the list given. If no platforms is given for the command, Nightlies will use this server to build all of platforms for the slot specified in configuration.xml

    additional (cleaning) steps

    remove files from the day before yesterday (per machine)

    <!-- SyntaxHighlightingPlugin -->
    12 0 * * * lxbuild111 rm -rf /build/nightlies/lhcb?/`date --date='2 days ago' | awk '{print $1}'`/*
    12 0 * * * lxbuild120 rm -rf /build/nightlies/lhcb?/`date --date='2 days ago' | awk '{print $1}'`/*
    12 0 * * * lxbuild135 rm -rf /build/nightlies/lhcb?/`date --date='2 days ago' | awk '{print $1}'`/*
    <!-- end SyntaxHighlightingPlugin -->

    remove log files from last week (per slot)

    <!-- SyntaxHighlightingPlugin -->
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb1.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb2.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb3.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb4.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb5.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb6.`date | awk '{print $1}'`_*
    <!-- end SyntaxHighlightingPlugin -->
    >
    >
    Nighty builds are started every night with use of acrontab of lhcbsoft user. Configuration is kept in SVN repository:
    <!-- SyntaxHighlightingPlugin -->
    https://svnweb.cern.ch/trac/lhcb/browser/LHCbNightlyConf/trunk/configuration.xml
    <!-- end SyntaxHighlightingPlugin -->
     
    Changed:
    <
    <

    Configuration

    current configuration file is taken from AFS and can be found:

    >
    >
    Nightlies summary webpage:
    <!-- SyntaxHighlightingPlugin -->
    http://cern.ch/lhcb-nightlies
    <!-- end SyntaxHighlightingPlugin -->

    Configuration editor:

     %SYNTAX{ syntax="python"}%
    Changed:
    <
    <
    /afs/cern.ch/lhcb/software/nightlies/conf/configuration.xml
    >
    >
    https://cern.ch/lhcb-nightlies/editor.html
     %ENDSYNTAX%
    Deleted:
    <
    <
    This file is checked out from SVN. Any changes made to it should be comitted bck to SVN
     
    Changed:
    <
    <

    configuration history (from last 7 days):

    >
    >

    Server

    Updated version of LHCb Nightlies run in client-server architecture. Server reads configuration file once, while it's started - we start it every night, just after midnight. Configuration is automatically checked out from SVN repository. To be able to start server on the same port the following day, we also kill the server just before the midnight:
    <!-- SyntaxHighlightingPlugin -->
    58 23 * * * lxbuild135 ~kkruzele/public/killtree.py kill `ps xf | grep "nightliesServer.sh" | grep -v "grep" | awk '{print $1}'`
    20 0 * * * lxbuild135 ~/bin/nightliesServer.sh
    <!-- end SyntaxHighlightingPlugin -->

    Clients

    Client must be given at least one parameter on the command line - it's the CMTCONFIG which it supposed to be running with. After startup, each client connects to the server and gets one (slot,platform) pair which matches given CMTCONFIG (dbg platforms can be built with opt cmtconfigs...). When it finished the build, it takes another (slot,platform) pair as long as there is still one ready to built. If not, the client quits the loop. Some platforms are waiting for specific conditions to be fulfilled - if so, they may not be taken by the client when the client is ready. In that case, it may happen that the client finishes its work, and there are still platforms to be built. Therefore there should be more than one lients started during the night - some of them maybe even in the afternoon.
    <!-- SyntaxHighlightingPlugin -->
    7 1 * * * lxbuild111 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh slc4_amd64_gcc34 | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild111_1_`date | awk '{print $1}'`_mainlog.txt
    8 1 * * * lxbuild135 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh x86_64-slc5-gcc43-opt | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild135_1_`date | awk '{print $1}'`_mainlog.txt
    9 1 * * * lxbuild156 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh slc4_ia32_gcc34 | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild156_1_`date | awk '{print $1}'`_mainlog.txt
    10 1 * * * volhcb27 /afs/cern.ch/user/l/lhcbsoft/bin/nightliesClient.sh x86_64-slc5-gcc43-opt | tee /afs/cern.ch/lhcb/software/nightlies/www/logs/client_lxbuild157_1_`date | awk '{print $1}'`_mainlog.txt
    <!-- end SyntaxHighlightingPlugin -->

    Summary web page generation

    <!-- SyntaxHighlightingPlugin -->
    */15 * * * * lxplus export AFSROOT=/afs && /afs/cern.ch/user/l/lhcbsoft/PRODUCTION/prod-LCG/LHCbGenNightlies.py /afs/cern.ch/lhcb/software/nightlies/www/index-LHCb-cache.html
    <!-- end SyntaxHighlightingPlugin -->

    Cleaning steps

    AFS

    <!-- SyntaxHighlightingPlugin -->
    5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-branches/`date | awk '{print $1}'`/*
    5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-patches/`date | awk '{print $1}'`/*
    5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-prerelease/`date | awk '{print $1}'`/*
    5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-head/`date | awk '{print $1}'`/*
    5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-gaudi-head/`date | awk '{print $1}'`/*
    5 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/lhcb-lcg-head/`date | awk '{print $1}'`/*
    <!-- end SyntaxHighlightingPlugin -->

    Log files

    <!-- SyntaxHighlightingPlugin -->
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-gaudi-head.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-head-windows.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-lcg-head.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-branches.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-patches.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-prerelease.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb-head.`date | awk '{print $1}'`_*
    <!-- end SyntaxHighlightingPlugin -->

    /build directories

    <!-- SyntaxHighlightingPlugin -->
    10 0 * * * lxbuild111 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'`
    10 0 * * * lxbuild120 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'`
    10 0 * * * lxbuild135 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'`
    10 0 * * * lxbuild156 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'`
    10 0 * * * lxbuild157 rm -rf /build/nightlies/lhcb-*/`date | awk '{print $1}'`
    <!-- end SyntaxHighlightingPlugin -->

    Configuration

    current configuration file is taken from SVN repository (not anymore from AFS):

    unless you are sure what you are doing, to edit configuration use only the web editor:

     %SYNTAX{ syntax="python"}%
    Changed:
    <
    <
    /afs/cern.ch/lhcb/software/nightlies/conf/history/..xml -- obsolete, will be removed /afs/cern.ch/lhcb/software/nightlies///configuration.xml (Nighlties release area)
    >
    >
    https://cern.ch/lhcb-nightlies/editor.html
     %ENDSYNTAX%

    xml file format:

    Line: 369 to 397
     

    action

    • if only some platforms were broken, check the machines they where started on (lxbuild111/120/135) (for example consulting acrontab) - the process can be restarted only for the selected machines by performing: clean, and run actions for the given slot on the build server. *do not launch cleanAFS action unless you want to restart the slot on all platforms. IMPORTANT: if you do not clean AFS release area, the configuration.xml file from the night will be taken as a configuration for the restarted platforms.
    Changed:
    <
    <

    How to restart the Nightlies

    >
    >

    How to restart the Nightlies (obsolete!)

     Production slots of the Nightlies run from lhcbsoft account. One has to login as lhcbsoft to be able to restart the Nightlies.

    Linux

    • check if the Nightlies for the slot you want to restart are still running (on any of the lxbuild machines) and stop them if so
    Line: 385 to 413
     
    • run (right click on it, and then Run): "[lhcbsoft] Nightlies - lhcb5" task in case of lhcb5 slot
    • if you restart them during the day: it's better to choose the task with "(10am)" in the name if you start it about 10 - These tasks are the copies of the main one with limited working time to stop them before midnight of the next day (when the new builds should start)
    Deleted:
    <
    <
    -- KarolKruzelecki - 22-Oct-2009
     
    META TOPICMOVED by="KarolKruzelecki" date="1247756095" from="Main.LHCbNightliesConfiguration" to="LHCb.LHCbNightliesConfiguration"

    Revision 62010-08-06 - unknown

    Line: 1 to 1
     
    META TOPICPARENT name="LHCbNightlies"

    LHCb Nightly builds setup and configuration

    Line: 64 to 64
      < ... >
    Deleted:
    <
    <
     
    Added:
    >
    >
     
    Line: 74 to 76
     
    Changed:
    <
    <
    <!-- slot contents -->
    >
    >
     
    Changed:
    <
    <
    >
    >
     
    <!-- slot contents -->
    < ... >
    Line: 100 to 127
     
    • mailsubjectprefix
    • shownotfinishedplatforms - if set to true platforms which are not finished will also be shown on the summary page (isStarted file flags are used then instead of isDone)
    Added:
    >
    >
    Schema constraints:
    • parameter - 'name' and 'value' are obligatory
    • parameters - The tag 'parameters' must exist but may be empty
     

    configuration/general/ignore

    Each line of the log files is compared to all of the given error/warning statements and if any of error/warning values can be found in the line, the line is not marked as one containing an error/warning. Errors and warings are considered separately, but the values are common for all of the slots. Regular expressions (type="regex") or fnmat/glob expressions (type="fnmatch") can be used here. In the case of regex - remember about .* in the beginning and the end of the expression (unless you are sure what you are doing). %SYNTAX{ syntax="python"}%
    Deleted:
    <
    <
     
    Added:
    >
    >
      %ENDSYNTAX%
    Added:
    >
    >
    Schema constraints:
    • error - 'value' is obligatory, 'type' is optional
    • warning - 'value' is obligatory, 'type' is optional
    • ignore - The tag must exist but may be empty, order is errors first and then warnings
     

    configuration/general/mailto

    All of the projects which are used in one or more slots must have a section here as follows: %SYNTAX{ syntax="python"}%
    Line: 116 to 154
      %ENDSYNTAX%
    Changed:
    <
    <
    Name of the project should be given as they are (LHCb and DaVinci, not LHCB, DAVINCI). tag is used to send summary of the builds to the responsible person after the build is finished. The section can contain none, one, or more tags. edit and parent attributes are not in use at the moment. tag is not in use at the moment.
    >
    >
    Name of the project should be given as they are (LHCb and DaVinci, not LHCB, DAVINCI). mail tag is used to send summary of the builds to the responsible person after the build is finished. The section can contain none, one, or more mail tags. edit and parent attributes are not in use at the moment. repository tag is not in use at the moment.

    Schema constraints:

    • mail - 'address' is obligatory
    • repository - 'path' and 'type' are obligatory
    • project - 'name' is obligatory, order is mails first and then repositories. Must be configured for every project which is used in one or more of the slots, can be empty (no mail, repository tags)
    • mailto - this tag must exist
     
    Deleted:
    <
    <
    section: project must be configured for every project which is used in one or more of the slots, can be empty (no mail, repository tags)
     

    configuration/general/builders

    %SYNTAX{ syntax="python"}%
    Line: 128 to 171
     %ENDSYNTAX% In this section it can be configured which builders should be copied from the given location instead of being generated automatically (environment variables can be used as shown - case sensitive).
    Added:
    >
    >
    Schema constraints:
    • project - 'path' and 'name' are obligatory
    • builders - this tag is optional
     

    per slot configuration

    defined values, usage ofenvironmental variables

    • %SLOT% - slot name
    Line: 141 to 188
     
    Added:
    >
    >
    • description - extra information about the slot
     
    • mails - if set to false no summary e-mails will be send after the each of the projects in this slot is finished
    • hidden - if set to false slot will not be visible on the summary page (used by LCG Nightlies for the release slot)
    • computedependencies - if set to false, QUICK=2 is passed to make
    • renice - nice value used for the processes that run nightlies for the slot
    Added:
    >
    >
    Schema constraints:
    • slot - 'name' is obligatory.
    Order of the children tags is :
    • paths obligatory
    • cmtprojectpath obligatory
    • platforms obligatory
    • waitfor optional
    • ...
    • cmtextratags optional
    • ...
    • lblogin optional
    • ...
    • runbefore optional
    • ...
    • runafter optional
    • ...
    • days obligatory
    • projects obligatory
     

    paths configuration

    %SYNTAX{ syntax="python"}%
    Line: 161 to 229
     
    • releasedir - AFS directory to copy to configuration.xml file (Nightlies copy the file once and all the other servers will use exactly the same version of configuration.xml from AFS). The directory is used also for copying the software to, after the build is finished. Must exist and write persmission must be set for lhcbsoft account
    • wwwdir - where to copy html log files
    Added:
    >
    >
    Schema constraints:
    • path - 'value' and 'name' are obligatory
     

    CMTPROJECTPATH

    %SYNTAX{ syntax="python"}%
    Line: 173 to 244
     %ENDSYNTAX% CMTPROJECTPATH variable, order is important, local build directory is prepended.
    Added:
    >
    >
    Schema constraints:
    • path - 'value' is obligatory
     

    CMTEXTRATAGS

    <!-- SyntaxHighlightingPlugin -->
    <cmtextratags value="use-distcc" />
    <!-- end SyntaxHighlightingPlugin -->
    Setup of CMTEXTRATAGS value.
    Added:
    >
    >
    Schema constraints:
    • cmtextratags - 'value' is obligatory
     

    "wait for" flag

    %SYNTAX{ syntax="python"}%
    Line: 192 to 269
     %ENDSYNTAX% This section defines platforms to be build in the slot. If wait for is used, this is the order the files (isDone files) are looked for. The one which is found (if they are different for each of the platforms) makes the system start checkout and build stages for the platform it describes. If wait for is not used - this is the order of platforms to be build. if any platform is specified as a parameter for the starting script (lhcbNightlies-????.sh), only the platforms found there will be taken into consideration for the machine.
    Added:
    >
    >
    Schema constraints:
    • waitfor - 'flag' is obligatory
    • platform - 'name' is obligatory
    • platforms - must contain at least one platform tag
     

    LbLogin settings

    %SYNTAX{ syntax="python"}%
    Line: 199 to 281
     %ENDSYNTAX% Should be used only for setting up different login environment. The given string is prepanded to the make cmt action inside builders, and it's launched just before the cmt action (in the same command).
    Added:
    >
    >
    Schema constraints:
    • lblogin - no constraints, but should define at least 'linux' or 'win' or 'mac'
     

    runbefore/runafter

    %SYNTAX{ syntax="python"}%
    Line: 206 to 291
     %ENDSYNTAX% Commands to run before/after the run action is started.
    Added:
    >
    >
    Schema constraints:
    • runbefore/runafter - no constraints, but should define at least 'linux' or 'win' or 'mac'
     

    days

    <!-- SyntaxHighlightingPlugin -->
    <days mon="true" tue="true" wed="true" thu="true" fri="true" sat="true" sun="true" />
    <!-- end SyntaxHighlightingPlugin -->
    Setting any of the weekdays to false will make the Nightlies not run this day.
    Added:
    >
    >
    Schema constraints:
    • days - 'mon', 'tue', 'wed', 'thu', 'fri', 'sat', 'sun' are obligatory
     

    list of projects

    %SYNTAX{ syntax="python"}%
    Line: 234 to 326
     
    • headofeverything - if set to true version of all the packages from project container requirements file will be changed to head.
    • docs - Doxygen documentation will be generated after build of the project, if set to true
    Added:
    >
    >
    Schema constraints:
    • project - 'name' and 'tag' are obligatory
    • projects - May be empty (no project tags)
     

    tuning the project configuration

    %SYNTAX{ syntax="python"}%
    Line: 249 to 345
     
    • dependence - modify the project.cmt file to use different project version (for projects one depends on). IMPORTANT: for the time being, while setting the dependence, lowercase project names should be used. Usage of the real project names must be tested.
    • change - change the specified package version; it overwrites headofeverything ("headofeverything but...").
    Added:
    >
    >
    Schema constraints:
    • dependence - 'project' and 'tag' are obligatory
    • change - 'package' and 'value' are obligatory
    • project - order is dependences first and then changes
     

    Troubleshooting

    Logs

    • each acrontab job output is sent to the lhcbsoft e-mail account unless output is empty. In the case of LHCb Nightlies there should be an e-mail for every job because some diagnostic information is always printed. This log contains checkout log which is not stored in any separate file. Build output is not printed to this log.

    Revision 52010-05-14 - MarcoCattaneo

    Line: 1 to 1
     
    META TOPICPARENT name="LHCbNightlies"

    LHCb Nightly builds setup and configuration

    Line: 47 to 47
     
    <!-- SyntaxHighlightingPlugin -->
    /afs/cern.ch/lhcb/software/nightlies/conf/configuration.xml
    <!-- end SyntaxHighlightingPlugin -->
    Added:
    >
    >
    This file is checked out from SVN. Any changes made to it should be comitted bck to SVN
     

    configuration history (from last 7 days):

    %SYNTAX{ syntax="python"}%

    Revision 42009-10-22 - unknown

    Line: 1 to 1
     
    META TOPICPARENT name="LHCbNightlies"

    LHCb Nightly builds setup and configuration

    Line: 6 to 6
     

    System bootstrap

    Nighty builds are started every night with use of acrontab of lhcbsoft user. Actions are separated, because some of them have to be done once per slot and the others once per slot and per machine.

    main steps

    Changed:
    <
    <

    clean AFS release area of slot (only once per slot)

    >
    >

    clean AFS release area of slot (only once per slot)

     
    <!-- SyntaxHighlightingPlugin -->
    45 0 * * * lxplus.cern.ch ~/bin/lhcbNightlies-prod.sh cleanAFS lhcb2
    <!-- end SyntaxHighlightingPlugin -->
    Changed:
    <
    <

    clean build directories (once per machine, per slot)

    >
    >

    clean build directories (once per machine, per slot)

     
    <!-- SyntaxHighlightingPlugin -->
    0 2 * * * lxbuild111 ~/bin/lhcbNightlies-prod.sh clean lhcb2
    0 2 * * * lxbuild120 ~/bin/lhcbNightlies-prod.sh clean lhcb2
    0 2 * * * lxbuild135 ~/bin/lhcbNightlies-prod.sh clean lhcb2
    <!-- end SyntaxHighlightingPlugin -->
    Changed:
    <
    <

    start Nightlies (once per machine, per slot)

    >
    >

    start Nightlies (once per machine, per slot)

     %SYNTAX{ syntax="python"}%30 2 * * * lxbuild111 ~/bin/lhcbNightlies-prod.sh run lhcb2 50 4 slc4_ia32_gcc34 slc4_ia32_gcc34_dbg slc4_amd64_gcc43_dbg 40 2 * * * lxbuild120 ~/bin/lhcbNightlies-prod.sh run lhcb2 50 4 slc4_amd64_gcc34 slc4_amd64_gcc43 slc4_amd64_gcc34_dbg
    Changed:
    <
    <
    10 2 * * * lxbuild135 ~/bin/lhcbNightlies-prod.sh run lhcb2 50 8 x86_64-slc5-gcc34 x86_64-slc5-gcc43 x86_64-slc5-gcc43-opt x86_64-slc5-gcc34-opt x86_64-slc5-gcc43-dbg x86_64-slc5-gcc34-dbg Main.#i686-slc5-gcc34-dbg i686-slc5-gcc34-dbg%ENDSYNTAX%
    >
    >
    10 2 * * * lxbuild135 ~/bin/lhcbNightlies-prod.sh run lhcb2 50 8 x86_64-slc5-gcc34 x86_64-slc5-gcc43 x86_64-slc5-gcc43-opt x86_64-slc5-gcc34-opt x86_64-slc5-gcc43-dbg x86_64-slc5-gcc34-dbg #i686-slc5-gcc34-dbg i686-slc5-gcc34-dbg%ENDSYNTAX%
     

    Starting points

    Changed:
    <
    <
    • ~/bin/lhcbNightlies-prod.sh - production version of LbMain.Scripts (/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod)
    • ~/bin/lhcbNightlies-dev.sh - developement version of LbMain.Scripts (/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/dev)
    • ~/bin/lhcbNightlies-user.sh - (lhcbsoft) user version of LbMain.Scripts (/afs/cern.ch/user/l/lhcbsoft/cmtuser/LBSCRIPTS/LBSCRIPTS_v99r0)
    >
    >
    • ~/bin/lhcbNightlies-prod.sh - production version of LbScripts (/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod)
    • ~/bin/lhcbNightlies-dev.sh - developement version of LbScripts (/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/dev)
    • ~/bin/lhcbNightlies-user.sh - (lhcbsoft) user version of LbScripts (/afs/cern.ch/user/l/lhcbsoft/cmtuser/LBSCRIPTS/LBSCRIPTS_v99r0)
     

    "run" action parameters

    • ~/bin/lhcbNightlies-prod.sh run SLOT J_VALUE L_VALUE [PLAT1 [PLAT2 [...]]]
    Line: 97 to 97
     
    • wwwtitle - header of the Nightlies summary page
    • mailfrom - (cern) e-mail address - sender of the build/test results
    • mailsubjectprefix
    Changed:
    <
    <
    • shownotfinishedplatforms - if set to true platforms whiMain.ch are not finished will also be shown on the summary page (isStarted file flags are used then instead of isDone)
    >
    >
    • shownotfinishedplatforms - if set to true platforms which are not finished will also be shown on the summary page (isStarted file flags are used then instead of isDone)
     

    configuration/general/ignore

    Changed:
    <
    <
    EaMain.ch line of the log files is compared to all of the given error/warning statements and if any of error/warning values can be found in the line, the line is not marked as one containing an error/warning. Errors and warings are considered separately, but the values are common for all of the slots. Regular expressions (type="regex") or fnmatMain.ch/glob expressions (type="fnmatMain.ch") can be used here. In the case of regex - remember about .* in the beginning and the end of the expression (unless you are sure what you are doing).
    >
    >
    Each line of the log files is compared to all of the given error/warning statements and if any of error/warning values can be found in the line, the line is not marked as one containing an error/warning. Errors and warings are considered separately, but the values are common for all of the slots. Regular expressions (type="regex") or fnmat/glob expressions (type="fnmatch") can be used here. In the case of regex - remember about .* in the beginning and the end of the expression (unless you are sure what you are doing).
     %SYNTAX{ syntax="python"}%
    Line: 108 to 108
     %ENDSYNTAX%

    configuration/general/mailto

    Changed:
    <
    <
    All of the projects whiMain.ch are used in one or more slots must have a section here as follows:
    >
    >
    All of the projects which are used in one or more slots must have a section here as follows:
     %SYNTAX{ syntax="python"}%
    Line: 117 to 117
     %ENDSYNTAX% Name of the project should be given as they are (LHCb and DaVinci, not LHCB, DAVINCI). tag is used to send summary of the builds to the responsible person after the build is finished. The section can contain none, one, or more tags. edit and parent attributes are not in use at the moment. tag is not in use at the moment.
    Changed:
    <
    <
    section: project must be configured for every project whiMain.ch is used in one or more of the slots, can be empty (no mail, repository tags)
    >
    >
    section: project must be configured for every project which is used in one or more of the slots, can be empty (no mail, repository tags)
     

    configuration/general/builders

    %SYNTAX{ syntax="python"}%
    Line: 125 to 125
      %ENDSYNTAX%
    Changed:
    <
    <
    In this section it can be configured whiMain.ch builders should be copied from the given location instead of being generated automatically (environment variables can be used as shown - case sensitive).
    >
    >
    In this section it can be configured which builders should be copied from the given location instead of being generated automatically (environment variables can be used as shown - case sensitive).
     

    per slot configuration

    defined values, usage ofenvironmental variables

    Line: 140 to 140
     
    Changed:
    <
    <
    • mails - if set to false no summary e-mails will be send after the eaMain.ch of the projects in this slot is finished
    >
    >
    • mails - if set to false no summary e-mails will be send after the each of the projects in this slot is finished
     
    • hidden - if set to false slot will not be visible on the summary page (used by LCG Nightlies for the release slot)
    • computedependencies - if set to false, QUICK=2 is passed to make
    • renice - nice value used for the processes that run nightlies for the slot
    Line: 156 to 156
     all paths in this section are obligatory

    • builddir - local build directory (must exist in filesystem and write permission has to be set for lhcbsoft user)
    Changed:
    <
    <
    • buildersdir - local directory for storing building scripts (plugins whiMain.ch in case of LHCb software are common, generated automatically)
    >
    >
    • buildersdir - local directory for storing building scripts (plugins which in case of LHCb software are common, generated automatically)
     
    • releasedir - AFS directory to copy to configuration.xml file (Nightlies copy the file once and all the other servers will use exactly the same version of configuration.xml from AFS). The directory is used also for copying the software to, after the build is finished. Must exist and write persmission must be set for lhcbsoft account
    • wwwdir - where to copy html log files
    Line: 182 to 182
     
    <!-- SyntaxHighlightingPlugin -->
    <waitfor flag="/afs/cern.ch/sw/lcg/app/nightlies/dev1/%DAY%/isDone-%PLATFORM%" />
    <!-- end SyntaxHighlightingPlugin -->
    Changed:
    <
    <
    If waitfor flag is configured, compilation of eaMain.ch platform (see below) is not started until the file is available in given location.
    >
    >
    If waitfor flag is configured, compilation of each platform (see below) is not started until the file is available in given location.
     
    <!-- SyntaxHighlightingPlugin -->
    <platforms>
        <platform name="slc4_amd64_gcc34_dbg" />
        <platform name="x86_64-slc5-gcc34-dbg" />
    </platforms>
    <!-- end SyntaxHighlightingPlugin -->
    Changed:
    <
    <
    This section defines platforms to be build in the slot. If wait for is used, this is the order the files (isDone files) are looked for. The one whiMain.ch is found (if they are different for eaMain.ch of the platforms) makes the system start checkout and build stages for the platform it describes. If wait for is not used - this is the order of platforms to be build. if any platform is specified as a parameter for the starting script (lhcbNightlies-????.sh), only the platforms found there will be taken into consideration for the machine.
    >
    >
    This section defines platforms to be build in the slot. If wait for is used, this is the order the files (isDone files) are looked for. The one which is found (if they are different for each of the platforms) makes the system start checkout and build stages for the platform it describes. If wait for is not used - this is the order of platforms to be build. if any platform is specified as a parameter for the starting script (lhcbNightlies-????.sh), only the platforms found there will be taken into consideration for the machine.
     

    LbLogin settings

    <!-- SyntaxHighlightingPlugin -->
    <lblogin linux="source /afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/scripts/LbLogin.sh --no-cache --quiet --cmtvers=v1r20p20090520 ; export PATH=$PATH:/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/LBSCRIPTS_v4r1/InstallArea/scripts ; export VERBOSE=1" />
    <lblogin win="call %HOMEDRIVE%/local/lib/LbLogin.bat" />
    <!-- end SyntaxHighlightingPlugin -->
    Changed:
    <
    <
    Should be used only for setting up different login environment. The given string is prepanded to the make cmt action insiMain.de builders, and it's launched just before the cmt action (in the same command).
    >
    >
    Should be used only for setting up different login environment. The given string is prepanded to the make cmt action inside builders, and it's launched just before the cmt action (in the same command).
     

    runbefore/runafter

    %SYNTAX{ syntax="python"}%
    Line: 226 to 226
      %ENDSYNTAX%
    Changed:
    <
    <
    List of projects to be build in the slot; order is important. For eaMain.ch of the projects the following keys are obligatory:
    >
    >
    List of projects to be build in the slot; order is important. For each of the projects the following keys are obligatory:
     
    • name - real name of the project (LHCb, DaVinci, not: LHCB, davinci)
    • tag
    Additional keys (not obligatory):
    Line: 244 to 244
      %ENDSYNTAX%
    Changed:
    <
    <
    The following can be set additionally for eaMain.ch of the projects in the slot:
    >
    >
    The following can be set additionally for each of the projects in the slot:
     
    • dependence - modify the project.cmt file to use different project version (for projects one depends on). IMPORTANT: for the time being, while setting the dependence, lowercase project names should be used. Usage of the real project names must be tested.
    • change - change the specified package version; it overwrites headofeverything ("headofeverything but...").

    Troubleshooting

    Logs

    Changed:
    <
    <
    • eaMain.ch acrontab job output is sent to the lhcbsoft e-mail account unless output is empty. In the case of LHCb Nightlies there should be an e-mail for every job because some diagnostic information is always printed. This log contains checkout log whiMain.ch is not stored in any separate file. Build output is not printed to this log.
    • build log can be found for eaMain.ch project on the summary webpage and on build noMain.de local disk (files are kept on local discs for one day only!):
      /build/nightlies/<slot>/<day>/<project>/<tag>/logs/<platform>.log
    • test log can be found for eaMain.ch project on the build noMain.de local disk only (files are kept on local discs for one day only!):
      /build/nightlies/<slot>/<day>/<project>/<tag>/logs/<platform>-tests.log
    • test log summary can be found for eaMain.ch project on the summary webpage and on build noMain.de local disk (files are kept on local discs for one day only!):
      /build/nightlies/<slot>/<day>/<project>/<tag>/logs/<platform>-qmtest.log
    >
    >
    • each acrontab job output is sent to the lhcbsoft e-mail account unless output is empty. In the case of LHCb Nightlies there should be an e-mail for every job because some diagnostic information is always printed. This log contains checkout log which is not stored in any separate file. Build output is not printed to this log.
    • build log can be found for each project on the summary webpage and on build node local disk (files are kept on local discs for one day only!):
      /build/nightlies/<slot>/<day>/<project>/<tag>/logs/<platform>.log
    • test log can be found for each project on the build node local disk only (files are kept on local discs for one day only!):
      /build/nightlies/<slot>/<day>/<project>/<tag>/logs/<platform>-tests.log
    • test log summary can be found for each project on the summary webpage and on build node local disk (files are kept on local discs for one day only!):
      /build/nightlies/<slot>/<day>/<project>/<tag>/logs/<platform>-qmtest.log
     

    Possible problems

    Line: 265 to 265
     

    action

    Changed:
    <
    <
    • if only some platforms were broken, check the machines they where started on (lxbuild111/120/135) (for example consulting acrontab) - the process can be restarted only for the selected machines by performing: clean, and run actions for the given slot on the build server. *do not launMain.ch cleanAFS action unless you want to restart the slot on all platforms. IMPORTANT: if you do not clean AFS release area, the configuration.xml file from the night will be taken as a configuration for the restarted platforms.
    >
    >
    • if only some platforms were broken, check the machines they where started on (lxbuild111/120/135) (for example consulting acrontab) - the process can be restarted only for the selected machines by performing: clean, and run actions for the given slot on the build server. *do not launch cleanAFS action unless you want to restart the slot on all platforms. IMPORTANT: if you do not clean AFS release area, the configuration.xml file from the night will be taken as a configuration for the restarted platforms.
     
    Changed:
    <
    <
    -- KarolKruzelecki - 2009-09-08
    >
    >

    How to restart the Nightlies

    Production slots of the Nightlies run from lhcbsoft account. One has to login as lhcbsoft to be able to restart the Nightlies.

    Linux

    • check if the Nightlies for the slot you want to restart are still running (on any of the lxbuild machines) and stop them if so
    • if you want to restart the whole slot (all platforms), run:
      ~/bin/lhcbNightlies-prod.sh cleanAFS <slot>
      This will remove the contents of the slot's AFS directory. If you want to restart only some of the platforms, skip this step.
    • clean the local build directories and make the system checkout all the files again when started.
      ~/bin/lhcbNightlies-prod.sh clean <slot>
      It must be done on each machine you are going to use, once per slot and per machine.
    • start the Nightlies for each machine/slot (it's recommended to use acrontab job, or screen):
      ~/bin/lhcbNightlies-prod.sh run <slot> <-j> <-l> <list_of_platforms_to_build_on_this_machine>
      (details in the top of this page)

    Windows

    • check if the Nightlies for the slot you want to restart are still running (cmd window) and stop them if so (close the window with X)
    • remove the contents of the slot directory on AFS (from Windows, or from Linux):
      /afs/cern.ch/lhcb/software/nightlies/<slot>/<day>
      You can use:
      ~/bin/lhcbNightlies-prod.sh cleanAFS <slot>
      from Linux.
    • check if Kerberberos token is valid, or obtain the new one (Network Identity Manager application)
    • go to: Start -> Control Panel -> Scheduled Tasks
    • run (right click on it, and then Run): "[lhcbsoft] Nightlies - lhcb5" task in case of lhcb5 slot
    • if you restart them during the day: it's better to choose the task with "(10am)" in the name if you start it about 10 - These tasks are the copies of the main one with limited working time to stop them before midnight of the next day (when the new builds should start)

    -- KarolKruzelecki - 22-Oct-2009

     
    META TOPICMOVED by="KarolKruzelecki" date="1247756095" from="Main.LHCbNightliesConfiguration" to="LHCb.LHCbNightliesConfiguration"

    Revision 32009-09-08 - unknown

    Line: 1 to 1
     
    META TOPICPARENT name="LHCbNightlies"

    LHCb Nightly builds setup and configuration

    Line: 13 to 13
     0 2 * * * lxbuild120 ~/bin/lhcbNightlies-prod.sh clean lhcb2 0 2 * * * lxbuild135 ~/bin/lhcbNightlies-prod.sh clean lhcb2%ENDSYNTAX%

    start Nightlies (once per machine, per slot)

    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    30 2 * * * lxbuild111 ~/bin/lhcbNightlies-prod.sh run lhcb2 4 slc4_ia32_gcc34 slc4_ia32_gcc34_dbg slc4_amd64_gcc43_dbg
    40 2 * * * lxbuild120 ~/bin/lhcbNightlies-prod.sh run lhcb2 4 slc4_amd64_gcc34 slc4_amd64_gcc43 slc4_amd64_gcc34_dbg
    10 2 * * * lxbuild135 ~/bin/lhcbNightlies-prod.sh run lhcb2 8 x86_64-slc5-gcc34 x86_64-slc5-gcc43 x86_64-slc5-gcc43-opt x86_64-slc5-gcc34-opt x86_64-slc5-gcc43-dbg x86_64-slc5-gcc34-dbg Main.#i686-slc5-gcc34-dbg i686-slc5-gcc34-dbg
    <!-- end SyntaxHighlightingPlugin -->
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    30 2 * * * lxbuild111 ~/bin/lhcbNightlies-prod.sh run lhcb2 50 4 slc4_ia32_gcc34 slc4_ia32_gcc34_dbg slc4_amd64_gcc43_dbg
    40 2 * * * lxbuild120 ~/bin/lhcbNightlies-prod.sh run lhcb2 50 4 slc4_amd64_gcc34 slc4_amd64_gcc43 slc4_amd64_gcc34_dbg
    10 2 * * * lxbuild135 ~/bin/lhcbNightlies-prod.sh run lhcb2 50 8 x86_64-slc5-gcc34 x86_64-slc5-gcc43 x86_64-slc5-gcc43-opt x86_64-slc5-gcc34-opt x86_64-slc5-gcc43-dbg x86_64-slc5-gcc34-dbg Main.#i686-slc5-gcc34-dbg i686-slc5-gcc34-dbg
    <!-- end SyntaxHighlightingPlugin -->
     

    Starting points

    • ~/bin/lhcbNightlies-prod.sh - production version of LbMain.Scripts (/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod)
    Line: 23 to 23
     
    • ~/bin/lhcbNightlies-user.sh - (lhcbsoft) user version of LbMain.Scripts (/afs/cern.ch/user/l/lhcbsoft/cmtuser/LBSCRIPTS/LBSCRIPTS_v99r0)

    "run" action parameters

    Changed:
    <
    <
    • ~/bin/lhcbNightlies-prod.sh run SLOT NUMBER [PLAT1 [PLAT2 [...]]]
    • SLOT - name of the slot
    • NUMBER - currently it's -j value, it is going to be -l value instead (now only for lhcbNightlies-user.sh).
    >
    >
    • ~/bin/lhcbNightlies-prod.sh run SLOT J_VALUE L_VALUE [PLAT1 [PLAT2 [...]]]
    • SLOT - name of the slot,
    • J_VALUE - -j value for make,
    • L_VALUE - -l value for make,
     
    • PLAT1 PLAT2 ... - limit the platforms to be build on the given machine to the list given. If no platforms is given for the command, Nightlies will use this server to build all of platforms for the slot specified in configuration.xml

    additional (cleaning) steps

    Line: 136 to 137
     

    main slot settings

    %SYNTAX{ syntax="python"}%
    Changed:
    <
    <
    >
    >
     %ENDSYNTAX%
    • name - to create a new slot AFS space has to be allocated for releasedir. lhcb[1-6] are production slots, lhcb-test[1-2] are test slots started from different (not lhcbsoft) account.
    • mails - if set to false no summary e-mails will be send after the eaMain.ch of the projects in this slot is finished
    • hidden - if set to false slot will not be visible on the summary page (used by LCG Nightlies for the release slot)
    • computedependencies - if set to false, QUICK=2 is passed to make
    Added:
    >
    >
    • renice - nice value used for the processes that run nightlies for the slot
     

    paths configuration

    %SYNTAX{ syntax="python"}%
    Line: 229 to 231
     
    • tag
    Additional keys (not obligatory):
    • headofeverything - if set to true version of all the packages from project container requirements file will be changed to head.
    Added:
    >
    >
    • docs - Doxygen documentation will be generated after build of the project, if set to true
     

    tuning the project configuration

    %SYNTAX{ syntax="python"}%
    Line: 264 to 267
     

    action

    • if only some platforms were broken, check the machines they where started on (lxbuild111/120/135) (for example consulting acrontab) - the process can be restarted only for the selected machines by performing: clean, and run actions for the given slot on the build server. *do not launMain.ch cleanAFS action unless you want to restart the slot on all platforms. IMPORTANT: if you do not clean AFS release area, the configuration.xml file from the night will be taken as a configuration for the restarted platforms.
    Changed:
    <
    <
    -- KarolKruzelecki - 16 Jul 20Main.09
    >
    >
    -- KarolKruzelecki - 2009-09-08
     
    META TOPICMOVED by="KarolKruzelecki" date="1247756095" from="Main.LHCbNightliesConfiguration" to="LHCb.LHCbNightliesConfiguration"

    Revision 22009-07-16 - KarolKruzelecki

    Line: 1 to 1
    Changed:
    <
    <
    META TOPICPARENT name="KarolKruzelecki"
    >
    >
    META TOPICPARENT name="LHCbNightlies"
     

    LHCb Nightly builds setup and configuration

    System bootstrap

    Nighty builds are started every night with use of acrontab of lhcbsoft user. Actions are separated, because some of them have to be done once per slot and the others once per slot and per machine.

    main steps

    Changed:
    <
    <
    • clean AFS release area of slot (only once per slot)
    <!-- SyntaxHighlightingPlugin -->
    45 0 * * * lxplus.cern.ch ~/bin/lhcbNightlies.sh cleanAFS lhcb2
    <!-- end SyntaxHighlightingPlugin -->
    • clean build directories (once per machine, per slot)
    <!-- SyntaxHighlightingPlugin -->
    0 2 * * * lxbuild111 ~/bin/lhcbNightlies.sh clean lhcb2
    0 2 * * * lxbuild120 ~/bin/lhcbNightlies.sh clean lhcb2
    0 2 * * * lxbuild135 ~/bin/lhcbNightlies.sh clean lhcb2
    <!-- end SyntaxHighlightingPlugin -->
    • start Nightlies (once per machine, per slot)
    <!-- SyntaxHighlightingPlugin -->
    30 2 * * * lxbuild111 ~/bin/lhcbNightlies.sh run lhcb2 4 slc4_ia32_gcc34 slc4_ia32_gcc34_dbg slc4_amd64_gcc43_dbg
    40 2 * * * lxbuild120 ~/bin/lhcbNightlies.sh run lhcb2 4 slc4_amd64_gcc34 slc4_amd64_gcc43 slc4_amd64_gcc34_dbg
    10 2 * * * lxbuild135 ~/bin/lhcbNightlies.sh run lhcb2 8 x86_64-slc5-gcc34 x86_64-slc5-gcc43 x86_64-slc5-gcc43-opt x86_64-slc5-gcc34-opt x86_64-slc5-gcc43-dbg x86_64-slc5-gcc34-dbg #i686-slc5-gcc34-dbg i686-slc5-gcc34-dbg
    <!-- end SyntaxHighlightingPlugin -->
    >
    >

    clean AFS release area of slot (only once per slot)

    <!-- SyntaxHighlightingPlugin -->
    45 0 * * * lxplus.cern.ch ~/bin/lhcbNightlies-prod.sh cleanAFS lhcb2
    <!-- end SyntaxHighlightingPlugin -->

    clean build directories (once per machine, per slot)

    <!-- SyntaxHighlightingPlugin -->
    0 2 * * * lxbuild111 ~/bin/lhcbNightlies-prod.sh clean lhcb2
    0 2 * * * lxbuild120 ~/bin/lhcbNightlies-prod.sh clean lhcb2
    0 2 * * * lxbuild135 ~/bin/lhcbNightlies-prod.sh clean lhcb2
    <!-- end SyntaxHighlightingPlugin -->

    start Nightlies (once per machine, per slot)

    <!-- SyntaxHighlightingPlugin -->
    30 2 * * * lxbuild111 ~/bin/lhcbNightlies-prod.sh run lhcb2 4 slc4_ia32_gcc34 slc4_ia32_gcc34_dbg slc4_amd64_gcc43_dbg
    40 2 * * * lxbuild120 ~/bin/lhcbNightlies-prod.sh run lhcb2 4 slc4_amd64_gcc34 slc4_amd64_gcc43 slc4_amd64_gcc34_dbg
    10 2 * * * lxbuild135 ~/bin/lhcbNightlies-prod.sh run lhcb2 8 x86_64-slc5-gcc34 x86_64-slc5-gcc43 x86_64-slc5-gcc43-opt x86_64-slc5-gcc34-opt x86_64-slc5-gcc43-dbg x86_64-slc5-gcc34-dbg Main.#i686-slc5-gcc34-dbg i686-slc5-gcc34-dbg
    <!-- end SyntaxHighlightingPlugin -->

    Starting points

    • ~/bin/lhcbNightlies-prod.sh - production version of LbMain.Scripts (/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod)
    • ~/bin/lhcbNightlies-dev.sh - developement version of LbMain.Scripts (/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/dev)
    • ~/bin/lhcbNightlies-user.sh - (lhcbsoft) user version of LbMain.Scripts (/afs/cern.ch/user/l/lhcbsoft/cmtuser/LBSCRIPTS/LBSCRIPTS_v99r0)

    "run" action parameters

    • ~/bin/lhcbNightlies-prod.sh run SLOT NUMBER [PLAT1 [PLAT2 [...]]]
    • SLOT - name of the slot
    • NUMBER - currently it's -j value, it is going to be -l value instead (now only for lhcbNightlies-user.sh).
    • PLAT1 PLAT2 ... - limit the platforms to be build on the given machine to the list given. If no platforms is given for the command, Nightlies will use this server to build all of platforms for the slot specified in configuration.xml
     

    additional (cleaning) steps

    Changed:
    <
    <
    • remove files from the day before yesterday
    >
    >

    remove files from the day before yesterday (per machine)

     
    <!-- SyntaxHighlightingPlugin -->
    12 0 * * * lxbuild111 rm -rf /build/nightlies/lhcb?/`date --date='2 days ago' | awk '{print $1}'`/*
    12 0 * * * lxbuild120 rm -rf /build/nightlies/lhcb?/`date --date='2 days ago' | awk '{print $1}'`/*
    12 0 * * * lxbuild135 rm -rf /build/nightlies/lhcb?/`date --date='2 days ago' | awk '{print $1}'`/*
    <!-- end SyntaxHighlightingPlugin -->
    Changed:
    <
    <
    • remove log files from last week
    >
    >

    remove log files from last week (per slot)

     %SYNTAX{ syntax="python"}%10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb1.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb2.`date | awk '{print $1}'`_* 10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb3.`date | awk '{print $1}'`_*
    Line: 38 to 49
     

    configuration history (from last 7 days):

    %SYNTAX{ syntax="python"}%
    Changed:
    <
    <
    /afs/cern.ch/lhcb/software/nightlies/conf/history/..xml
    >
    >
    /afs/cern.ch/lhcb/software/nightlies/conf/history/..xml -- obsolete, will be removed
     /afs/cern.ch/lhcb/software/nightlies///configuration.xml (Nighlties release area) %ENDSYNTAX%
    Line: 79 to 90
     
    Added:
    >
    >
     %ENDSYNTAX%
    Added:
    >
    >
    • wwwurl, logurl, rrdurl - URLs to find the files (index.html, log files, test statistic graphs)
    • wwwtitle - header of the Nightlies summary page
    • mailfrom - (cern) e-mail address - sender of the build/test results
    • mailsubjectprefix
    • shownotfinishedplatforms - if set to true platforms whiMain.ch are not finished will also be shown on the summary page (isStarted file flags are used then instead of isDone)
     

    configuration/general/ignore

    Changed:
    <
    <
    Each line of the log files is compared to all of the given error/warning statements and if any of error/warning values can be found in the line, the line is not marked as one containing an error/warning. Errors and warings are considered separately, but the values are common for all of the slots.
    >
    >
    EaMain.ch line of the log files is compared to all of the given error/warning statements and if any of error/warning values can be found in the line, the line is not marked as one containing an error/warning. Errors and warings are considered separately, but the values are common for all of the slots. Regular expressions (type="regex") or fnmatMain.ch/glob expressions (type="fnmatMain.ch") can be used here. In the case of regex - remember about .* in the beginning and the end of the expression (unless you are sure what you are doing).
     %SYNTAX{ syntax="python"}%
    Added:
    >
    >
     %ENDSYNTAX%

    configuration/general/mailto

    Changed:
    <
    <
    All of the projects which are used in one or more slots must have a section here as follows:
    >
    >
    All of the projects whiMain.ch are used in one or more slots must have a section here as follows:
     %SYNTAX{ syntax="python"}%
    Line: 98 to 116
     %ENDSYNTAX% Name of the project should be given as they are (LHCb and DaVinci, not LHCB, DAVINCI). tag is used to send summary of the builds to the responsible person after the build is finished. The section can contain none, one, or more tags. edit and parent attributes are not in use at the moment. tag is not in use at the moment.
    Added:
    >
    >
    section: project must be configured for every project whiMain.ch is used in one or more of the slots, can be empty (no mail, repository tags)

    configuration/general/builders

    <!-- SyntaxHighlightingPlugin -->
    <builders>
        <project name="Geant4" path="%LHCBRELEASES%/../nightlies/plugins/Geant4" />
    </builders>
    <!-- end SyntaxHighlightingPlugin -->
    In this section it can be configured whiMain.ch builders should be copied from the given location instead of being generated automatically (environment variables can be used as shown - case sensitive).
     

    per slot configuration

    Added:
    >
    >

    defined values, usage ofenvironmental variables

    • %SLOT% - slot name
    • %DAY% - today - three letter weekday (Mon, Tue, ...)
    • %YESTERDAY% - yesterday
    • %PLATFORM% - platform name
    • Environment variable can be used in the same way - between two percent characters (case sensitive)

    main slot settings

     %SYNTAX{ syntax="python"}%
    Added:
    >
    >
    %ENDSYNTAX%
    • name - to create a new slot AFS space has to be allocated for releasedir. lhcb[1-6] are production slots, lhcb-test[1-2] are test slots started from different (not lhcbsoft) account.
    • mails - if set to false no summary e-mails will be send after the eaMain.ch of the projects in this slot is finished
    • hidden - if set to false slot will not be visible on the summary page (used by LCG Nightlies for the release slot)
    • computedependencies - if set to false, QUICK=2 is passed to make

    paths configuration

    %SYNTAX{ syntax="python"}%
     
    Added:
    >
    >
    %ENDSYNTAX% all paths in this section are obligatory

    • builddir - local build directory (must exist in filesystem and write permission has to be set for lhcbsoft user)
    • buildersdir - local directory for storing building scripts (plugins whiMain.ch in case of LHCb software are common, generated automatically)
    • releasedir - AFS directory to copy to configuration.xml file (Nightlies copy the file once and all the other servers will use exactly the same version of configuration.xml from AFS). The directory is used also for copying the software to, after the build is finished. Must exist and write persmission must be set for lhcbsoft account
    • wwwdir - where to copy html log files

    CMTPROJECTPATH

    %SYNTAX{ syntax="python"}%
     
    Line: 114 to 167
     
    Added:
    >
    >
    %ENDSYNTAX% CMTPROJECTPATH variable, order is important, local build directory is prepended.

    CMTEXTRATAGS

    <!-- SyntaxHighlightingPlugin -->
    <cmtextratags value="use-distcc" />
    <!-- end SyntaxHighlightingPlugin -->
    Setup of CMTEXTRATAGS value.

    "wait for" flag

    <!-- SyntaxHighlightingPlugin -->
    <waitfor flag="/afs/cern.ch/sw/lcg/app/nightlies/dev1/%DAY%/isDone-%PLATFORM%" />
    <!-- end SyntaxHighlightingPlugin -->
    If waitfor flag is configured, compilation of eaMain.ch platform (see below) is not started until the file is available in given location. %SYNTAX{ syntax="python"}%
     
    Changed:
    <
    <
    >
    >
    %ENDSYNTAX% This section defines platforms to be build in the slot. If wait for is used, this is the order the files (isDone files) are looked for. The one whiMain.ch is found (if they are different for eaMain.ch of the platforms) makes the system start checkout and build stages for the platform it describes. If wait for is not used - this is the order of platforms to be build. if any platform is specified as a parameter for the starting script (lhcbNightlies-????.sh), only the platforms found there will be taken into consideration for the machine.

    LbLogin settings

    <!-- SyntaxHighlightingPlugin -->
    <lblogin linux="source /afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/scripts/LbLogin.sh --no-cache --quiet --cmtvers=v1r20p20090520 ; export PATH=$PATH:/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/LBSCRIPTS_v4r1/InstallArea/scripts ; export VERBOSE=1" />
    <lblogin win="call %HOMEDRIVE%/local/lib/LbLogin.bat" />
    <!-- end SyntaxHighlightingPlugin -->
    Should be used only for setting up different login environment. The given string is prepanded to the make cmt action insiMain.de builders, and it's launched just before the cmt action (in the same command).

    runbefore/runafter

    %SYNTAX{ syntax="python"}%
     
    Added:
    >
    >
    %ENDSYNTAX% Commands to run before/after the run action is started.

    days

    %SYNTAX{ syntax="python"}%
     
    Added:
    >
    >
    %ENDSYNTAX% Setting any of the weekdays to false will make the Nightlies not run this day.

    list of projects

    %SYNTAX{ syntax="python"}%
     
    Deleted:
    <
    <
    <!--dependence project="gaudi" tag="GAUDI_v20r4" /-->
     
    Line: 135 to 223
     
    Deleted:
    <
    <
     %ENDSYNTAX%
    Added:
    >
    >
    List of projects to be build in the slot; order is important. For eaMain.ch of the projects the following keys are obligatory:
    • name - real name of the project (LHCb, DaVinci, not: LHCB, davinci)
    • tag
    Additional keys (not obligatory):
    • headofeverything - if set to true version of all the packages from project container requirements file will be changed to head.

    tuning the project configuration

    <!-- SyntaxHighlightingPlugin -->
    <project name="Gauss" tag="GAUSS_HEAD" headofeverything="true">
            <dependence project="geant4" tag="GEANT4_v91r2p1" />
        </project>
        <project name="LHCb" tag="LHCB_HEAD" headofeverything="true">
            <dependence project="gaudi" tag="GAUDI_HEAD" />
            <change package="Phys/LoKiArrayFunctors" value="vanya_20090424"/>
            <change package="Det/DetDescSvc" value="v2r2"/>
        </project>
    <!-- end SyntaxHighlightingPlugin -->
    The following can be set additionally for eaMain.ch of the projects in the slot:
    • dependence - modify the project.cmt file to use different project version (for projects one depends on). IMPORTANT: for the time being, while setting the dependence, lowercase project names should be used. Usage of the real project names must be tested.
    • change - change the specified package version; it overwrites headofeverything ("headofeverything but...").

    Troubleshooting

    Logs

    • eaMain.ch acrontab job output is sent to the lhcbsoft e-mail account unless output is empty. In the case of LHCb Nightlies there should be an e-mail for every job because some diagnostic information is always printed. This log contains checkout log whiMain.ch is not stored in any separate file. Build output is not printed to this log.
    • build log can be found for eaMain.ch project on the summary webpage and on build noMain.de local disk (files are kept on local discs for one day only!):
      /build/nightlies/<slot>/<day>/<project>/<tag>/logs/<platform>.log
    • test log can be found for eaMain.ch project on the build noMain.de local disk only (files are kept on local discs for one day only!):
      /build/nightlies/<slot>/<day>/<project>/<tag>/logs/<platform>-tests.log
    • test log summary can be found for eaMain.ch project on the summary webpage and on build noMain.de local disk (files are kept on local discs for one day only!):
      /build/nightlies/<slot>/<day>/<project>/<tag>/logs/<platform>-qmtest.log

    Possible problems

    checkout problem

    visible symptoms

    • some of the projects' builds are "red" in summary page with almost empty build log (only summary in the top of page)
    • different platforms broken at different projects, and starting from these projects everything is "red"

    confirm

    action

    • if only some platforms were broken, check the machines they where started on (lxbuild111/120/135) (for example consulting acrontab) - the process can be restarted only for the selected machines by performing: clean, and run actions for the given slot on the build server. *do not launMain.ch cleanAFS action unless you want to restart the slot on all platforms. IMPORTANT: if you do not clean AFS release area, the configuration.xml file from the night will be taken as a configuration for the restarted platforms.
     
    Added:
    >
    >
    -- KarolKruzelecki - 16 Jul 20Main.09
     
    Deleted:
    <
    <
    -- KarolKruzelecki - 08 Jul 2009
     \ No newline at end of file
    Added:
    >
    >
    META TOPICMOVED by="KarolKruzelecki" date="1247756095" from="Main.LHCbNightliesConfiguration" to="LHCb.LHCbNightliesConfiguration"

    Revision 12009-07-08 - KarolKruzelecki

    Line: 1 to 1
    Added:
    >
    >
    META TOPICPARENT name="KarolKruzelecki"

    LHCb Nightly builds setup and configuration

    System bootstrap

    Nighty builds are started every night with use of acrontab of lhcbsoft user. Actions are separated, because some of them have to be done once per slot and the others once per slot and per machine.

    main steps

    • clean AFS release area of slot (only once per slot)
    <!-- SyntaxHighlightingPlugin -->
    45 0 * * * lxplus.cern.ch ~/bin/lhcbNightlies.sh cleanAFS lhcb2
    <!-- end SyntaxHighlightingPlugin -->
    • clean build directories (once per machine, per slot)
    <!-- SyntaxHighlightingPlugin -->
    0 2 * * * lxbuild111 ~/bin/lhcbNightlies.sh clean lhcb2
    0 2 * * * lxbuild120 ~/bin/lhcbNightlies.sh clean lhcb2
    0 2 * * * lxbuild135 ~/bin/lhcbNightlies.sh clean lhcb2
    <!-- end SyntaxHighlightingPlugin -->
    • start Nightlies (once per machine, per slot)
    <!-- SyntaxHighlightingPlugin -->
    30 2 * * * lxbuild111 ~/bin/lhcbNightlies.sh run lhcb2 4 slc4_ia32_gcc34 slc4_ia32_gcc34_dbg slc4_amd64_gcc43_dbg
    40 2 * * * lxbuild120 ~/bin/lhcbNightlies.sh run lhcb2 4 slc4_amd64_gcc34 slc4_amd64_gcc43 slc4_amd64_gcc34_dbg
    10 2 * * * lxbuild135 ~/bin/lhcbNightlies.sh run lhcb2 8 x86_64-slc5-gcc34 x86_64-slc5-gcc43 x86_64-slc5-gcc43-opt x86_64-slc5-gcc34-opt x86_64-slc5-gcc43-dbg x86_64-slc5-gcc34-dbg #i686-slc5-gcc34-dbg i686-slc5-gcc34-dbg
    <!-- end SyntaxHighlightingPlugin -->

    additional (cleaning) steps

    • remove files from the day before yesterday
    <!-- SyntaxHighlightingPlugin -->
    12 0 * * * lxbuild111 rm -rf /build/nightlies/lhcb?/`date --date='2 days ago' | awk '{print $1}'`/*
    12 0 * * * lxbuild120 rm -rf /build/nightlies/lhcb?/`date --date='2 days ago' | awk '{print $1}'`/*
    12 0 * * * lxbuild135 rm -rf /build/nightlies/lhcb?/`date --date='2 days ago' | awk '{print $1}'`/*
    <!-- end SyntaxHighlightingPlugin -->
    • remove log files from last week
    <!-- SyntaxHighlightingPlugin -->
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb1.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb2.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb3.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb4.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb5.`date | awk '{print $1}'`_*
    10 0 * * * lxplus rm -rf /afs/cern.ch/lhcb/software/nightlies/www/logs/lhcb6.`date | awk '{print $1}'`_*
    <!-- end SyntaxHighlightingPlugin -->

    Configuration

    current configuration file is taken from AFS and can be found:

    <!-- SyntaxHighlightingPlugin -->
    /afs/cern.ch/lhcb/software/nightlies/conf/configuration.xml
    <!-- end SyntaxHighlightingPlugin -->

    configuration history (from last 7 days):

    <!-- SyntaxHighlightingPlugin -->
    /afs/cern.ch/lhcb/software/nightlies/conf/history/<day>.<slot>.xml 
    /afs/cern.ch/lhcb/software/nightlies/<slot>/<day>/configuration.xml (Nighlties release area)
    <!-- end SyntaxHighlightingPlugin -->

    xml file format:

    <!-- SyntaxHighlightingPlugin -->
    <configuration>
        <general>
            <parameters>
                <parameter name="" value="" />
                < ... >
            </parameters>
            <ignore>
                <warning value="" />
                <error value="" />
            </ignore>
            <mailto>
                <project name="lhcb">
                    <mail address="marco.cattaneo@cern.ch" edit="false" parent="false" />
                    <repository path="isscvs.cern.ch:/local/reps/lhcb" type="cvs"/>
                </project>
            </mailto>
        </general>
        <slot>
            <!-- slot contents -->
        </slot>
        <slot>
            <!-- slot contents -->
        </slot>
        < ... >
    </configuration>
    <!-- end SyntaxHighlightingPlugin -->

    configuration/general/parameters currently available:

    <!-- SyntaxHighlightingPlugin -->
    <parameter name="wwwurl" value="http://cern.ch/lhcb-nightlies" />
    <parameter name="logurl" value="http://cern.ch/lhcb-nightlies/logs" />
    <parameter name="rrdurl" value="http://cern.ch/lhcb-nightlies/rrd" />
    <parameter name="wwwtitle" value="Summaries of nightly builds for LHCb" />
    <parameter name="mailfrom" value="karol.kruzelecki@cern.ch" />
    <parameter name="mailsubjectprefix" value="[LHCb Nightlies]" />
    <!-- end SyntaxHighlightingPlugin -->

    configuration/general/ignore

    Each line of the log files is compared to all of the given error/warning statements and if any of error/warning values can be found in the line, the line is not marked as one containing an error/warning. Errors and warings are considered separately, but the values are common for all of the slots.
    <!-- SyntaxHighlightingPlugin -->
    <warning value="" />
    <error value="" />
    <!-- end SyntaxHighlightingPlugin -->

    configuration/general/mailto

    All of the projects which are used in one or more slots must have a section here as follows:
    <!-- SyntaxHighlightingPlugin -->
    <project name="lhcb">
        <mail address="marco.cattaneo@cern.ch" edit="false" parent="false" />
        <repository path="isscvs.cern.ch:/local/reps/lhcb" type="cvs"/>
    </project>
    <!-- end SyntaxHighlightingPlugin -->
    Name of the project should be given as they are (LHCb and DaVinci, not LHCB, DAVINCI). tag is used to send summary of the builds to the responsible person after the build is finished. The section can contain none, one, or more tags. edit and parent attributes are not in use at the moment. tag is not in use at the moment.

    per slot configuration

    <!-- SyntaxHighlightingPlugin -->
    <slot name="lhcb2" description="lhcb 2 HEAD of everything on top of GAUDI_v21r0 and LCG_56" mails="true" hidden="false" computedependencies="false">
        <paths>
            <path name="builddir" value="/build/nightlies/%SLOT%/%DAY%" />
            <path name="buildersdir" value="/build/builders/%SLOT%" />
            <path name="releasedir" value="/afs/cern.ch/lhcb/software/nightlies/%SLOT%/%DAY%" />
            <path name="wwwdir" value="/afs/cern.ch/lhcb/software/nightlies/www/logs" />
        </paths>
        <cmtprojectpath>
            <path value="/afs/cern.ch/user/m/marcocle/public" />
            <path value="/afs/cern.ch/sw/Gaudi/releases" />
            <path value="/afs/cern.ch/sw/lcg/app/nightlies/dev1/%DAY%" />
            <path value="/afs/cern.ch/sw/lcg/app/releases" />
            <path value="/afs/cern.ch/lhcb/software/releases" />
        </cmtprojectpath>
        <platforms>
            <platform name="slc4_amd64_gcc34_dbg" />
            <platform name="x86_64-slc5-gcc34-dbg" />
        </platforms>
        <waitfor flag="/afs/cern.ch/sw/lcg/app/nightlies/dev1/%DAY%/isDone-%PLATFORM%" />
        <runbefore linux="any bash command && sequence of commands" win="any windows shell command" />
        <runafter linux="any bash command && sequence of commands" win="any windows shell command"/>
        <days mon="true" tue="true" wed="true" thu="true" fri="true" sat="true" sun="true" />
        <projects>
            <project name="Gaudi" tag="GAUDI_v21r0">
                <dependence project="lcgcmt" tag="LCGCMT_56" />
            </project>
            <project name="LHCb" tag="LHCB_HEAD" headofeverything="true">
                <!--dependence project="gaudi" tag="GAUDI_v20r4" /-->
            </project>
            <project name="Lbcom" tag="LBCOM_HEAD" headofeverything="true" />
            <project name="Gauss" tag="GAUSS_HEAD" headofeverything="true">
                <dependence project="geant4" tag="GEANT4_v91r2p1" />
            </project>
            <project name="Panoptes" tag="PANOPTES_HEAD" headofeverything="true" />
        </projects>
    </slot>
    <!-- end SyntaxHighlightingPlugin -->

    -- KarolKruzelecki - 08 Jul 2009

     
    This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
    Ideas, requests, problems regarding TWiki? Send feedback