Difference: LHCbNightliesNewMachine (1 vs. 6)

Revision 62013-09-27 - MarcoClemencic

Line: 1 to 1
Changed:
<
<
META TOPICPARENT name="LHCbNightlies"
>
>
META TOPICPARENT name="LHCbNightliesOld"
 

How to prepare a new machine for running the nightlies

Overview of existing machines

Revision 52012-10-22 - ThomasHartmann

Line: 1 to 1
 
META TOPICPARENT name="LHCbNightlies"

How to prepare a new machine for running the nightlies

Added:
>
>

Overview of existing machines

LHCbNightliesPrerequirements
 

Scientific Linux release specific information

  • SLC6
    • link
Line: 26 to 29
 
      • 3 0 * * * buildlhcbXX ~/bin/killTree.sh nightlies `date --date='1 day ago' +\%a` > /build/nightlies/OldLogs/KillClient.log 2>&1

SLS sensor

Added:
>
>
  • A SLS sensor is to be placed for a new machine to monitor its disk usage
 
  • to monitor the build machine, a script collects the information and pushes it to SLS as xml
    • run the monitor from acrontab and let it place the xml onto AFS to be picked up from AFS
      • */20 * * * * buildlhcbXX export PYTHONPATH=/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/python:$PYTHONPATH && /afs/cern.ch/user/l/lhcbsoft/PRODUCTION/LHCbNightliesProd/LHCbNightlies/NightliesSensors/AFSSensor.py --aw /afs/cern.ch/lhcb/software/nightlies/www/LHCb_Nightlies_buildlhcbXX.xml --dirdepth 0 --dirroot /build/nightlies --xmlid LHCb_Nightlies_buildlhcb04 >  /build/nightlies/OldLogs/localdisk.out 2>&1

Revision 42012-10-18 - ThomasHartmann

Line: 1 to 1
 
META TOPICPARENT name="LHCbNightlies"

How to prepare a new machine for running the nightlies

Line: 25 to 25
 
    • kill all remaining jobs from yesterday
      • 3 0 * * * buildlhcbXX ~/bin/killTree.sh nightlies `date --date='1 day ago' +\%a` > /build/nightlies/OldLogs/KillClient.log 2>&1
Changed:
<
<

local SLS sensor

>
>

SLS sensor

 
  • to monitor the build machine, a script collects the information and pushes it to SLS as xml
    • run the monitor from acrontab and let it place the xml onto AFS to be picked up from AFS
      • */20 * * * * buildlhcbXX export PYTHONPATH=/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/python:$PYTHONPATH && /afs/cern.ch/user/l/lhcbsoft/PRODUCTION/LHCbNightliesProd/LHCbNightlies/NightliesSensors/AFSSensor.py --aw /afs/cern.ch/lhcb/software/nightlies/www/LHCb_Nightlies_buildlhcbXX.xml --dirdepth 0 --dirroot /build/nightlies --xmlid LHCb_Nightlies_buildlhcb04 >  /build/nightlies/OldLogs/localdisk.out 2>&1
Line: 36 to 36
 
        • ask Victor, if he can clone an existing sensor for you
        • when available, you can add the path to the sensor xml (were you let acrontab write to ) to the sensor etc.
Added:
>
>

local services

  • to organize log files in the local build directory logrotate is used
    • normally the general logrotate configuration in /afs/cern.ch/user/l/lhcbsoft/public/PRODUCTION/LHCbNightliesProd/LHCbNightlies/NightliesConfig/logRotateLocal.conf should be fine
      • the local state file is normally placed at /build/nightlies/logrotate.local
    • set up the local crontab, i.e.
      • */2 * * * * date -Is >>  /build/nightlies/meminfo.txt; cat /proc/meminfo >> /build/nightlies/meminfo.txt; ps aux --forest >> /build/nightlies/meminfo.txt; echo "-----------" >> /build/nightlies/meminfo.txt
        55 23 * * * tar cjf /build/nightlies/OldLogs/ProcessesMeminfo.tar.bz2  /build/nightlies/meminfo.txt > /dev/null 2>&1; rm /build/nightlies/meminfo.txt
        55 1  * * * /usr/sbin/logrotate -s /build/nightlies/logrotate.local /afs/cern.ch/user/l/lhcbsoft/public/PRODUCTION/LHCbNightliesProd/LHCbNightlies/NightliesConfig/logRotateLocal.conf
  -- ThomasHartmann - 12-Oct-2012

Revision 32012-10-16 - ThomasHartmann

Line: 1 to 1
 
META TOPICPARENT name="LHCbNightlies"

How to prepare a new machine for running the nightlies

Line: 13 to 13
 
 
    • standard is /build/nightlies/ -->
+rw for lhcbsoft
  • prepare for use of distcc
    • create /build/nightlies/.distcc/ directory
  • Added:
    >
    >
    • for some old logs a local directory is used
      • create = /build/nightlies/OldLogs/=

    • add the machine to acrontab
      • build jobs
        • 09 02 * * * buildlhcbXX nice -n 5 ~/bin/nightliesClient.sh x86_64-slc5-icc11-opt > /afs/cern.ch/lhcb/software/nightlies/www/logs/nightliesClient/nightliesClient_`date +\%a`_XXc.txt 2>&1
      • clean local build dirs
        • 2 0 * * * buildlhcbXX rm -rf /build/nightlies/lhcb-*/`date --date='1 days ago' +\%a`/*
        • 10 0 * * * buildlhcbXX rm -rf /build/nightlies/lhcb-*/`date +\%a`
      • kill all remaining jobs from yesterday
        • 3 0 * * * buildlhcbXX ~/bin/killTree.sh nightlies `date --date='1 day ago' +\%a` > /build/nightlies/OldLogs/KillClient.log 2>&1

    local SLS sensor

    • to monitor the build machine, a script collects the information and pushes it to SLS as xml
      • run the monitor from acrontab and let it place the xml onto AFS to be picked up from AFS
        • */20 * * * * buildlhcbXX export PYTHONPATH=/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/python:$PYTHONPATH && /afs/cern.ch/user/l/lhcbsoft/PRODUCTION/LHCbNightliesProd/LHCbNightlies/NightliesSensors/AFSSensor.py --aw /afs/cern.ch/lhcb/software/nightlies/www/LHCb_Nightlies_buildlhcbXX.xml --dirdepth 0 --dirroot /build/nightlies --xmlid LHCb_Nightlies_buildlhcb04 >  /build/nightlies/OldLogs/localdisk.out 2>&1
    • prepare the SLS interface
      • the admin page is available under https://apex.cern.ch/pls/htmldb_itcore/f?p=116:LOGIN:888212673570801:::::
      • if you not yet an account for SDB ask Victor for example
        • you will most probable get an account, able to edit existing sensors but not create new ones
          • ask Victor, if he can clone an existing sensor for you
          • when available, you can add the path to the sensor xml (were you let acrontab write to ) to the sensor etc.
      -- ThomasHartmann - 12-Oct-2012

    Revision 22012-10-15 - ThomasHartmann

    Line: 1 to 1
     
    META TOPICPARENT name="LHCbNightlies"

    How to prepare a new machine for running the nightlies

    Line: 7 to 7
     
      • link
    • SLC5
    Added:
    >
    >

    general steps

    • machine accessible... -->
    ask IT for moving the machine into production... wink
  • local build directory exists and is accessible (depends on the slot configuration!)
    • standard is /build/nightlies/ --> +rw for lhcbsoft
  • prepare for use of distcc
    • create /build/nightlies/.distcc/ directory
  •  -- ThomasHartmann - 12-Oct-2012

    Revision 12012-10-12 - ThomasHartmann

    Line: 1 to 1
    Added:
    >
    >
    META TOPICPARENT name="LHCbNightlies"

    How to prepare a new machine for running the nightlies

    Scientific Linux release specific information

    • SLC6
      • link
    • SLC5

    -- ThomasHartmann - 12-Oct-2012

     
    This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
    Ideas, requests, problems regarding TWiki? Send feedback