How to prepare a new machine for running the nightlies

Overview of existing machines

LHCbNightliesPrerequirements

Scientific Linux release specific information

  • SLC6
    • link
  • SLC5

general steps

  • machine accessible... --> ask IT for moving the machine into production... wink
  • local build directory exists and is accessible (depends on the slot configuration!)
    • standard is /build/nightlies/ --> +rw for lhcbsoft
  • prepare for use of distcc
    • create /build/nightlies/.distcc/ directory
  • for some old logs a local directory is used
    • create = /build/nightlies/OldLogs/=

  • add the machine to acrontab
    • build jobs
      • 09 02 * * * buildlhcbXX nice -n 5 ~/bin/nightliesClient.sh x86_64-slc5-icc11-opt > /afs/cern.ch/lhcb/software/nightlies/www/logs/nightliesClient/nightliesClient_`date +\%a`_XXc.txt 2>&1
    • clean local build dirs
      • 2 0 * * * buildlhcbXX rm -rf /build/nightlies/lhcb-*/`date --date='1 days ago' +\%a`/*
      • 10 0 * * * buildlhcbXX rm -rf /build/nightlies/lhcb-*/`date +\%a`
    • kill all remaining jobs from yesterday
      • 3 0 * * * buildlhcbXX ~/bin/killTree.sh nightlies `date --date='1 day ago' +\%a` > /build/nightlies/OldLogs/KillClient.log 2>&1

SLS sensor

  • A SLS sensor is to be placed for a new machine to monitor its disk usage
  • to monitor the build machine, a script collects the information and pushes it to SLS as xml
    • run the monitor from acrontab and let it place the xml onto AFS to be picked up from AFS
      • */20 * * * * buildlhcbXX export PYTHONPATH=/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/python:$PYTHONPATH && /afs/cern.ch/user/l/lhcbsoft/PRODUCTION/LHCbNightliesProd/LHCbNightlies/NightliesSensors/AFSSensor.py --aw /afs/cern.ch/lhcb/software/nightlies/www/LHCb_Nightlies_buildlhcbXX.xml --dirdepth 0 --dirroot /build/nightlies --xmlid LHCb_Nightlies_buildlhcb04 >  /build/nightlies/OldLogs/localdisk.out 2>&1
  • prepare the SLS interface
    • the admin page is available under https://apex.cern.ch/pls/htmldb_itcore/f?p=116:LOGIN:888212673570801:::::
    • if you not yet an account for SDB ask Victor for example
      • you will most probable get an account, able to edit existing sensors but not create new ones
        • ask Victor, if he can clone an existing sensor for you
        • when available, you can add the path to the sensor xml (were you let acrontab write to ) to the sensor etc.

local services

  • to organize log files in the local build directory logrotate is used
    • normally the general logrotate configuration in /afs/cern.ch/user/l/lhcbsoft/public/PRODUCTION/LHCbNightliesProd/LHCbNightlies/NightliesConfig/logRotateLocal.conf should be fine
      • the local state file is normally placed at /build/nightlies/logrotate.local
    • set up the local crontab, i.e.
      • */2 * * * * date -Is >>  /build/nightlies/meminfo.txt; cat /proc/meminfo >> /build/nightlies/meminfo.txt; ps aux --forest >> /build/nightlies/meminfo.txt; echo "-----------" >> /build/nightlies/meminfo.txt
        55 23 * * * tar cjf /build/nightlies/OldLogs/ProcessesMeminfo.tar.bz2  /build/nightlies/meminfo.txt > /dev/null 2>&1; rm /build/nightlies/meminfo.txt
        55 1  * * * /usr/sbin/logrotate -s /build/nightlies/logrotate.local /afs/cern.ch/user/l/lhcbsoft/public/PRODUCTION/LHCbNightliesProd/LHCbNightlies/NightliesConfig/logRotateLocal.conf

-- ThomasHartmann - 12-Oct-2012

Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r6 - 2013-09-27 - MarcoClemencic
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback