How to prepare a new machine for running the nightlies

Overview of existing machines

Scientific Linux release specific information

• SLC6
  • link
• SLC5

general steps

• machine accessible... --> ask IT for moving the machine into production...
• local build directory exists and is accessible (depends on the slot configuration!)
  • standard is /build/nightlies/ --> +rw for lhcbsoft
• prepare for use of distcc
  • create /build/nightlies/.distcc/ directory
• for some old logs a local directory is used
  • create = /build/nightlies/OldLogs/=

• add the machine to acrontab
  • build jobs
    • 09 02 * * * buildlhcbXX nice -n 5 ~/bin/nightliesClient.sh x86_64-slc5-icc11-opt > /afs/cern.ch/lhcb/software/nightlies/www/logs/nightliesClient/nightliesClient_`date +\%a`_XXc.txt 2>&1
  • clean local build dirs
    • 2 0 * * * buildlhcbXX rm -rf /build/nightlies/lhcb-*/`date --date='1 days ago' +\%a`/*
    • 10 0 * * * buildlhcbXX rm -rf /build/nightlies/lhcb-*/`date +\%a`
  • kill all remaining jobs from yesterday
    • 3 0 * * * buildlhcbXX ~/bin/killTree.sh nightlies `date --date='1 day ago' +\%a` > /build/nightlies/OldLogs/KillClient.log 2>&1

SLS sensor

• A SLS sensor is to be placed for a new machine to monitor its disk usage
• to monitor the build machine, a script collects the information and pushes it to SLS as xml
  • run the monitor from acrontab and let it place the xml onto AFS to be picked up from AFS
    • */20 * * * * buildlhcbXX export PYTHONPATH=/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/python:$PYTHONPATH && /afs/cern.ch/user/l/lhcbsoft/PRODUCTION/LHCbNightliesProd/LHCbNightlies/NightliesSensors/AFSSensor.py --aw /afs/cern.ch/lhcb/software/nightlies/www/LHCb_Nightlies_buildlhcbXX.xml --dirdepth 0 --dirroot /build/nightlies --xmlid LHCb_Nightlies_buildlhcb04 >  /build/nightlies/OldLogs/localdisk.out 2>&1
• prepare the SLS interface
  • the admin page is available under https://apex.cern.ch/pls/htmldb_itcore/f?p=116:LOGIN:888212673570801:::::
  • if you not yet an account for SDB ask Victor for example
    • you will most probable get an account, able to edit existing sensors but not create new ones
      • ask Victor, if he can clone an existing sensor for you
      • when available, you can add the path to the sensor xml (were you let acrontab write to ) to the sensor etc.

local services

• to organize log files in the local build directory logrotate is used
  • normally the general logrotate configuration in /afs/cern.ch/user/l/lhcbsoft/public/PRODUCTION/LHCbNightliesProd/LHCbNightlies/NightliesConfig/logRotateLocal.conf should be fine
    • the local state file is normally placed at /build/nightlies/logrotate.local
  • set up the local crontab, i.e.

    • */2 * * * * date -Is >>  /build/nightlies/meminfo.txt; cat /proc/meminfo >> /build/nightlies/meminfo.txt; ps aux --forest >> /build/nightlies/meminfo.txt; echo "-----------" >> /build/nightlies/meminfo.txt
      55 23 * * * tar cjf /build/nightlies/OldLogs/ProcessesMeminfo.tar.bz2  /build/nightlies/meminfo.txt > /dev/null 2>&1; rm /build/nightlies/meminfo.txt
      55 1  * * * /usr/sbin/logrotate -s /build/nightlies/logrotate.local /afs/cern.ch/user/l/lhcbsoft/public/PRODUCTION/LHCbNightliesProd/LHCbNightlies/NightliesConfig/logRotateLocal.conf

-- ThomasHartmann - 12-Oct-2012