CDR - Central Data Recording

Simone Bifani, Emanuele Leonardi and Stefano Venditti will take over the CDR responsability from Venelin Kozhuharov for the P-326 run in year 2007

CDR mailing list: na48-cdr@cernNOSPAMPLEASE.ch

CDR setup: Castor notes

DAQ

CDR Chain

Machines

• na48cdr machines: lxfsrb6102, lxfsra2406

• na48cdr disk-servers: lxfsra1001, lxfsra1002, lxfsra1003, lxfsra1004
• na48cdr old machine: lxfs6033

• log-in to NA48CDR machine (data transfer & set-up for L3 jobs):

     ssh na48cdr@lxfsrb6102.cern.ch

• log-in to LXPLUS machines (L3 jobs submission to the CERN Batch System):

     ssh na48cdr@lxplus.cern.ch

The na48cdr = lxfsrb6102 alias has been defined.

Directories

Directories have been arranged with the following conventions:

• NA48CDR machine:
  • config: configuration files
  • execs: script files (csh, perl)
  • logfiles: logs info
  • src: program sources
  • status: tag files

• LXPLUS machines:
  • bin: program sources
  • log: logs info
  • status: tag files

e.g.:

• /shift/lxfsrb6102/data02/na48cdr/cdr/src : CDR program sources
• /shift/lxfsrb6102/data02/na48cdr/l3/logfiles : L3 logfiles
• /shift/lxfsrb6102/data02/na48cdr/reprocessing/status : Reprocessing tag files
• /afs/cern.ch/user/n/na48cdr/lsf/bin : Batch monitor program sources

Take a Look at Crontab

• NA48CDR machine:

:~> crontab -l

#1,8,16,24,35,40,48,51,56 *  *  *  * /shift/lxfsrb6102/data02/na48cdr/cdr_na48/execs/cdr_watchdog >> /shift/lxfsrb6102/data02/na48cdr/cdr_na48/logfiles/logfile_watchdog 2>&1
#9 0,2,4,6,8,10,12,14,16,18,20,22 * * * /shift/lxfsrb6102/data02/na48cdr/marco/check_bad_burstlets.pl >> /shift/lxfsrb6102/data02/na48cdr/cdr_na48/logfiles/log.check_bad_burstlets.pl 2>&1
#1 * * * * /shift/lxfsrb6102/data02/na48cdr/venelin/scripts/chk_status.csh >& /dev/null
#1,31 * * * * /shift/lxfsrb6102/data02/na48cdr/venelin/scripts/stop_tape.csh >> /shift/lxfsrb6102/data02/na48cdr/venelin/logs/stop_tape.log 2>&1
#1,8,16,24,32,40,48,56 *  *  *  * /shift/lxfsrb6102/data02/na48cdr/cdr_na48/execs/cdr_watchdog_linux_cdrserver >> /shift/lxfsrb6102/data02/na48cdr/cdr_na48/logfiles/logfile_watchdog_linux_cdrserver 2>&1
#1,8,16,24,35,40,48,56 * * * * /shift/lxfsrb6102/data03/na48db2004/software/scripts/startw2port.pl >> /shift/lxfsrb6102/data03/na48db2004/log/w2port-watchdog.log
#0 * * * * /shift/lxfsrb6102/data03/na48db2004/software/scripts/rcmysql start >> /shift/lxfsrb6102/data03/na48db2004/log/mysql-start.log
#2,17,32,47 * * * * /shift/lxfsrb6102/data02/na48cdr/l3/scripts_2003/html_info.csh > /shift/lxfsrb6102/data02/na48cdr/l3/scripts_2003/info.html
#0 0 * * * /shift/lxfsrb6102/data03/na48db2004/software/scripts/DB-Snapshot.sh run >> /shift/lxfsrb6102/data03/na48db2004/log/db-backup.log
#0 12 * * 3 /shift/lxfsrb6102/data03/na48db2004/software/scripts/DB-Snapshot.sh calibration goldcmp30 goldraw30 >> /shift/lxfsrb6102/data03/na48db2004/log/db-backup.log

• LXPLUS machines:

:~> acrontab -l

#0,6,10,20,30,43,50 * * * * lxplus011 /afs/cern.ch/user/n/na48cdr/lsf/bin/watchdog.csh >> /afs/cern.ch/user/n/na48cdr/lsf/log/watchdog.log 2>&1
#1 * * * * lxplus011 /afs/cern.ch/user/n/na48cdr/lsf/bin/logrotate.csh >> /afs/cern.ch/user/n/na48cdr/lsf/log/logrotate.log 2>&1
#4,19,34,49 * * * * lxplus011 /usr/local/bin/rfcp /shift/lxfsrb6102/data02/na48cdr/l3/scripts_2003/info.html /afs/cern.ch/na48/www/private/L3rtr/info.html >& /dev/null
#1,6,11,16,21,26,31,36,41,46,51,56 * * * * lxplus011 /afs/cern.ch/user/n/na48cdr/public/L3Mon/L3Mon.csh >& /dev/null
#3 * * * * lxplus011 /afs/cern.ch/user/n/na48cdr/venelin/scripts/chk_status.csh >> /afs/cern.ch/user/n/na48cdr/venelin/logs/chk_status.log 2>&1

Check Deamons, Programs & Scripts

• NA48CDR machine:
  • cdr_watchdog: checks all needed deamons, programs and scripts are running on the na48cdr machine
  • CDR_watchdog_linux_CDRserver: copies data files (RAW) from the Online PC-Farm (Meryn) to the Private Disk-Servers (CERN IT)
  • create_burstlet_list_files: looks for all the burstlets that belong to the same burst and when they are available fills a path list file to be used by the L3 filter (00 = burstlets header file, 02-05 = burstlet files)
  • remote_copy_daemon_linux
  • L3_control_pool: looks for a list file and if it is available sends a notification (tag file) to submit an L3 job to process data (GoldRAW)
  • delete_files_rawdata: checks if a burst has been successfully filtered by the L3 and deletes all the processed burstlets
  • check_bad_burstlets.pl
  • chk_status.csh: sends L3 processes info to the Online PC-Farm monitor (control-room)
  • stop_tape.csh
  • startw2port.pl: copies the run status from the DAQ (control-room) to the na48cdr machine (MySQL database)
  • rcmysql: puts processed burst info into the database
  • html_info.csh
  • DB-Snapshot.sh: creates a copy of the database

• LXPLUS machines:
  • watchdog.csh: checks all needed deamons, programs and scripts are running on lxplus machines
  • logrotate.csh
  • info.html
  • L3Mon.csh
  • chk_status.csh

Tag Files

Tag files report scripts status. These flags are usually placed in /shift/lxfsrb6102/data02/na48cdr/cdr/status (na48cdr machine).

• CDR_watchdog_linux_CDRserver: writes a tag for each copied burst in /copy_done and /copy2_done to report a successful transfer while in /incomplete_copy_done for a failed copy
• L3_control_pool: deletes the tags from /copy2_done if the corresponding burst has been processed by L3
• delete_files_rawdata:

Data Streams

L3 processes RAW data (burstlets files) stored in the Private Disk-Servers producing different streams and choosing in which storage element (Private Disk-Servers, CASTOR) save them:

• GoldRAW: burst files without reconstruction
• MainRAW: GoldRAW copy useful for calibration
• GoldCMP: burst files with reconstruction
• COmPACT: GoldCMP copy with extended DCH information
• Calibration: LKr calibration