Introduction

A set of scripts is in place to run the offline HLT Trigger Monitoring on Bytestream files that have been run through the HLT offline by the hdebug package ( see https://twiki.cern.ch/twiki/bin/view/Atlas/OfflineHLT ).

The jobs that run over the BS are based on the setup and JobOptions used for processing the data at Tier0. The code that fills the monitoring histograms is contained in two packages: TrigHLTMonitoring TrigHLTOfflineMon. The former will run on Tier0, once HLT is being run online. For more information on these packages, see https://twiki.cern.ch/twiki/bin/view/Atlas/HLTMonitoringPage

Setup

The shifter will be working from the trigcomm acount, under the directory /afs/cern.ch/user/t/trigcomm/offline_monitoring To set up, do:

source ~/offline_monitoring/setup.sh
(currently, this sets up a recent Tier0 nightly). The same command is called by the jobs that are submitted to the caf.

How to run jobs.

A set of scripts is in place in ~trigcomm/offline_monitoring/script to facilitate submitting monitoring jobs. Below, we explain their use.

  • From the hlt running webpages, identify a run and version number you would like to run on. For example, let's say you want to run monitoring on run 88069-v2 (the -v2) is a postfix to identify which HLT. Now, run
      ./get_files_list_for_run.sh 88069-v2 
      
    This will look in the directory where hdebug writes it's output files the directory "88069-v2", and produce a lists of files, split up by stream; e.g. ~/offline_monitoring/scripts/jobs/input_files_88069-v2-BPTX. (You normally don't have to worry about these files) It will also check get the relevant trigger configuration xml files from the trigger database and put them in ~/offline_monitoring/xml.

  • The script above also prints a suggestion about which jobs to submit. In this case, for each stream, you have something like
bsub -J "mon[1-20]" -oo /afs/cern.ch/user/t/trigcomm/offline_monitoring/logfiles/stdout_88069-v2-BPTX_%J_%I -eo /afs/cern.ch/user/t/trigcomm/offline_monitoring/logfiles/stderr_88069-v2-BPTX_%J_%I  -q atlastrig /afs/cern.ch/user/t/trigcomm/offline_monitoring/scripts/do_monitoring.sh 88069-v2-BPTX 10

* You can now submit the jobs. If you're happy with the suggestion from above, just do:

./get_files_list_for_run.sh 88069-v2 | grep ^bsub | sh
The script that is run in the batch jobs is /offline_monitoring/scripts/do_monitoring.sh. It takes 3 or 4 parameters:
    • run+version+stream id (e.g. 88069-v2-BPTX). This is used to determine the input files.
    • run+version only (e.g. 88069-v2). This is used to determine which xml file should be used
    • N_files_per_job: How many input files to process in one batch job. (this determines how many jobs you should submit. Jobs should be submitted as an lsf array so that each job knows which set of files it must process)
    • [optional] If the 4th argument is do "dorec", the reconstruction will be turned on the athena job. This will produce esd files and also allow for monitoring tasks that use the reconstruction results. The script runs the JO file $TestArea/run/do_rec.py. So please be careful modifying things in the current $TestArea (e.g. ~offline_monitoring/testareas/AtlasTier0-rel_2). It is not write protected, since shifters will need to make changes. For debugging, the script can also run from the command line in ~trigcomm/offline_monitoring/. Then it will run in /tmp/monitoring_job/run.

  • When jobs are running, You can monitor, kill, suspend etc the jobs in the regular way with bjobs.

  • When (some of) the jobs have finished, you can run the post-processing, like
./post_proc 88069-v2.
This does the following
    • merges root histograms form different jobs that have run on the same stream
    • produce the offline dqfm plots and writes html for them
    • copies the files (html plots, ntuples requested by slices and histogram files) to the web area on v0atlas17.

Where the output goes.

The output of the monitoring jobs is writting to castor for long-term storage

  • /castor/cern.ch/user/t/trigcomm/offmon/RUNNR-STREAM/esd for esd files (if any)
  • /castor/cern.ch/user/t/trigcomm/offmon/RUNNR-STREAM/mon for the monitoring histograms

The monitoring histograms are also written to ~trigcomm/offline_monitoring/histograms for easy access and processing by the 'post_proc.sh' script.

The web index for the offline monitoring part is at http://voatlas17.cern.ch/offmon/ For each processed run, and for each stream there is a directory giving access to the offline dqfm results and the root file with the monitoring histograms.

-- AartHeijboer - 16 Sep 2008

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2008-09-17 - AartHeijboer
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback