Group Production on the ATLAS Production System

Introduction

This page describes the group production on the ATLAS production system.

Group production coordination

Name Roles
Junji Tojo Group production coordination
Stephane Jezequel ADCoS
Borut Kersevan Production team
Alexei Klimentov ADCoS
Pavel Nevski Production system, ADCoS

Contact persons from groups and related links

Group Subgroup Contact person Links
ID Pixel Stephen Gibson -
Beam spot - James Walder -
egamma - Andrea Bocci -
Jet/MET - - -
muon - Magda Anna Chelstowska -
B-Tagging - Johanna Fleckner -
SM WZ - -
SUSY R-Parity Violating/Long-lived Nick Barlow, Christian Ohm TWiki documentation
Exotics - John Butler -
Trigger L1Calo John Morris -

Work flow in the group production

  • Group conveners/leaders appoint contact persons for the group production and let the coordinator know.
  • The coordinator works together with the contact persons in the table above, so that the group production can be run on the production system.
  • The contact persons work on the following items.
    • Prepare/arrange packages necessary for the group production, e.g. D3PDMaker and PATJobTransforms etc, which have to be included into AtlasProduction caches and/or Physics Analysis Projects
    • If there are new output data types, e.g. NTUP_XXX in Reco_trf.py, give the information to the coordinator.
    • Create a production tag if necessary.
    • Submit validation tasks and validate a production tag, if the production tag is new.
    • Send the production requests to the coordinator.
    • Check the availability of the group disks, when submit validation tasks or send the production requests.
    • If the production for the group runs routinely, task submission for bulk requests can be done by the contact persons (confirmed by the coordinator).
    • If problems are found during the production, the contact persons are asked by the coordinator to investigate them.
  • The coordinator works on the following items.
    • If there are new output data types, get the information from contact persons and get in touch with the JobTransforms (trf) experts for the modification of output list in the trf.
    • Register new output data types into AMI.
    • Receive (bulk) production requests from contact persons.
    • Submit tasks for the requests.
  • Tasks/jobs monitoring
    • The monitoring is done by the ADCoS shifters.
    • If problems are found, those are filled into Savannah. The contact persons are asked to investigate them.

Details

Input datasets available for the group production

  • Available datasets as input
    • Datasets produced on the production system, including both data and MC.
    • Datasets produced at Tier0 and replicated to Tier1s.
    • Statistical extension of centrally produced MC datasets can be requested.
    • The data types include ESD, AOD, DESDM_EGAMMA, DESDM_CALJET, DESD_MINBIAS and so on.
  • AMI dataset search will help to find datasets.
  • Contact persons need to specify a list of full dataset names in the request form. Examples are below.
    • Datasets produced at Tier0
      • data10_7TeV.00155160.physics_L1Calo.recon.ESD.f255
      • data10_7TeV.00155160.physics_L1Calo.merge.DESDM_EGAMMA.f255_m472
      • data10_7TeV.00155160.physics_MinBias.merge.DESD_MBIAS.f255_m472
    • Datasets from reprocessing
      • data10_7TeV.00155160.physics_L1Calo.merge.ESD.r1299_p161
      • data10_7TeV.00155160.physics_L1Calo.merge.DESDM_EGAMMA.r1299_p161_p165
      • data10_7TeV.00155160.physics_MinBias.merge.DESD_MBIAS.r1299_p161_p165
    • Datasets from MC production
      • mc09_7TeV.105001.pythia_minbias.recon.ESD.e517_s764_s767_r1302
      • mc09_7TeV.105802.JF17_pythia_jet_filter.merge.AOD.e505_s765_s767_r1302_r1306

Output datasets and destination sites

  • All the outputs are shipped to sites which have group disks (e.g. PHYS-SM) and are NOT saved on disks used for the ATLAS central production.
  • The group disks and the corresponding sites can be found in the link below.
  • Contact persons need to specify destination site(s) (e.g. AGLT2_PHYS-SM site as one of PHYS-SM group disks) in the request form, which have enough disk space to store outputs.

Output data types

  • Output data types, e.g. NTUP_EGAMMA, NTUP_MINBIAS, DESDM_EGAMMA, DESDM_CALJET, DESD_MINBIAS etc, are defined in Reco_trf.py, which is used in most of the group production
  • If there are new output data types for a group production, please let the coordinator know.
  • The coordinator will get in touch with the experts to include the new output types into Reco_trf.py and will register them into AMI.

New production tag

  • New production tag can be created by contact persons.
  • Procedure
    • Register your user ID in Registration Page.
    • Go to New Tag Creation Page
    • Fill the form to specify JobTransforms and AtlasProduction/PhysicsAnalysisProject cache. An example here is for Reco_trf.py using AtlasProduction-15.6.9.13.
    • Click "continue". Then, you will be navigated to the form to specify JobTransforms parameters.
    • Fill JobTransforms parameters. An example here is for ESD->NTUP_EGAMMA.
    • Select "debug" and click "continue". Then, you might get hints.
    • Select "create" and click "continue". Then, you will get a new tag.
  • Example tags can be searched in the following links.

Request form

  • Example 1 : a request form for ESD->NTUP production on data
  • Example 2 : a request form for ESD->NTUP production on MC
  • Example 3 : a request form for ESD->NTUP production on data with open-ended run number
  • More details later.

Validation task submission

  • Task submission can be done using panda.
  • Fill out the form like this:
    • Project: data10_cvalid for data or valid1 for MC
    • Dataset: paste a link to a text file that describes your task (format to be more carefully documented, but for now have a look at this template or contact Junji directly). Please note that the text file must end with a new (empty) line.
    • Transformation Type: pick the appropriate option that describes your task
    • Transformation Version: (leave this as it is for now, the release will be chosen based on the production tag you supply)
    • Configuration Tag: enter the production tag you want to use for this tag
    • userid: your production system user name
  • Click "Continue"
  • Check that everything is ok on the page that appears, and enter the security code in the box right above where it appears, and click the "TaskSubmit" button (if you get an error message saying "wrong key, go back and try again", start over from scratch with the task request and it should work).

Links

Edit | Attach | Watch | Print version | History: r17 < r16 < r15 < r14 < r13 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r17 - 2010-07-27 - Ohm
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback