Run 2 Production Management

This page shall provide special infos for production management of data processing during LHC Run2. For general info on production management see the Production Management Guide

Please also try to keep the spreadsheet at https://social.cern.ch/me/roiser/_layouts/15/WopiFrame.aspx?sourcedoc=/me/roiser/Documents/Run2-DataProcessing.xlsx&action=default up to date with the latest status of productions.

Which productions to setup

  • So far we only have setup "validation" productions, these are writing output into the "/validation" configuration of the Bookkeeping. For the "50 ns rampup" data we start to use the "/LHCb" configuration
  • Per magnet polarity there are 7 requests to handle which generate 10 transformations. If a new request needs to be made the easiest will be to "dulicate" (as lhcb_prmgr) an old request which is close to the new one and make the necessary changes in there.

Things left to do - but currently cannot be done

  • In /validation we have not processed yet successfully the MagUp data (run range 153603 to 154140), before doing this we need to
    • wait for the DQ flags to be set BAD for the “first 3 MagUp validation runs” by Marco, i.e. runs 153759, 153770, 154030, please wait for his confirmation
    • Note we had another issue of not receiving MagUp runs from last weeks HLT2 processing, looking at the BK right now I see that we have additional runs, so this seems fixed
    • For the actual processing I propose to do launch new productions from the requests below (info also available in the excel spreadsheet linked above):
      • 90000000 - full stream
        • requests 27526 (Reco15b) & 27527 (Stripping22)
      • 94000000 - turbo stream
        • request 27646 (Turbo01c), turbo validation requests 27649 (Reco15b/Turbo01c), 27650 (Stripping22)
      • 95100000 - turcal stream
        • request 27652 (Reco15b/Turbo01c)
      • 96000000 - nobias stream
        • request 27535 (Reco15b)
      • Note: for the Full, Turcal and Nobias streams productions have already been launched which have not created tasks/jobs yet, but they have picked up the "wrong" files (runs 153759, 153770, 154030) as Unused files. In case the changing of the DQ flag for these runs removes them also from the productions (I don't know) these productions coudl also be re-started, but this is risky. Better clean them and launch new ones as suggested.
  • For the EM data processing in /LHCb currently new versions of DaVinci are released, after the current validation (especially of MagDown data) has finished one probably has a clearer picture on which steps to use for the different EM processings.
    • Colleagues to contact for the different steps in the workflows are
      • Reconstruction: Marco Cattaneo, Michel De Cian
      • DQ: Marco Adinolfi
      • Stripping: Andrea Contu, Ricardo Vazquez
      • Turbo: Sean Benson
    • The only thing most probably known at the moment is that we'll use conditions tag "cond-20150625" which needs to be set by the step providers above
    • Note: I have not created any requests yet for EM processing. IIRC if I create a request it cannot be handled further by another production manager, so you will need to do this yourself.

Additional info

  • Request Management is currently only available in the "old" portal, i.e. https://lhcb-web-dirac.cern.ch/DIRAC , every other operation can be executed in the new portal http://lhcb-portal-dirac.cern.ch/DIRAC/
    • Note: the Request Manager has now been ported to the new portal, you may try it out, a few fixes here and there are being done.
  • Some presentations about production management statuses were done in the PPTS meeting
  • In the "everytingElse" template when specificying the production types don't leave a space between those, e.g. correct "Turbo,Merge", otherwise the production type is not found
  • in some productions the histograms need to be uploaded explicitly to the "CERN-HIST-EOS" space token, the quotes around the strings in this option need to be double quotes, single quotes don't work, see also screenshots

some notes about the screenshots

  • the screenshots of the production generation templates below show the validation production generations for MagDown data of one of the last tests, everything with a "red triangle" is an option which has changed from the default. These changes can be taken for the early measurements productions - except
    • the run ranges, if specified
    • the line "GENERAL: Set true for validation prod" needs to be left at the default "False" value such that we output into the "LHCb" configuration in the Bookkeeping instead of "/validation"
      • of course if we need more validation productions before the EM processing this needs to be set to "True" again wink

For each of the workflows there is also a diagram with explanations of the requests, transformations, steps and output file formats attached.

"Early Measurements" Workflows

90000000 - Full

Notes:

  • This is the "normal" processing we have also done during Run1, i.e. Reconstruction & Stripping/Merging
  • If the Reconstruction outputs into a new processing pass you need to wait for at least the first Reco job to finish such that the complete BK path is generated. This path is then needed for the input data selection in the Stripping request
  • Templates for both production generations are is the "RecoStripping_run.py" one
  • For Stripping we are running with 1 input file, see also screenshot where group size = 1

Workflow Diagram

Run2_-_Full_Stream_Processing.jpg

Production Generation Screenshots

Reconstruction:

20150625-Full-Reco15b-MagDown-Validation.png

Stripping/Merging:

20150625-Full-Stripping22onReco15b-MagDown-Valdiation.png

94000000 - Turbo

Workflow Diagram

Note: the left "stream" is the "Turbo Processing", the right one is "Turbo Validation"

Run2_-_Turbo_Stream_Processing.jpg

"Turbo Processing"

Notes:

  • In the generation of the Turbo production make sure there is NO space between "Turbo,Merge" in the generation template, otherwise the production type is not found by the Dirac agents
  • Production generation template is "everyThingElse_run.py"

Production Generation Screenshot

20150625-Turbo-Turbo01b-MagDown-Validation.png

"Turbo Validation Processing"

Notes:

  • Production generation templates are "everThingElse_run.py" for the Reconstruction and "RecoStripping_run.py" for the Stripping
  • For Stripping we are running with 1 input file, see also screenshot where group size = 1

Production Generation Screenshots

Reconstruction:

20150625-TurboVal-Reco15b_Turbo01b-MagDown-Validation.png

Stripping:

20150625-TurboVal-Stripping22onReco15b_Turbo01b-MagDown-Valdiation.png

95100000 - Turcal

Note:

  • Production generation template is "everThingElse_run.py"

Workflow Diagram

Run2_-_Turcal_Stream_Processing.jpg

Production Generation Screenshot

20150625-TurboCalibration-Reco15b_Turbo01b-MagDown-Validation.png

96000000 - Nobias

Note:

  • Production generation template is "RecoStripping_run.py"

Workflow Diagram

Run2_-_Nobias_Stream_Processing.jpg

Production Generation Screenshot

20150625-NoBias-Reco15b-Validation.png

Run II "post TS2" workflows

Checklist (Wishlist)

1. Get the right runs from the Online team. Good runs should be marked as "OFFLINE IN BKK" in Run DB https://lbrundb.cern.ch/

  • Would be nice to extract runs from the Run DB following LHC schedule, e.g. post TS2 or post MD1
2. Collect the steps for the several workflows. Currently we create transformations for:
  • Reconstruction of FULL stream (data type 90000000)
    • This transformation requires two step, one for Reco and one for DQ
  • Turbo Processing (data type 94000000)
    • This transformation requires two steps, one for Turbo and one for Merging
  • Turbo Calibration (data type 95100000)
    • This transformation requires two steps, one for Reco and one for Turbo Calibration
  • Stripping on FULL stream
    • This transformation requires two steps, one for Stripping data and one for Merging
  • Steps are available from the Step Manager. It would be nice that, once we declare an incoming campaign for data processing, people in charge of preparing these steps are promptly informed in order to have the steps ready in a reasonable time interval (e.g. 7 days or so). In this way people could check that all the relevant applications involved in the productions (Brunel, DaVinci, Tesla...) are available and with the right version.
3. Duplicate a given transformation/request. Find out the latest transformation for the desired data type (e.g. 90000000), duplicate it and clean the processing steps, which are likely to be outdated. Insert the right steps for the desired production.

4. Sign the request.

  • would be nice if the prmgr role could skip the sign procedure and go directly to the Generation part, especially considering that the new portal clears the selections, changing the role, and force the user to select again the right transformation request
5. The Generation steps are almost the same as the EM workflows, except for the Stripping one, where you should put 1 in the field PROD-2: Ancestor depth to allow the production to navigate back to RAW data.
  • For the next round of productions we'll need to add “MDST.DST”:”Tier1-BUFFER”

-- StefanRoiser - 2015-06-24

Edit | Attach | Watch | Print version | History: r15 | r8 < r7 < r6 < r5 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r6 - 2015-09-30 - MarcoCorvo
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback