Run 2 Production Management

This page shall provide special infos for production management of data processing during LHC Run2. For general info on production management see the Production Management Guide

Please also try to keep the spreadsheet at up to date with the latest status of productions.

Which productions to setup

  • So far we only have setup "validation" productions, these are writing output into the "/validation" configuration of the Bookkeeping. For the "50 ns rampup" data we start to use the "/LHCb" configuration
  • Per magnet polarity there are 7 requests to handle which generate 10 transformations. If a new request needs to be made the easiest will be to "dulicate" (as lhcb_prmgr) an old request which is close to the new one and make the necessary changes in there.

Things left to do - but currently cannot be done

  • In /validation we have not processed yet successfully the MagUp data (run range 153603 to 154140), before doing this we need to
    • wait for the DQ flags to be set BAD for the “first 3 MagUp validation runs” by Marco, i.e. runs 153759, 153770, 154030, please wait for his confirmation
    • Note we had another issue of not receiving MagUp runs from last weeks HLT2 processing, looking at the BK right now I see that we have additional runs, so this seems fixed
    • For the actual processing I propose to do launch new productions from the requests below (info also available in the excel spreadsheet linked above):
      • 90000000 - full stream
        • requests 27526 (Reco15b) & 27527 (Stripping22)
      • 94000000 - turbo stream
        • request 27646 (Turbo01c), turbo validation requests 27649 (Reco15b/Turbo01c), 27650 (Stripping22)
      • 95100000 - turcal stream
        • request 27652 (Reco15b/Turbo01c)
      • 96000000 - nobias stream
        • request 27535 (Reco15b)
      • Note: for the Full, Turcal and Nobias streams productions have already been launched which have not created tasks/jobs yet, but they have picked up the "wrong" files (runs 153759, 153770, 154030) as Unused files. In case the changing of the DQ flag for these runs removes them also from the productions (I don't know) these productions coudl also be re-started, but this is risky. Better clean them and launch new ones as suggested.
  • For the EM data processing in /LHCb currently new versions of DaVinci are released, after the current validation (especially of MagDown data) has finished one probably has a clearer picture on which steps to use for the different EM processings.
    • Colleagues to contact for the different steps in the workflows are
      • Reconstruction: Marco Cattaneo, Michel De Cian
      • DQ: Marco Adinolfi
      • Stripping: Andrea Contu, Ricardo Vazquez
      • Turbo: Sean Benson
    • The only thing most probably known at the moment is that we'll use conditions tag "cond-20150625" which needs to be set by the step providers above
    • Note: I have not created any requests yet for EM processing. IIRC if I create a request it cannot be handled further by another production manager, so you will need to do this yourself.

Additional info

  • Request Management is currently only available in the "old" portal, i.e. , every other operation can be executed in the new portal
    • Note: the Request Manager has now been ported to the new portal, you may try it out, a few fixes here and there are being done.
  • Some presentations about production management statuses were done in the PPTS meeting
  • In the "everytingElse" template when specificying the production types don't leave a space between those, e.g. correct "Turbo,Merge", otherwise the production type is not found
  • in some productions the histograms need to be uploaded explicitly to the "CERN-HIST-EOS" space token, the quotes around the strings in this option need to be double quotes, single quotes don't work, see also screenshots

some notes about the screenshots

  • the screenshots of the production generation templates below show the validation production generations for MagDown data of one of the last tests, everything with a "red triangle" is an option which has changed from the default. These changes can be taken for the early measurements productions - except
    • the run ranges, if specified
    • the line "GENERAL: Set true for validation prod" needs to be left at the default "False" value such that we output into the "LHCb" configuration in the Bookkeeping instead of "/validation"
      • of course if we need more validation productions before the EM processing this needs to be set to "True" again wink

For each of the workflows there is also a diagram with explanations of the requests, transformations, steps and output file formats attached.

"Early Measurements" Workflows

90000000 - Full


  • This is the "normal" processing we have also done during Run1, i.e. Reconstruction & Stripping/Merging
  • If the Reconstruction outputs into a new processing pass you need to wait for at least the first Reco job to finish such that the complete BK path is generated. This path is then needed for the input data selection in the Stripping request
  • Templates for both production generations are is the "" one
  • For Stripping we are running with 1 input file, see also screenshot where group size = 1

Workflow Diagram


Production Generation Screenshots





94000000 - Turbo

Workflow Diagram

Note: the left "stream" is the "Turbo Processing", the right one is "Turbo Validation"


"Turbo Processing"


  • In the generation of the Turbo production make sure there is NO space between "Turbo,Merge" in the generation template, otherwise the production type is not found by the Dirac agents
  • Production generation template is ""

Production Generation Screenshot


"Turbo Validation Processing"


  • Production generation templates are "" for the Reconstruction and "" for the Stripping
  • For Stripping we are running with 1 input file, see also screenshot where group size = 1

Production Generation Screenshots





95100000 - Turcal


  • Production generation template is ""

Workflow Diagram


Production Generation Screenshot


96000000 - Nobias


  • Production generation template is ""

Workflow Diagram


Production Generation Screenshot


Run II "post TS2" workflows

Checklist (Wishlist)

1. Get the right runs from the Online team. Good runs should be marked as "OFFLINE IN BKK" in Run DB

  • Would be nice to extract runs from the Run DB following LHC schedule, e.g. post TS2 or post MD1
2. Collect the steps for the several workflows. Currently we create transformations for:
  • Reconstruction of FULL stream (data type 90000000)
    • This transformation requires two step, one for Reco and one for DQ
  • Turbo Processing (data type 94000000)
    • This transformation requires two steps, one for Turbo and one for Merging
  • Turbo Calibration (data type 95100000)
    • This transformation requires two steps, one for Reco and one for Turbo Calibration
  • Stripping on FULL stream
    • This transformation requires two steps, one for Stripping data and one for Merging
  • Steps are available from the Step Manager. It would be nice that, once we declare an incoming campaign for data processing, people in charge of preparing these steps are promptly informed in order to have the steps ready in a reasonable time interval (e.g. 7 days or so). In this way people could check that all the relevant applications involved in the productions (Brunel, DaVinci, Tesla...) are available and with the right version.
3. Duplicate a given transformation/request. Find out the latest transformation for the desired data type (e.g. 90000000), duplicate it and clean the processing steps, which are likely to be outdated. Insert the right steps for the desired production.
  • The flag of DataQuality must be set to OK+UNCHECKED
  • Note that this is true only for first pass processing, that is for the very first full processing chain. In case of reprocessing the DQ flag must be set to OK. The reason is that DQ guys are not always "fast enough" to flag bad data so, as a first approximation, we process all data. If then they decide to flag something BAD, in the second pass we can avoid picking those bad runs.

4. Sign the request.

  • would be nice if the prmgr role could skip the sign procedure and go directly to the Generation part, especially considering that the new portal clears the selections, changing the role, and force the user to select again the right transformation request
5. The Generation steps are almost the same as the EM workflows, except for the Stripping one, where you should put 1 in the field PROD-2: Ancestor depth to allow the production to navigate back to RAW data.
  • For pXX (where XX is Ar, He, Ne) Reconstructed data should go directly into Tier1-DST, in case a further stripping processing is needed, otherwise they go in the usual Tier1-BUFFER destination
  • For the next round of productions we'll need to add "MDST.DST":"Tier1-RDST" as an optional SE, while the other streams will go to Tier1-DST.

How to recover lost files

To reproduce some lost files these are the steps to perform:

1. Duplicate the request from the original production (48237 for the lost RDSTs) keeping the processing steps

2. Submit and sign as usual (tech, ppg)

3. Edit the production filling the usual parameters and avoiding to provide the run range in case of reco prods

4. DO NOT push the "generate" button, but save the Python script (from ScriptPreview) somewhere on your machine, calling it, say,

5. Open and modify the saved script commenting out the line

# pr.bkQueries = ['Full']
6. and adding
  pr.bkQueries = ['']

WARNING: the lines need to be modified in the right place, e.g. under w1 for DataReconstruction or under w2 for DataStripping

7. Open a shell on lxplus and issue the usual commands to work with LHCbDirac in a Grid environment

lb-run LHCbDirac/latest lhcb-proxy-init

8. launch the script via:

lb-run LHCbDirac/latest python

This will output a production number

9. Add input files with the dirac command:

lb-run LHCbDirac/latest dirac-production-add-files <prod number> <-l LFN> or <--File file-with-lfns.txt>

MarcoCorvo - 2016-02-29

This topic: LHCb > WebHome > LHCbComputing > Run2ProductionManagement
Topic revision: r11 - 2016-08-16 - FedericoStagni
This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback