-- AntonioFalabella - 08-Dec-2011

Working Groups Customized Productions

Introduction

LHCbDIRAC is used to run centralized processing tasks on selected datasets. A processing task in LHCbDIRAC jargon is referred to as a Production. A Production is set up keeping in mind the abstract concept of workflow. A workflow is basically a sequence of atomic processing steps that run specific LHCb software on a certain list of input data. Each step produces its output files so that within the workflow the output of each step feed in the next step. With LHCbDIRAC It is than possible to setup a production request with a customized workflow to meet the needs of working groups.

This possibility can be helpful in situations where some task need to be performed on a well defined dataset in the bookkeping. The advantages for a working group to have its own centralized productions are not negligible. First of all they can run on all the data present in the bookkeeping basically at run time without keeping track of new files published, because when a production have been started it will gather new file of the same type as they are produced. The result are then published in the bookkeeping so that everyone can use them. Last but not least, when your are sure that your code works fine the control is in the hands of the prodution team that take care of the technical aspects.

Working Group Software Package

COMMENT FROM DAN: Suggest that this section be simplified. In the simplest case, code can simply be released in WG/{yourWG}Config. We should add simple instructions to tag_package WG/{yourWG}Config and then the step can be defined with respect to that "{yourWG}Config.vXrY" release after deployment.

At the time being LHCbDIRAC can run software that uses a release of the following main projects: Gauss, Boole, Brunel, DaVinci, LHCb and Moore. LHCbDIRAC runs the code with the aid of python options. The python options coded by the working groups must be released in a package that is known by SetupProject. So for example let's suppose you write your_script.py that uses the package extrapackage of the project Project . You will release your code such that the following lines can be issued:

> SetupProject <Project>/<version> <command> <command args...>
> gaudirun.py path/your_script.py
where path means the path of your script in the lhcb releases tree.

So let's make a working example. We have the python options for the Erasmus package. You can set the right environment by issueing:

> SetupProject --use=ProdConf --use=PIDCalib Erasmus v13r5p2  

Some notes about software

Note the presence of the option --use=ProdConf, this sets up the prodconf package along with your application package. Although this is not strickly needed to run interactly it is used by LHCbDirac to determine some parameters at runtime. Among the others the filenames and filetypes. Anyone who want to run a production should test the code against prodconf in order to simulate the LHCbDirac behaviour.

The example above refers to SetupProject and not to the new environament tools because the compatibility is still not full.

Setting Up a New Step

Through the LHCbDirac web app you can define a new step to for your application

The step can be built by everyone that has lhcb_tech rights access to LHCbDIRAC web portal which can be reached through this link: To access this page you need to load in your browser a valid grid certificate. In the example below you can see the step manager page.

test-0

Click on New on the top bar to access the new step configuration form

test-0
Here you have to fill in the entries corresponding to your case:

  • Name: Some string meaningful for you (please put here only alphanumeric characters - better if you also avoid blank spaces e.g. WG_Selection_DVv29r2_CPConfigv1r5)
  • Processing Pass: Something meaningful that remind your input dataset (e.g. RecoXX-StrippingXX) COMMENT FROM DAN: I was requested to change this to something meaningful like, in my case, "WG B2OC AmAn DDhDalitz Selection2"
  • Application: Here you can choose between the list of possible projects and versions.
  • Option Files: This the path of your script. (i.e. whatever you would need to put on the command line to gaudirun.py)
  • Extra packages: Here you have to issue the package that contain your script. The syntax here is that of SetupProject i.e. Package.vxry. (in the simplest case, e.g. "B2OCConfig.v1r34")
  • Runtime project: Additional project that can be used. Check the SetupProject page for the list of projects. (can be left blank)
  • CondDB: Choose the conditions CondDB tag.
  • DDDB: Choose the conditions DDDB tag.
  • Visible: Visibility flag. If it's Y a folder containing the processing pass of the step will be created in the bookkeeping.
  • Usable: Y if you want to use the step. If you plain to update your step with a new one it's better to set this flag to obsolete so that the step will no more appear when creating request production( to understand why read the next paragraph production request paragraph )
  • Input File Type: In this form you can choose the file name of input files. The file name are registred in bookkeeping so you can choose an existing one or Add a brand new file name. Usually you don't need to create a new filename for the input because you pick up file from bookkeeping and by definition they are already there.
  • Output File Type: As before you can choose an existing filename or create a brand new one that will be registered in the bookkeeping. It's reasonable here that you need to input your filename, so even if there is no particular rule yet it is recommended to follow the typing schema of the other filenames. COMMENT FROM DAN This must match a line in your DaVinci script like "TupleFile = "B2OC_AMAN_B2DDHDALITZSEL.ROOT""

At this point you can click on Save and your step will appear in the list of Registered Steps.

Testing a new step

See https://twiki.cern.ch/twiki/bin/view/LHCb/ProdReqPreLaunchTest

Submit a Production Request

When your step or steps are ready you can submit a new Request. On the LHCbDirac web app click on Production Request to access the production requests manager and click on

test-0

To submit a new request you have to login as lhcb_user. This can done from the Settings menu on the bottom left part of the page.

Doing this a New request button will appear, click on it and choose the production type.

test-0

After this the New_request form will be opened:

test-0

Request pad

  • Name: Some string meaningful for you (please put here only alphanumeric characters - better if you also avoid blank spaces e.g. Name_of_the_WG_Selection_Additional_infos)
  • Information also: Here you can add some email of people that you think they have to know about your production. By default all people of the production team receive an email when a production request has been submitted.

test-0

Input data pad

  • Select from BK: First select your input data from the bookkeeping.
  • DQ flag: Here select OK.
  • Production: Here you can select a particular production on which run your code. If you are interested on all the productions select ALL.
  • TCKs: Here you can even choose to run on some particular trigger configurations. Choose 0 if you need to run on all the possible configurations.

Processing pass() pad Here you add the step (or steps) you prepared before. If you click on Add step the system list you all the steps with the input type specified in the Input data pad. This will work the same for the others steps. The system let you choose the steps compatible with the output data type of the previous step.

test-0

Event pad_ Choose how many events you want to run (usually -1 - all events).

Comments Here you can write some comments useful for the production team. So far the most important additional information that the prodution team need to know is the method used for writing the output. For example in the Comments pad you would write:

Request to be run on small fraction of file for validation.

test-0

When you will finish you can "Save without submission" for later changes or "Submit to the production team".

At this the production team take care of the technical details, make some preliminary tests.

-- AntonioFalabella - 09-Jan-2012

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng New_Request.png r1 manage 82.9 K 2017-01-27 - 17:05 AntonioFalabella  
PNGpng New_Request_2.png r1 manage 105.1 K 2017-01-27 - 16:53 AntonioFalabella  
PNGpng New_Request_add_step.png r1 manage 98.0 K 2017-01-27 - 16:53 AntonioFalabella  
PNGpng New_Request_bkk.png r1 manage 72.4 K 2017-01-27 - 16:53 AntonioFalabella  
PNGpng New_Request_submit.png r1 manage 117.2 K 2017-01-27 - 16:53 AntonioFalabella  
PNGpng New_Request_type.png r1 manage 75.5 K 2017-01-27 - 16:53 AntonioFalabella  
PNGpng New_Step.png r1 manage 48.1 K 2011-12-08 - 14:41 AntonioFalabella  
PNGpng New_request_form.png r1 manage 54.0 K 2011-12-13 - 10:45 AntonioFalabella  
PNGpng New_request_form2.png r1 manage 61.1 K 2011-12-13 - 12:07 AntonioFalabella  
PNGpng New_request_form3.png r1 manage 118.1 K 2011-12-13 - 17:08 AntonioFalabella  
PNGpng Prod_type.png r1 manage 188.1 K 2011-12-13 - 10:26 AntonioFalabella  
PNGpng Request_manager.png r1 manage 187.6 K 2011-12-13 - 10:12 AntonioFalabella  
PNGpng Step_1.png r1 manage 106.2 K 2017-01-27 - 16:26 AntonioFalabella  
PNGpng Step_2.png r1 manage 108.3 K 2017-01-27 - 16:28 AntonioFalabella  
PNGpng Step_Manager_FINAL.png r2 r1 manage 59.3 K 2012-01-09 - 15:17 AntonioFalabella  
PDFpdf Tutorial-Training.pdf r1 manage 2895.9 K 2012-10-25 - 17:19 FedericoStagni Training tutorial slides
PNGpng request_FINAL.png r1 manage 100.7 K 2012-01-09 - 15:28 AntonioFalabella  
PNGpng step_manager_page.png r1 manage 142.3 K 2011-12-08 - 14:31 AntonioFalabella  
Edit | Attach | Watch | Print version | History: r19 < r18 < r17 < r16 < r15 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r19 - 2018-02-19 - DanielJohnson
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback