Difference: ProductionAPI (3 vs. 4)

Revision 42009-04-09 - StuartPaterson

Line: 1 to 1
 
META TOPICPARENT name="ProductionProcedures"
-- StuartPaterson - 08 Apr 2009
Line: 173 to 173
 

Running any XML workflow locally

Changed:
<
<
Once you have an XML workflow this section explains how to run it locally. The only caveat to mention here is that it has only been tested in the DIRAC Env scripts environment (some configuration changes may be required for this to work in the SetupProject Dirac case e.g. DIRACROOT, DIRACPYTHON, defining local SEs / DIRAC site name).
>
>
Once you have an XML workflow file this section explains how to run it locally. The only caveat to mention here is that it has only been tested in the DIRAC Env scripts environment (some configuration changes may be required for this to work in the SetupProject Dirac case e.g. DIRACROOT, DIRACPYTHON, defining local SEs / DIRAC site name).
  The following simple script will allow to run any workflow XML file (user / production / sam job XML from an input sandbox) locally:
Line: 191 to 191
  The only caveat to mention here is that the workflow library specified in the production is not currently downloaded (this can be done by hand with a dirac-dms-get-file or the local version is picked up). Also for testing workflows with input data files they must be accessible locally wink
Added:
>
>
For production jobs the finalization modules are disabled by default such that uploading output data / publishing BK records is not performed during local testing. Setting the JOBID environment variable to an integer will enable these modules if desired (but be especially careful in the Merging production case because input files would be removed).
 

The production create() method

create() will allow to publish the production to the production management system and to the BK. This currently relies on the conditions information being present in the worklow. Production parameters can also be added at this point (to be implemented). The create() method takes the following arguments and default values:

Line: 297 to 299
 
(DIRAC3-production)[paterson@lxplus223 ~]$ cat insertBKPass4614.py 
# Bookkeeping publishing script created on Wed Apr  8 12:34:44 2009 by
Changed:
<
<
# by $Id: ProductionAPI.txt,v 1.3 2009/04/08 21:35:50 StuartPaterson Exp $
>
>
# by $Id: ProductionAPI.txt,v 1.4 2009/04/09 15:39:08 StuartPaterson Exp $
 from BookkeepingClient import BookkeepingClient bkClient = BookkeepingClient() bkDict = {'Production': 4614,'Steps': {'Step1': {'ApplicationName': 'Boole', 'ApplicationVersion': 'v16r3', 'ExtraPackages': '', 'DDDb': 'head-20081002', 'OptionFiles': 'Boole-2008.py', 'CondDb': 'head-20081002'}, 'Step0': {'ApplicationName': 'Gauss', 'ApplicationVersion': 'v35r1', 'ExtraPackages': 'AppConfig.v1r1', 'DDDb': 'head-20081002', 'OptionFiles': 'Gauss-2008.py;$APPCONFIGROOT/options/Gauss/RICHmirrorMisalignments.py;$DECFILESROOT/options/10000000.opts', 'CondDb': 'head-20081002'}}, 'GroupDescription': 'MC08-v1', 'SimulationConditions': {'BeamEnergy': '7 TeV', 'Generator': 'Pythia', 'Luminosity': 'Fixed 2 10**32', 'MagneticField': '-1', 'BeamCond': 'Collisions', 'DetectorCond': 'VeloClosed', 'SimDescription': 'Beam7TeV-VeloClosed-MagDown'}}
Line: 380 to 382
 
(DIRAC3-production)[paterson@lxplus243 ~]$ cat insertBKPass12345.py
# Bookkeeping publishing script created on Sun Apr  5 17:15:17 2009 by
Changed:
<
<
# by $Id: ProductionAPI.txt,v 1.3 2009/04/08 21:35:50 StuartPaterson Exp $
>
>
# by $Id: ProductionAPI.txt,v 1.4 2009/04/09 15:39:08 StuartPaterson Exp $
 from BookkeepingClient import BookkeepingClient bkClient = BookkeepingClient() bkDict = {'Production': 12345, 'Steps': {'Step1': {'ApplicationName': 'DaVinci', 'ApplicationVersion': 'v22r1', 'ExtraPackages': 'AppConfig.v2r2', 'DDDb': 'head-20090112', 'OptionFiles': '$APPCONFIGOPTS/DaVinci/DVMonitorDst.py', 'CondDB': 'head-20090112'}, 'Step0': {'ApplicationName': 'Brunel', 'ApplicationVersion': 'v34r2', 'ExtraPackages': 'AppConfig.v2r2', 'DDDb': 'head-20090112', 'OptionFiles': '$APPCONFIGOPTS/Brunel/FEST-200903.py', 'CondDB': 'head-20090112'}}, 'GroupDescription': 'FEST-Reco-v1', 'DataTakingConditions': 'DataTaking6153'}
Line: 522 to 524
 So in summary local testing of a workflow with default production ID / production job ID is useful for seeing exactly what the workflow does and the sanity of job options etc. etc.
Changed:
<
<

Example 3 - Running a merging production

>
>

Example 3 - Testing a merging production

  In the below example I use a non-standard set of data but the template can always be modified for the real case. Here (because data was available and deletable) I try to merge two FEST reconstruction RDSTs.
Changed:
<
<
The template for merging productions looks like the following:
>
>
The template for merging productions looks like the following (note the absence of the inputProduction parameter, this is in Example 4 but not necessary for local testing):
 
merge = Production()
Line: 596 to 598
  The production input data in the above template will be overwritten automatically for each job after the production is created and the BK query / regular expression is suitably defined and the jobs are submitted.
Added:
>
>

Example 4 - Creating a merging production

This assumes a production already exists in the system, created with the API that will have set the BKProcessingPass information in the workflow body. For the below example we will assume the recent B inclusive production outputs need to be merged (production 4614).

The template for merging productions looks similar to Example 3 but with the inputProduction parameter specified:

merge = Production()
merge.setProdType('Merge')
merge.setWorkflowName('Merge_BInclusive_4614')
merge.setWorkflowDescription('An example of merging two inputs.')
merge.setBKParameters('MC','2008','MC08-v1','DataTaking6153')
merge.setDBTags('head-20081002','head-20081002')
mergeEventType = '10000000'
mergeData=[]
mergeData.append('/lhcb/MC/2008/DIGI/00004614/0000/00004614_00002000_2.digi')
mergeData.append('/lhcb/MC/2008/DIGI/00004614/0000/00004614_00001998_2.digi')
mergeDataType='DIGI'
mergeSE='Tier1-RDST'
mergeProd = 4614
merge.addMergeStep('v26r3',optionsFile='$STDOPTS/PoolCopy.opts',eventType=mergeEventType,inputData=mergeData,inputDataType=mergeDataType,outputSE=mergeSE,inputProduction=mergeProd)
#removeInputData is False by default (all other finalization modules default to True)
merge.addFinalizationStep(removeInputData=True)
merge.setWorkflowLib('v9r11')
merge.setFileMask('digi')
merge.setProdPriority('9')
merge.createWorkflow()

Setting the inputProduction parameter means that before doing anything the API will attempt to add the processing pass of the previous production to the current one (preserving the step numbering of course). If we create the workflow for this template we see:

2009-04-09 15:33:22 UTC mergingExample.py  INFO: Adding processing pass Step1 from production 4614:
2009-04-09 15:33:22 UTC mergingExample.py  INFO: {'ApplicationName': 'Boole', 'ApplicationVersion': 'v16r3', 'ExtraPackages': '', 'DDDb': '@{DDDBTag}', 'OptionFiles': 'Boole-2008.py', 'CondDB': '@{CondDBTag}'}
2009-04-09 15:33:22 UTC mergingExample.py  INFO: Adding processing pass Step0 from production 4614:
2009-04-09 15:33:22 UTC mergingExample.py  INFO: {'ApplicationName': 'Gauss', 'ApplicationVersion': 'v35r1', 'ExtraPackages': 'AppConfig.v1r1', 'DDDb': '@{DDDBTag}', 'OptionFiles': 'Gauss-2008.py;$APPCONFIGROOT/options/Gauss/RICHmirrorMisalignments.py;$DECFILESROOT/options/10000000.opts', 'CondDB': '@{CondDBTag}'}
2009-04-09 15:33:22 UTC mergingExample.py  INFO: Default options for Merging are:
2009-04-09 15:33:22 UTC mergingExample.py  INFO: OutputStream("InputCopyStream").Output = "DATAFILE='PFN:@{outputData}' TYP='POOL_ROOTTREE' OPT='RECREATE'"
2009-04-09 15:33:22 UTC mergingExample.py  INFO: Setting workflow library LFN to LFN:/lhcb/applications/WorkflowLib-wkf-v9r11.tar.gz
Indicating that the Gauss and Boole steps of the production 4614 were added. If we examine the generated code of the above template (by adding the 'print merge.createCode()' API method) this shows e.g.
...
j.BKProcessingPass = {'Step2': {'ApplicationName': 'LHCb', 'ApplicationVersion': 'v26r3', 'ExtraPackages': '', 'DDDb': 'head-20081002', 'OptionFiles': '$STDOPTS/PoolCopy.opts', 'CondDb': 'head-20081002'}, 'Step1': {'ApplicationName': 'Boole', 'ApplicationVersion': 'v16r3', 'ExtraPackages': '', 'DDDb': 'head-20081002', 'OptionFiles': 'Boole-2008.py', 'CondDB': 'head-20081002'}, 'Step0': {'ApplicationName': 'Gauss', 'ApplicationVersion': 'v35r1', 'ExtraPackages': 'AppConfig.v1r1', 'DDDb': 'head-20081002', 'OptionFiles': 'Gauss-2008.py;$APPCONFIGROOT/options/Gauss/RICHmirrorMisalignments.py;$DECFILESROOT/options/10000000.opts', 'CondDB': 'head-20081002'}}  # type=dict in=True out=False BKProcessingPassInfo
...
and you can see the additional LHCb merging step that has been added here. Although in this case the input production 4614 is visible in the BK we are using the workflow to retrieve this information so it doesn't have to be entered. The input production for merging can be published using create(publish=True,bkScript=True) meaning it is invisible until the resulting BK script is executed. The merging production will automatically pick up the processing pass information from this production via the inputProduction parameter. The merging production will then run with the step number corresponding to the last step of the input production + 1.

To run the above template it then becomes the same procedure as Example 1 (but for a production with defined input data).

To create the production and a local BK script using the regexp approach use:

mask = '/lhcb/MC/2008/DIGI/00004614.*\.digi$'
numberOfFiles=2
merge.create(publish=True,fileMask=mask,groupSize=numberOfFiles,bkScript=True)

to directly publish the merging production to the production management system and BK use:

mask = '/lhcb/MC/2008/DIGI/00004614.*\.digi$'
numberOfFiles=2
merge.create(publish=True,fileMask=mask,groupSize=numberOfFiles,bkScript=False)
  That's all folks!
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback