MonteCarlo procedures

Here you find some guidelines and real use cases to use the scripts in the svn.

Getting started

To get the WmAgentScripts scripts, run the following:

$ svn co svn+ssh://svn.cern.ch/reps/WmAgentScripts

Getting the status of the requests

reqinfo.py

Assignment

Regular assignment for a new FSIM request

First of all, let's check the lumis/job, which we want to be set to 20 (the number of jobs will then change also from 104021 to 5201):

[vocms23] /afs/cern.ch/user/s/spinoso > python WmAgentScripts/mc/reqinfo.py -w spinoso_SUS-Summer12_FS52-00002_R1816_B12_FSIM_01_LHE_120921_130337_5581 -g
spinoso_SUS-Summer12_FS52-00002_R1816_B12_FSIM_01_LHE_120921_130337_5581 (MonteCarloFromGEN,assignment-approved at ?)
 Priority: 99000 Team:  Timeev: 10 Jobs: 104021 Sizeev: 500 Hours/job: 0
 ReqEvents: 31200000 ExpectedEvts/job: 0.0 FilterEff: 1.0 CMSSW_5_2_6 Lumis/Job: 1
 PrimaryDataset: SMS-T5gg_Mgluino-400to2000_mNLSP-100to2000_8TeV-Pythia6Z GlobalTag: START52_V9 CPUHours: 86666
 InputDataset: {'status': 'VALID', 'lumicount': 104021, 'name': '/SMS-T5gg_Mgluino-400to2000_mNLSP-100to2000_8TeV-Pythia6Z/Summer12-START52_V9_FSIM-v1/GEN', 'events': 31200000}

[vocms23] /afs/cern.ch/user/s/spinoso > python WmAgentScripts/changeSplittingWorkflow.py spinoso_SUS-Summer12_FS52-00002_R1816_B12_FSIM_01_LHE_120921_130337_5581 20
200 OK
Successfully updated splitting parameters for /spinoso_SUS-Summer12_FS52-00002_R1816_B12_FSIM_01_LHE_120921_130337_5581/MonteCarloFromGEN  <a href="details/spinoso_SUS-Summer12_FS52-00002_R1816_B12_FSIM_01_LHE_120921_130337_5581">Details</A> <a href=".">Browse</A><BR>

[vocms23] /afs/cern.ch/user/s/spinoso > python WmAgentScripts/mc/reqinfo.py -w spinoso_SUS-Summer12_FS52-00002_R1816_B12_FSIM_01_LHE_120921_130337_5581 -g

spinoso_SUS-Summer12_FS52-00002_R1816_B12_FSIM_01_LHE_120921_130337_5581 (MonteCarloFromGEN,assignment-approved at ?)
 Priority: 99000 Team:  Timeev: 10 Jobs: 5201 Sizeev: 500 Hours/job: 16
 ReqEvents: 31200000 ExpectedEvts/job: 0.0 FilterEff: 1.0 CMSSW_5_2_6 Lumis/Job: 20
 PrimaryDataset: SMS-T5gg_Mgluino-400to2000_mNLSP-100to2000_8TeV-Pythia6Z GlobalTag: START52_V9 CPUHours: 86666
 InputDataset: {'status': 'VALID', 'lumicount': 104021, 'name': '/SMS-T5gg_Mgluino-400to2000_mNLSP-100to2000_8TeV-Pythia6Z/Summer12-START52_V9_FSIM-v1/GEN', 'events': 31200000}

The global tag above is START52_V9, so the processing version automatically generated by the script would be START52_V9-v1, which is not what we want for FSIM:

[vocms23] /afs/cern.ch/user/s/spinoso > python WmAgentScripts/mc/mcassign.py -w spinoso_SUS-Summer12_FS52-00002_R1816_B12_FSIM_01_LHE_120921_130337_5581 -z FNAL

Matching requests:

REQUEST TEAM PRIORITY ACQERA PROCVS ZONE
spinoso_SUS-Summer12_FS52-00002_R1816_B12_FSIM_01_LHE_120921_130337_5581 production 99000 Summer12 START52_V9-v1 FNAL

So the option -p START52_V9_FSIM-v1 must be passed explicitely:

[vocms23] /afs/cern.ch/user/s/spinoso > python WmAgentScripts/mc/mcassign.py -w spinoso_SUS-Summer12_FS52-00002_R1816_B12_FSIM_01_LHE_120921_130337_5581 -z FNAL --small -p START52_V9_FSIM-v1

Matching requests:

REQUEST TEAM PRIORITY ACQERA PROCVS ZONE
spinoso_SUS-Summer12_FS52-00002_R1816_B12_FSIM_01_LHE_120921_130337_5581 production 99000 Summer12 START52_V9_FSIM-v1 FNAL

In this last command, the script checks the DBS for the dataset name /SMS-T5gg_Mgluino-400to2000_mNLSP-100to2000_8TeV-Pythia6Z/Summer12-START52_V9_FSIM-v1/AODSIM and not /SMS-T5gg_Mgluino-400to2000_mNLSP-100to2000_8TeV-Pythia6Z/Summer12-START52_V9-v1/AODSIM, so now we can safely proceed with assignment. Besides note that the correct datatier (AODSIM) is applied by the script.

Balancing assignment among several WMA instances

We have a bug list of new requests, even having different acquisition era, and we want to assign them to RAL. But we also want to distribute them to the different WMA instances. We can put all those requests in a single text file (ral.txt), and use the -t option to specify the teams, like this:

[vocms23] /afs/cern.ch/user/s/spinoso > python WmAgentScripts/mc/mcassign-devel.py -l ral.txt -z RAL -t production,production,integration

Matching requests:

REQUEST TEAM PRIORITY ACQERA PROCVS ZONE
spinoso_TOP-Summer12-00115_R1773_B165_01_LHE_120911_120116_6894 production 95000 Summer12 START52_V9-v1 RAL
spinoso_HIG-Summer11-01182_R1778_B116_01_LHE_120911_115119_8730 production 95000 Summer11 START311_V2-v1 RAL
spinoso_HIG-Summer11-01183_R1778_B116_02_LHE_120911_115122_5334 integration 95000 Summer11 START311_V2-v1 RAL
spinoso_HIG-Summer11-01184_R1778_B116_03_LHE_120911_115124_7331 production 95000 Summer11 START311_V2-v1 RAL
spinoso_HIG-Summer11-01185_R1778_B116_04_LHE_120911_115127_1841 production 95000 Summer11 START311_V2-v1 RAL
spinoso_HIG-Summer11-01186_R1778_B116_05_LHE_120911_115130_3802 integration 95000 Summer11 START311_V2-v1 RAL
spinoso_HIG-Summer11-01187_R1778_B116_06_LHE_120911_115132_7501 production 95000 Summer11 START311_V2-v1 RAL

Note that production is repeated: this results in having 66% of the requests assigned to production (bigger WMA), and 33% to integration (smaller WMA).

Using the --assign option will make the assignment.

Assigning special requests

As asked (see https://hypernews.cern.ch/HyperNews/CMS/get/dataopsrequests/1802.html):

_I suggest to use v2 to avoid confusion. EWK-Summer12_GS_EMShowerImproved-00001 No special name EWK-Summer12_GS_EMShowerImproved-00002 Use "ExtendedX0Max" special name_

we want to modify the processing version with a special string for the second workflow, and use v2 for both of them.

First of all, let's use -g to check priority, hours and other parameters useful in assignment; here, most important, the global tag (START53_V7D):

[vocms23] /afs/cern.ch/user/s/spinoso > cat ezio.txt 
etorassa_EWK-Summer12_GS_EMShowerImproved-00001_batch1_v1__120912_180447_6327
etorassa_EWK-Summer12_GS_EMShowerImproved-00002_batch1_v1__120912_180452_8982
[vocms23] /afs/cern.ch/user/s/spinoso > python WmAgentScripts/mc/reqinfo.py -l ezio.txt -g

etorassa_EWK-Summer12_GS_EMShowerImproved-00001_batch1_v1__120912_180447_6327 (MonteCarloFromGEN,assignment-approved at ?)
 Priority: 101000 Team:  Timeev: 70 Jobs: 177500 Sizeev: 700 Hours/job: 7
 ReqEvents: 10074900 ExpectedEvts/job: 0.0 FilterEff: 0.1419 CMSSW_5_3_3_patch1 Lumis/Job: 1
 PrimaryDataset: DYJetsToLL_M-50_TuneZ2Star_8TeV-madgraph-tarball GlobalTag: START53_V7D CPUHours: 195900
 InputDataset: {'status': 'VALID', 'lumicount': 177500, 'name': '/DYJetsToLL_M-50_TuneZ2Star_8TeV-madgraph-tarball/Summer12-START50_V13-v1/GEN', 'events': 71000000}

etorassa_EWK-Summer12_GS_EMShowerImproved-00002_batch1_v1__120912_180452_8982 (MonteCarloFromGEN,assignment-approved at ?)
 Priority: 101000 Team:  Timeev: 70 Jobs: 177500 Sizeev: 700 Hours/job: 7
 ReqEvents: 10074900 ExpectedEvts/job: 0.0 FilterEff: 0.1419 CMSSW_5_3_3_patch1 Lumis/Job: 1
 PrimaryDataset: DYJetsToLL_M-50_TuneZ2Star_8TeV-madgraph-tarball GlobalTag: START53_V7D CPUHours: 195900
 InputDataset: {'status': 'VALID', 'lumicount': 177500, 'name': '/DYJetsToLL_M-50_TuneZ2Star_8TeV-madgraph-tarball/Summer12-START50_V13-v1/GEN', 'events': 71000000}

If needed, we can use WF Team scripts to modify lumis/job, priority…

Then we run reqinfo.py without options, to check other aspects, like the output dataset tier (GEN-SIM) or events expected:

[vocms23] /afs/cern.ch/user/s/spinoso > python WmAgentScripts/mc/reqinfo.py -l ezio.txt 

etorassa_EWK-Summer12_GS_EMShowerImproved-00001_batch1_v1__120912_180447_6327 (EWK-Summer12_GS_EMShowerImproved-00001,MonteCarloFromGEN,assignment-approved at ?)
 Priority: 101000 Team:  Jobs: Q:0 C:0 P:0 R:0 S:0 F:0 T:0
 /DYJetsToLL_M-50_TuneZ2Star_8TeV-madgraph-tarball/None-v0/GEN-SIM 0 (reached 0%, expect 10074900, status '')

etorassa_EWK-Summer12_GS_EMShowerImproved-00002_batch1_v1__120912_180452_8982 (EWK-Summer12_GS_EMShowerImproved-00002,MonteCarloFromGEN,assignment-approved at ?)
 Priority: 101000 Team:  Jobs: Q:0 C:0 P:0 R:0 S:0 F:0 T:0
 /DYJetsToLL_M-50_TuneZ2Star_8TeV-madgraph-tarball/None-v0/GEN-SIM 0 (reached 0%, expect 10074900, status '')

Now we have everything we need. The team will be mc. We will assign the first one, and modify the processing version for the second one. Here is the matching:

[vocms23] /afs/cern.ch/user/s/spinoso > cat ezio.txt 
etorassa_EWK-Summer12_GS_EMShowerImproved-00001_batch1_v1__120912_180447_6327
etorassa_EWK-Summer12_GS_EMShowerImproved-00002_batch1_v1__120912_180452_8982

[vocms23] /afs/cern.ch/user/s/spinoso > python WmAgentScripts/mc/mcassign-devel.py -t mc -w etorassa_EWK-Summer12_GS_EMShowerImproved-00001_batch1_v1__120912_180447_6327 -z FNAL -v v2

Matching requests:

REQUEST TEAM PRIORITY ACQERA PROCVS ZONE
etorassa_EWK-Summer12_GS_EMShowerImproved-00001_batch1_v1__120912_180447_6327 mc 101000 Summer12 START53_V7D-v2 FNAL

[vocms23] /afs/cern.ch/user/s/spinoso > python WmAgentScripts/mc/mcassign-devel.py -t mc -w etorassa_EWK-Summer12_GS_EMShowerImproved-00002_batch1_v1__120912_180452_8982 -z FNAL -p START53_V7D_ExtendedX0Max-v2

Matching requests:

REQUEST TEAM PRIORITY ACQERA PROCVS ZONE
etorassa_EWK-Summer12_GS_EMShowerImproved-00002_batch1_v1__120912_180452_8982 mc 101000 Summer12 START53_V7D_ExtendedX0Max-v2 FNAL

Using --assign will make the assignment.

Resubmission of a special request (FSIM resubmission)

This example is particularly useful to understand how to customize the processing version.

The request has already submitted in the past, so we expect to find an old request having the same PREPID in the ReqMgr. That would also suggest the presence of a previous version of the output dataset in DBS, so the version (v1, v2...) should be carefully chosen.

Moreover, FSIM requests want special strings in the processing version.

So, first of all, let's start investigation:

[vocms23] /afs/cern.ch/user/s/spinoso > python WmAgentScripts/mc/mcassign-devel.py -w spinoso_SUS-Summer12_FS52-00007_R1783_B10_01_LHE_120911_112125_1891

Matching requests:

REQUEST TEAM PRIORITY ACQERA PROCVS ZONE
spinoso_SUS-Summer12_FS52-00007_R1783_B10_01_LHE_120911_112125_1891 dataops 99000 Summer12 START52_V9-v1 FNAL

TESTED:   spinoso_SUS-Summer12_FS52-00007_R1783_B10_01_LHE_120911_112125_1891 dataops Summer12 START52_V9-v1 FNAL

The following requests have been TESTED:

spinoso_SUS-Summer12_FS52-00007_R1783_B10_01_LHE_120911_112125_1891

The output says that another request has FNAL as custodialT1. In fact, just for confirmation:

spinoso_SUS-Summer12_FS52-00007_R1599_B3_FSIM_04_LHE_120801_184526_9859 (SUS-Summer12_FS52-00007,MonteCarloFromGEN,announced at FNAL)
 Priority: 99000 Team: dataops Jobs: Q:0 C:0 P:0 R:0 S:0 F:0 T:0
 /SMS-T2bb_Msbottom-225to1200_mLSP-0to1175_8TeV-Pythia6Z/Summer12-START52_V9_FSIM-v1/AODSIM 7999432 (reached 99%, expect 8000000, status 'VALID')
  subscribed to T1_US_FNAL_MSS (Move,100%)
  request approved: https://cmsweb.cern.ch/phedex/prod/Request::View?request=386077

spinoso_SUS-Summer12_FS52-00007_R1783_B10_01_LHE_120911_112125_1891 (SUS-Summer12_FS52-00007,MonteCarloFromGEN,assignment-approved at ?)
 Priority: 99000 Team:  Jobs: Q:0 C:0 P:0 R:0 S:0 F:0 T:0
 /SMS-T2bb_Msbottom-225to1200_mLSP-0to1175_8TeV-Pythia6Z/None-v0/AODSIM 0 (reached 0%, expect 11400000, status '')

After verifying savannah for tape families, we could assume FNAL as custodial T1.

The command also suggests dataops as team to be used (as priority is less than 100k), we will change it to production as that team is not available at the moment.

Finally, it suggests also START52_V9-v1 as processing version: per default, it is <globaltag>-<vX>. But FSIM use special strings, so we will use the option -p <processing version>, copying/pasting/modifying the suggested value START52_V9-v1:

[vocms23] /afs/cern.ch/user/s/spinoso > python WmAgentScripts/mc/mcassign-devel.py -t production -w spinoso_SUS-Summer12_FS52-00007_R1783_B10_01_LHE_120911_112125_1891 -t production -p START52_V9_FSIM-v1

Matching requests:

REQUEST TEAM PRIORITY ACQERA PROCVS ZONE
Processing version already in use for spinoso_SUS-Summer12_FS52-00007_R1783_B10_01_LHE_120911_112125_1891 : /SMS-T2bb_Msbottom-225to1200_mLSP-0to1175_8TeV-Pythia6Z/Summer12-START52_V9_FSIM-v1/AODSIM

So the script complains about the presence of the resulting dataset name in DBS, we should move to v2:

[vocms23] /afs/cern.ch/user/s/spinoso > python WmAgentScripts/mc/mcassign-devel.py -t production -w spinoso_SUS-Summer12_FS52-00007_R1783_B10_01_LHE_120911_112125_1891 -p START52_V9_FSIM-v2

Matching requests:

REQUEST TEAM PRIORITY ACQERA PROCVS ZONE
spinoso_SUS-Summer12_FS52-00007_R1783_B10_01_LHE_120911_112125_1891 production 99000 Summer12 START52_V9_FSIM-v2 FNAL

TESTED:   spinoso_SUS-Summer12_FS52-00007_R1783_B10_01_LHE_120911_112125_1891 dataops Summer12 START52_V9_FSIM-v2 FNAL

The following requests have been TESTED:

spinoso_SUS-Summer12_FS52-00007_R1783_B10_01_LHE_120911_112125_1891

This matching is fine. Now we can add --assign to the same command line and assign the workflow.

LowPU2010 assignment

In this case, the acquisition era cannot be derived from the request name (Summer11/Summer12 is not there), so the script will complain; it's enough then to set the acquisition era using -a:

[vocms23] /afs/cern.ch/user/s/spinoso > cat fnal-lowpu2010.txt 
casarsa_QCD-LowPU2010-00045_4_v1__120918_182530_348
casarsa_QCD-LowPU2010-00046_4_v1__120918_182538_119
casarsa_QCD-LowPU2010-00047_4_v1__120918_182544_9193
casarsa_QCD-LowPU2010-00048_4_v1__120918_182549_6537
[vocms23] /afs/cern.ch/user/s/spinoso > python WmAgentScripts/mc/mcassign.py -l fnal-lowpu2010.txt -t mc -z FNAL -v v1 

Matching requests:

REQUEST TEAM PRIORITY ACQERA PROCVS ZONE
   !!! WARNING !!! Cannot guess acquisition era, using Summer12
casarsa_QCD-LowPU2010-00045_4_v1__120918_182530_348 mc 93000 Summer12 START42_V17C-v1 FNAL
   !!! WARNING !!! Cannot guess acquisition era, using Summer12
casarsa_QCD-LowPU2010-00046_4_v1__120918_182538_119 mc 93000 Summer12 START42_V17C-v1 FNAL
   !!! WARNING !!! Cannot guess acquisition era, using Summer12
casarsa_QCD-LowPU2010-00047_4_v1__120918_182544_9193 mc 93000 Summer12 START42_V17C-v1 FNAL
   !!! WARNING !!! Cannot guess acquisition era, using Summer12
casarsa_QCD-LowPU2010-00048_4_v1__120918_182549_6537 mc 93000 Summer12 START42_V17C-v1 FNAL


[vocms23] /afs/cern.ch/user/s/spinoso > python WmAgentScripts/mc/mcassign.py -l fnal-lowpu2010.txt -t mc -z FNAL -a Summer12

Matching requests:

REQUEST TEAM PRIORITY ACQERA PROCVS ZONE
casarsa_QCD-LowPU2010-00045_4_v1__120918_182530_348 mc 93000 Summer12 START42_V17C-v1 FNAL
casarsa_QCD-LowPU2010-00046_4_v1__120918_182538_119 mc 93000 Summer12 START42_V17C-v1 FNAL
casarsa_QCD-LowPU2010-00047_4_v1__120918_182544_9193 mc 93000 Summer12 START42_V17C-v1 FNAL
casarsa_QCD-LowPU2010-00048_4_v1__120918_182549_6537 mc 93000 Summer12 START42_V17C-v1 FNAL

Now we can proceed with setting the ProcessingVersion: in order to have "LowPU2010-START42_V17C-v1"; then we rerun the script, so that it will test the assignment:


[vocms23] /afs/cern.ch/user/s/spinoso > python WmAgentScripts/mc/mcassign.py -l fnal-lowpu2010.txt -t mc -z FNAL -a Summer12 -p LowPU2010-START42_V17C-v1

Matching requests:

REQUEST TEAM PRIORITY ACQERA PROCVS ZONE
casarsa_QCD-LowPU2010-00045_4_v1__120918_182530_348 mc 93000 Summer12 LowPU2010-START42_V17C-v1 FNAL
casarsa_QCD-LowPU2010-00046_4_v1__120918_182538_119 mc 93000 Summer12 LowPU2010-START42_V17C-v1 FNAL
casarsa_QCD-LowPU2010-00047_4_v1__120918_182544_9193 mc 93000 Summer12 LowPU2010-START42_V17C-v1 FNAL
casarsa_QCD-LowPU2010-00048_4_v1__120918_182549_6537 mc 93000 Summer12 LowPU2010-START42_V17C-v1 FNAL

Assigning ACDC

Sometimes the WF team could ask for help in assigning ACDC which cannot be assigned using the web interface. A good example is assigning ACDC to T2_EE_Estonia, which is not listed in the assignment page. In this case, only CLI assignment would work.

[vocms23] /afs/cern.ch/user/s/spinoso > cat edgar.txt 
efajardo_ACDC_merge_EXO-Summer12-01933_R1805_B169_14_LHE__120926_142929_3391
efajardo_ACDC_merge_EXO-Summer12-01930_R1805_B169_11_LHE__120926_142322_2581
efajardo_ACDC_merge_EXO-Summer12-01915_R1805_B169_02_LHE__120926_141543_5441
jen_a_spinoso_EXO-Summer12-01927_R1805_B169_09_LHE_120914_191050_1327_120928_204342_7613
[vocms23] /afs/cern.ch/user/s/spinoso > python WmAgentScripts/mc/mcassign.py -l edgar.txt -z T2_EE_Estonia -t mc -a Summer12 

Matching requests:

REQUEST TEAM PRIORITY ACQERA PROCVS ZONE
efajardo_ACDC_merge_EXO-Summer12-01933_R1805_B169_14_LHE__120926_142929_3391: not a MonteCarlo/MonteCarloFromGEN request!
efajardo_ACDC_merge_EXO-Summer12-01933_R1805_B169_14_LHE__120926_142929_3391 mc 1000002 Summer12 START52_V9-v1 T2_EE_Estonia
efajardo_ACDC_merge_EXO-Summer12-01930_R1805_B169_11_LHE__120926_142322_2581: not a MonteCarlo/MonteCarloFromGEN request!
efajardo_ACDC_merge_EXO-Summer12-01930_R1805_B169_11_LHE__120926_142322_2581 mc 1000002 Summer12 START52_V9-v1 T2_EE_Estonia
efajardo_ACDC_merge_EXO-Summer12-01915_R1805_B169_02_LHE__120926_141543_5441: not a MonteCarlo/MonteCarloFromGEN request!
efajardo_ACDC_merge_EXO-Summer12-01915_R1805_B169_02_LHE__120926_141543_5441 mc 1000002 Summer12 START52_V9-v1 T2_EE_Estonia
jen_a_spinoso_EXO-Summer12-01927_R1805_B169_09_LHE_120914_191050_1327_120928_204342_7613: not a MonteCarlo/MonteCarloFromGEN request!
jen_a_spinoso_EXO-Summer12-01927_R1805_B169_09_LHE_120914_191050_1327_120928_204342_7613 mc 1000000 Summer12 START52_V9-v1 T2_EE_Estonia

Please note that:

  • a warning is printed by the script, saying that the request is not a regular MC (in fact it's "Resubmission")
  • the acquisition era is not guessed correctly, that's why it must be always provided
  • also the team must be provided (see -t mc), as the ACDC should be run on the same WMA as the original one

-- VincenzoSpinoso - 15-Jun-2012

Edit | Attach | Watch | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r9 - 2012-10-01 - VincenzoSpinoso
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback