-- OscarEstradaPastor - 2017-04-05

Introduction

This study has been extended in the years 2017-2018. The new Twiki of the analysis is here: ZmumumapsOperationalGuide.

WARNING: This twiki is based on my experience performing the Zmumu recommendations for 20.7.7.9 release. Please, check that the setup and building commands are the adequate for the desired release.

For setting up release 21, please check the following twiki:

• https://twiki.cern.ch/twiki/bin/view/AtlasComputing/InDetAlignTutorialRun2Rel21

The Zmumu studies are based on the scripts found on /InnerDetector/InDetMonitoring/InDetPerformanceMonitoring folder.

1. First thing to do is to check out that package using git. Check alignment git tutorial how to setup git and check out the alignment packages:

   • https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/AlignmentGitAthenaRepository

2. You will need an extra private package so that athena doesn't crash. So do:

• cp /afs/cern.ch/work/s/sthenkel/public/forPF/Reconstruction /path/to/working/directory
• cd cmt
• cmt make

Producing flat N-tuples: panda/athena

At this stage we will have the environment properly set up. We need to make sure that everything is running smoothly. The samples are analyzed using athena. And a job option is needed as input. The job option analyzes each event of the sample calling the different Zmumu selection tools available in the /InnerDetector/InDetMonitoring/InDetPerformanceMonitoring/src. This can be done by using three different resources, namely:

• Local :This will be useful when testing that the job runs fine, before using big resources such as the batch system or the grid.

• Batch system: This will be useful when running multiple jobs, however in some cases you will need to request officially a transfer of the desired sample from the grid to the batch system. You need to have permission to send jobs in the

• Grid resources: This will be the most used resource to produce Zmumu maps once you know that everything works correctly out of the box. A grid user certificate is needed to be installed in your afs system.

JobOption

The job option is a python file located in the share folder, so:

• cd /path/to/working/dir/InnerDetector/InDetMonitoring/InDetPerformanceMonitoring/share
• [oestrada@lxplus053 athena]$ ls InnerDetector /InDetMonitoring/InDetPerformanceMonitoring/share/
  ElectronEoverPTracking.py runJPsiMonitoring.py runZmumuValidation_data_OnGrid.py
  IDPerfMon_jobOptions.py runJPsiMonitoring_data.py runZmumuValidation_in.py
  TRT_Monitoring_RecExCommonAddOn_jobOptions.py runJPsiMonitoring_data_grid.py runzmumu_DYNAMICALIGNMENT.py
  TrackMonitoring.py runJpsimumuValidation.py runzmumu_maps.py
  batch_EoverP.py runKShortValidation.py runzmumu_maps_grid.py
  jobOptions_useGRL.py runKShortValidation_grid.py runzmumu_recommendations.py
  runEoverPValidation.py runMonitoringZmumuValidation.py runzmumu_recommendations_grid.py
  runEoverPValidationBatch.py runMonitoringZmumuValidation_data.py runzmumu_run2paper.py
  runEoverPValidationData.py runZmumuValidation.py runzmumu_runallmonitoring.py
  runIDMuonMonitoring.py runZmumuValidation_data.py xrootd-patch.py
• All the runzmumu_* are joboptions that have been used or can potentially be used for running the Zmumu. Edit for instance, the runzmumu_recommendations.py and set the following options as it is done below:
  • # enable outputs
    zmumuval = True
    globalAlignmentMon = False #no refitting for extracting the recommendations, only bulk as it comes out
    dimuonmon = False
    
    
  • # MC
    MC_bool = False # choose depending on the case. This will set up different geometry tags and conditions database depending on if it is MC. Since we are interested on the bulk as it comes it does not really matter.
    
    #do Trigger
    DoTrigger = False
    
    # where to run
    grid = True # if you want to run on the Grid
    
    # handle input constants
    readPool = False
    inputConstants = "/afs/cern.ch/work/s/sthenkel/public/forPF/ScriptsToRunWeakModeMaps/runOnBatch/AlignmentConstants/Iter0_AlignmentConstants.roo
    t" # not using alignment constants for recommendations. Leave it as it is.
  • In the following option, one basically needs to download a pool file that it is feeded to the grid as input. This way the grid can recognize which kind of derivations it has to analyze (e.g, AOD, ESD, MCP, ...). There are some of them available in my public, but one can always download one file and put it in the joboption if needed or not any more recommended using a particular pool file.
    • if grid == True:
      PoolInput = ["/afs/cern.ch/work/o/oestrada/public/alignment2/Jpsi_20.7.3.8/20.7.3.8/JpsiNtuplesMC/data16_13TeV.00296939.physics_Main.merge
      .DESDM_MCP.f689_m1587/data16_13TeV.00296939.physics_Main.merge.DESDM_MCP.f689_m1587._0004.1"]
      #PoolInput = ["/afs/cern.ch/user/o/oestrada/eos/atlas/user/o/oestrada/mc15_13TeV.424100.Pythia8B_A14_CTEQ6L1_Jpsimu4mu4.merge.AOD.e3735_s27
      26_r7326_r6282/AOD.07103156._000849.pool.root.1"]
      
      
  • # include patch to resolve issue with xrootd in 20.7.7+ releases, Thanks Will diClemente for following up this -- https://its.cern.ch/jira/
    browse/ATEAM-340
    #include("InDetPerformanceMonitoring/xrootd-patch.py") # this is to use it on the batch, because of an error on the 20.7 release.
  • EvtMax = -1 #to run over all the events set it to -1
    SkipEvents = 0
    
    
  • if (False):
    if not MC_bool:
    print " -- salva -- globalflags.ConditionsTag -> manually configured "
    globalflags.ConditionsTag.set_Value_and_Lock("CONDBR2-BLKPA-2016-21") #not really used to analyze the bulk. If needed, set the if condition to True.
    #else:
    # globalflags.ConditionsTag.set_Value_and_Lock("OFLCOND-MC15c-SDR-05")
    else:
    print " -- salva -- globalflags.ConditionsTag -> auto configured "
    
    myDetDescr = "auto-configured"
    if (False):
    myDetDescr = "ATLAS-R2-2015-04-00-00"
    print " -- salva -- globalflags.DetDescrVersion -> manually configured to ", myDetDescr
    globalflags.DetDescrVersion.set_Value_and_Lock(myDetDescr)#ATLAS-R2-2015-04-00-00 ATLAS-R2-2016-00-01-00
    else:
    print " -- salva -- globalflags.DetDescrVersion -> no user setting -> ", myDetDescr
    
    
  • iDPerfMonZmumu = IDPerfMonZmumu (name = 'IDPerfMonZmumu',
    ReFitterTool1 = MuonRefitterTool,
    ReFitterTool2 = MuonRefitterTool2,
    OutputTracksName = "SelectedMuons",
    isMC = MC_bool, # If set to True, the truth parameters will be filled, have a look into FillTruthParameters
    doRefit = False, #True for evaluation of new alignment constants / False for update of weak mode recommendations -- refits tracks according to provided conditions db and/or external root file containing alignment constants
    doIPextrToPV = False, # True for IP resolution studies, extrapolates IP parameters to primary vertex
    UseTrackSelectionTool = False, # If set to True, it will use the specified TrackSelectionTool in the next Line
    TrackSelectionTool = m_TrackSelectorTool_TightPrimary,
    TrackParticleName = 'InnerDetectorTrackParticles', # Currently does not do anything, code fills IndetTrackParticles and CombinedTrackParticles Trees
    defaultTreeFolder = "/ZmumuValidationUserSel/default",#always filled with information retrieved from TrackParticle associated to selected muon. This is the tree to be used in the weak modes (bulk)
    truthTreeFolder = "/ZmumuValidationUserSel/truth", # Only filled if isMC is set to True
    refit1TreeFolder = "/ZmumuValidationUserSel/refit1",# Only filled if doRefit is set to True
    refit2TreeFolder = "/ZmumuValidationUserSel/refit2",# Only filled if doRefit is set to True
    combTreeFolder = "/ZmumuValidationUserSel/comb",#always filled with information retrieved from TrackParticle associated to selected muon
    doIsoSelection = False,
    doIPSelection = True,
    #loose selection to have Z and JPsi in the n tuple. Usually have jobs have been sent with MCP derivations to dispose all kind of muons from all kind of resonances in the same tuple, and then in the validation script one can select which resonance to study. That's why a loose selection is done below (in blue). If one wants to directly dispose only Zmumu, one can perform the selection in red.
    MassWindowLow = 1., #60.,
    MassWindowHigh 120.,#120.,
    PtLeadingMuon = 1., #15.0,
    PtSecondMuon = 1., #15.0,
    OpeningAngle 0.001, #1.00, # in radians. 1 radian ~60 deg
    OutputLevel DEBUG)
    job + iDPerfMonZmumu

Running locally: testing the job option

Running on the grid: setting up the environment

Running on the grid: Setting which data-sets to be used

For checking individual dataset, the AMI webpage is very useful. Keep this in your favourites. You will use it a lot:

https://ami.in2p3.fr/

However, you will need a whole list of datasets that has been contrasted by ATLAS (GoodRunList), which can be found here:

http://atlasdqm.web.cern.ch/atlasdqm/grlgen/All_Good/

To download the appropiate GRL go to the following directory:

• cd /path/to/workingdir/InnerDetector/InDetMonitoring/InDetPerformanceMonitoring/scripts/runMaps/getDataToScratch

• Edit AlignmentGetRunsFromGRL.py. You will find the following options:

#select GRL
GRL = "data16_13TeV.periodAllYear_DetStatus-v82-pro20-13_DQDefects-00-02-04_PHYS_StandardGRL_All_Good_25ns.xml"
dirGRL = "/afs/cern.ch/user/a/atlasdqm/grlgen/All_Good/"
datatype = "data16_13TeV"
stream = "physics_Main" #for Zmumu use physics_Main
derivation = "DAOD_ZMUMU" #DESDM_MCP, DAOD_MCP have also been used in the past, choose the appropiate
state = "merge" #merge is the usual state
#tag the output
tag = str(GRL) #"data16_13TeV.periodAllYear_DetStatus-v82-pro20-12_DQDefects-00-02-04_PHYS_StandardGRL_All_Good_25ns.xml"

• python AlignmentGetRunsFromGRL.py

This will create a txt file containing all the runs in the GRL for a particular derivation. This will be the input for pathena_sub.py

Running on the grid: sending the jobs

Now, we have the necessary input, for example a name.txt, for using the script to send the job to the grid. Copy it to your running grid directory:

• cd /path/to/workingdir/run_grid

• cp /path/to/workingdir/InnerDetector/InDetMonitoring/InDetPerformanceMonitoring/scripts/runMaps/getDataToScratch/name.txt .

• Edit pathena_sub.py . That is the standard script that has been used in the past, you might need to change several options for release 21.

• UserName = 'yourusername'
  Version = 'maps2016-A-I-DESDMt' #choose whatever text is useful to recognize what you are sending
  nJobs = 200000 #when sending the jobs, either you set the three options below or none of them
  FilesPerJob = 5
  Nfiles=90
  official = True
  UseConstants = False
  officialGroup = "perf-idtracking"
  officialTag= "--official --voms=atlas:/atlas/"+officialGroup+"/Role=production"
  excludeSites = "" #useful if you know that a particular site is failing
  #athenaVersion = '19.3.0.1'
  
  #athenaVersion = 'AtlasProduction,20.7.7.9'
  athenaVersion = 'AtlasProduction,21.0.8' #choose your athenaversion
  
  GBperJob = 'MAX' #set it as it is here
  JOs = os.path.expandvars('$TestArea/InnerDetector/InDetMonitoring/InDetPerformanceMonitoring/share/runzmumu_recommendations.py') #path to JO

In the end you have to use the following option inside the script:

 if UseConstants == False and official == False:       
os.system( 'pathena %s --long --inDS=%s --outDS=%s --nJobs=%d --nFilesPerJob=%d --nGBPerJob=%s --athenaTag=%s  --useShortLivedReplicas --excludedSite=%s' % ( JOs, DataSet, OutputDataSet, nJobs, FilesPerJob, GBperJob, athenaVersion, excludeSites) )

In my case, I ended up modifying a bit the command to avoid a grid error (--forceStaged option), which you can find below:

if UseConstants == False and official == False:
        command = 'pathena %s --long --inDS=%s --outDS=%s --nGBPerJob=%s --athenaTag=%s --cmtConfig=%s --useShortLivedReplicas --forceStaged' % ( JOs, [[DataSet][DataSet]], [[OutputDataSet][OutputDataSet]], nJobs,  athenaVersion,cmtConfig )
        print command
        os.system(command)

You can find all my modified scripts in my public, in case it is needed and you need to copy some stuff:

/afs/cern.ch/work/o/oestrada/public/alignment2/Zmaps_20.7.7.9/InnerDetector/InDetMonitoring/InDetPerformanceMonitoring/

Running on the grid: Downloading and merging the flat n-tuples

You can check how the job is going on your big panda user page, e.g mine is here:

http://bigpanda.cern.ch/user/Oscar.Estrada/?display_limit=100

Or by using pbook application (check the panda link of before).

Once the jobs are finished you need to download them using rucio. I also attach a useful twiki page:

http://rucio.cern.ch/client_howto.html

Basically you are going to need to use one command, i . e:

• rucio download nameofthedatasettodownload

Actually, the good thing is that these rucio commands are automatically put into a sh file for all the runs when executing pathena_sub.py, and you only need to make execute it. For example, for some tests I did, it created:

• #!/bin/bash 
  rucio download user.oestrada.maps2016-A-I-DESDMt..00308084.merge.DESDM_ZMUMU_ZmumuValidationUserSel/
  rucio download user.oestrada.maps2016-A-I-DESDMt..00304337.merge.DESDM_ZMUMU_ZmumuValidationUserSel/

• And instead typing for all the runs manually "rucio download something" , one can simply do:

source nameofthegeneratedscript.sh

Ok, so now I know how to download it, but where...?

This is a good question. You don't want to fill your work area with bunches of huge root files. The best practice should be to download it into your /eos/user/ directory. If you don't have one I strongly recommend you to ask for one to ATLAS. Otherwise, you can always download it into the work, but be sure not send all those files when sending the jobs to the grid, it will fail if the files are too big. (That's why I recommend to create your sending jobs directory in a folder parallel to the InnerDetector. )

And what do I do with those bunches of root files? Merge it!

For doing this, I used basically two private scripts that you can copy from my public. Feel free to improve them or use different. In the end, we want to merge them with the hadd command. But the problem is that hadd works for 500 root files maximum, if there are more it crashes. So we need to divide the input files in groups of 500 root files, merge them by chunks, merge the chunks in one final root file and delete the chunks files. This is made by the hadder.py script (provided by Steffen in the past).

First, you need to copy them from my public:

• cp /afs/cern.ch/user/o/oestrada/public/txtmaker_forbatch.py
• cp /afs/cern.ch/user/o/oestrada/public/hadder.py

The first script ( i.e txtmaker_forbatch.py) puts in a txt file the path to every root file. Just put the input directory that you want to. For example, I had organized all the pred0fix runs in this folder:

inputDir = /afs/cern.ch/work/o/oestrada/public/alignment2/JpsiNtuples/data16_recommendations/pred0fix
outputDir = "."

inputList = "Name" # this is the name you want to give to txt

Now that you have a txt with the root files you want to hadd, one must use the hadder.py script, as follows:

• python hadder.py Name.txt MergedRootFilesName

This will create a root file, whose name is MergedRootFilesName.root. And now, you are in the stage to analyze the events.

Analyzing the merged flat n-tuple and producing the root files: ZmumuValidationExample

You are going basically to use three scripts: one which analyzes the n-tuple (ZmumuValidationExample.cxx), one which runs the code (run_ZmumuValidationExample.C) and another to set the inputFiles (userFiles.C).

1. First step is to introduce the path to the MergedRootFile in the userFiles.C. Notice that the recommendation files from 2016 are still available, feel free to run it and reproduce the plots.
2. Next, you need to run the code in Zmumu sample mode, and sagitta analysis mode. You can find detailed information in two README's available in the scripts folder. Have a look at README and README2 for running the code.
3. Set the number of iterations, the (eta,phi) bins, etc... in the run_ZmumuValidationExample.C. Usually 20 iterations have run smoothly, for 20 bins in eta and phi.

Select which tree to analyze . Since you want bulk, you need:

 string tree_name = "Default_InDetTrackParticle";

Make sure that you are analyzing enough events by putting a big number in number of events, otherwise you may not analyze the whole sample:

  int nEvents = 1000000000;  
   bool isMC = false; 

And the name of the output root file:

string output_file = "300655_maps.root";

1. Open a root session.
2. root -b
3. .L run_ZmumuValidationExample+
4. run_ZmumuValidationExample(20,0,2) : #20 iterations, in the Z sample mode (0), for the sagitta, d0 and z0 plots (2)

Plotting the sagitta, d0 and z0 distributions

Next, you can plot the 2-D maps with the printCorrectionsMaps_simple.py

Plotting the statistical and systematics for sagitta, d0 and z0

Use the MakeSystd0z0.C

Plotting 1-D mean sagitta, d0 and z0 distributions

Use the ProjectionsFinal.C

Zmumu analysis code review

The main analysis file is ZmumuValidationExample.cxx . Currently the Zmumu name is a bit misleading. Actually it is prepared for analyzing dimuon events (μ+ μ-), or even more generically a set of events with a positive and a negative charged tracks. The input data is read from a Zmumu ntuple as explained above.

The code contains several C++ classes that performs different actions on the data. Each class is defined in a different file. There are several files with C++ codes to be considered:

1. ZmumuValidationExample.cxx. This is the main code.
2. run_ZmumuValidationExample.C. This file is the one the user calls
3. JPsiMassFit.C. This class is prepared for fitting the invariant mass distributions for several particle types: Z, J/ψ and Υ
4. deltaSagittaFitter.C. this class evaluates the δSagitta by comparing the pT spectrum of the μ+ and μ-
5. SagittaDistortionFitter.C. It computes δSagitta with a very similar method to the official one, but it also deals with Z, J/ψ and Υ
6. IPResolStudies.C. This class helps to evaluate the impact parameter (both d0 and z0) of both muons.

The input data (Zmumu ntuples) is defined in the:

1. userFiles.C. As it name suggests this file is very user dependent. In order to provide a list of input ntuples, the user must edit this file and make sure the s_fileNames list includes the proper files to be analyzed.

Configuring the analysis

There are many parameters that can be configured for each analysis, as the input data (done via userFiles.C), the particle and event selection cuts, the kind of analysis to be performed (sagitta distortion and/or radial expansion), the binning in (η, φ), the use of bias maps, etc. All this is done inside run_ZmumuValidationExample.C. Below one may find how to set some of the analysis parameters:

1. myZmumuValidationExample.SetEtaBins(15) → 15 bins are used in the η histograms and in the (η, φ) maps.
2. myZmumuValidationExample.SetPhiBins(20) → 20 bins are used in the φ histograms and in the (η, φ) maps
3. myZmumuValidationExample.SetPtCutLead (30.) → Leading muon pT cut bigger than 30 GeV.
4. myZmumuValidationExample.SetPtCut2nd (6.5) → pT cut bigger than 6.5 GeV of the second muon.