JoshMcFayden < Sandbox

<!-- This is the default ATLAS template. 
Please modify it in the sections indicated to create your topic! In particular, notice that at the bottom there are some sections that must be filled for publicly accessible pages.
If you have any comments/complaints about this template, then please email : Patrick Jussel (patrick dot jussel at cern dot ch) and/or Maria Smizanska (maria dot smizanska at cern dot ch)--> 

<!-- By default the title is the WikiWord used to create this topic !-->
<!-- if you want to modify it to something more meaningful, just replace %TOPIC% below with i.e "My Topic"!-->
<!---------------------------------------------------------  snip snip ----------------------------------------------------------------->

---+!!<nop>Josh !McFayden
%TOC%     <!--optional-->
%STARTINCLUDE%


<!--***********************************************************-->
<!--Do NOT remove the remaining lines, but add requested info as appropriate-->
<!--***********************************************************-->

---+ Setting up release from MR in r21
<verbatim>
mkdir testdir
cd testdir
mkdir source build run
cd build
setupATLAS
acmSetup AthGeneration,21.6,latest
acm sparse_clone_project athena zmarshal/athena.git 21.6_MGC_Update
acm add_pkg athena/Generators/MadGraphControl
acm compile
</verbatim>

---+ To-do table test
ExtensionsTestPage


------
---+*MC production notes*
-------
---++*Getting started with evgen*

See this !TWiki page for a very general overview of running generators in ATLAS:
[[https://twiki.cern.ch/twiki/bin/view/AtlasProtected/RunningGenerators][https://twiki.cern.ch/twiki/bin/view/AtlasProtected/RunningGenerators]]
And this one which is a bit more specific to MC15:
[[https://twiki.cern.ch/twiki/bin/view/AtlasProtected/AtlasMcProductionMC15][https://twiki.cern.ch/twiki/bin/view/AtlasProtected/AtlasMcProductionMC15]]

The basic idea is that you have some python jobOptions that load and run your MC generator of choice. You run this in Athena by doing
<verbatim>
setupATLAS
asetup <some release>,here

Generate_tf.py <arguments>
</verbatim>

Where the generate transform (Generate_tf.py) is essentially a bit of python that creates a simplified version of a more complicated Athena command/jobOptions that has nice command line arguments. <some release> should be whatever the most recent release that MC production is using, but this information can be hard to find and for now I would suggest you use 19.2.3.8.


---++ MadGraphControl
Then the main thing you will use to generate events is !MadGraphControl. This is a package for running !MadGraph5_aMC@NLO in Athena via some python steering. You can take a look at this !TWiki page to get an idea of how this works and some simple examples:
[[https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/MadGraph5aMCatNLOForAtlas][https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/MadGraph5aMCatNLOForAtlas]]


---++ Job submission script
Script is [[%ATTACHURL%/sendJobs.sh.text][sendJobs.sh]] (please remove ".text" suffix).

Usage is essentially:
<verbatim>
./sendJobs.sh <generation mode [-f:on-the-fly,-g: gridpack mode]> <run mode [-b:build tarball,-t:run from tarball> -i <DSID> -c <CoM energy> -a <athena release> -j <jobOptions> -n <number of jobs> [-p <gripack location> -s <site(s)>]
</verbatim>


   * A couple of examples:
      * On-the-fly:
<verbatim>
./sendJobs.sh -f -b -o -i "999999" -c 7000 -a 19.2.3.8,here -j MC15.999999.MadGraphPythia8EvtGen_A14NNPDF23LO_Zee_Incl.py -n "200"
./sendJobs.sh -f -t -o -i "999999" -c 7000 -a 19.2.3.8,here -j MC15.999999.MadGraphPythia8EvtGen_A14NNPDF23LO_Zee_Incl.py -n "200"
</verbatim>
(the first line builds the panda tarball, the second submits the jobs - this is useful if you want to send e.g. several DSIDs because you don't have to remake the tarball each time)

      * From gridpack:
<verbatim>
./sendJobs.sh -g -b -o -i "999999" -c 7000 -a 19.2.3.8,here -j MC15.999999.MadGraphPythia8EvtGen_A14NNPDF23LO_Zee_Incl.py -n "200" -p <gridpack location>
./sendJobs.sh -g -t -o -i "999999" -c 7000 -a 19.2.3.8,here -j MC15.999999.MadGraphPythia8EvtGen_A14NNPDF23LO_Zee_Incl.py -n "200" -p <gridpack location> -s ANALY_CERN_SLC6,ANALY_BNL_LONG
</verbatim>
(only send jobs to CERN and BNL)
I suggest you start first with running some of the examples on the MG5_aMC@NLO TWiki page above first as that's a bit simpler.


%BR%
%BR%
 %BR%


------
---+*MC production coordinator notes*
-------

---++*Meetings*
   * Physics Coord (Monday 15:00)
   * ADC Weekly (Tuesday 14:45)
   * Simulation Steering (Tuesday 16:00)
   * MC Production (Wednesday 10:00)
   * PMG subgroup+software+production (Monday, 13:00)
   * ProdSys2 ("coffee" meetings) (Tues & Thurs 9:00) [[http://indico.cern.ch/category/5851/][indico]]


---++*Useful webpages*
   * [[https://twiki.cern.ch/twiki/bin/view/AtlasProtected/AtlasProductionGroup][AtlasProductionGroup]]
   * [[https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/AtlasProductionGroupInternal][AtlasProductionGroupInternal]]
   * [[https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/AtlasProductionGroupReporting][For PC meeting summaries]]
   * [[https://twiki.cern.ch/twiki/bin/view/AtlasProtected/ProdSys2RequestCreation][ProdSys2RequestCreation]]
   * [[http://atlas-project-mc-production.web.cern.ch/atlas-project-mc-production/requests][MC production requests]]
   * [[https://its.cern.ch/jira/browse/PRODSYS-252?filter=14632][Things to fix in ProdSys2]]


---++*Misc*
---+++ tarball dir
/afs/cern.ch/atlas/software/kits/EvgenJobOpts/


---+++ Permissions
   * "pts add mcfayden zp:mcgen" should work...
   * "pts membership `whoami`" it will at least tell you what you have permissions for...
   * "fs sa . mcfayden rlidwka" also?



%BR%
%BR%
%BR%
%BR%
%BR%
%BR%

-------
---+*SUSY MC contact notes*
-------

---++ Workflow

   * Start
      * Fill in BIG spreadsheet as far as possible
      * Assign DSIDs
      * Request spreadsheet and jobOptions tarball
      * Check jobOptions (use scripts)
     
   * Follow JIRA workflow:
      * Start at "To do" --> ask for SUSY subgroup approval - "@A and @B, has this request been discussed, and are you happy with it?"
      * When approved click "Initial sign-off" --> "In progress"
      * "Create" issue with MC prod people (see below) 
      * Add spreadsheet to SUSY MC spreadsheets page [[https://drive.google.com/#folders/0ByEl32nvxJogS3gxMmMxWXU5Yk0][here]]
      * "In progress" --> Get jobOptions and spreadsheet etc. click "Inputs/jOs" ...
      * ... this takes us to --> "Prepare"
      * "Prepare" --> Ask for validation --> "Validation" (validation optional to an extent)
      * "Validation" --> if validation ok --> "Request ready"
      * "Request ready" --> ask SUSY group convenors for approval
            * Here add in comments "We now have MC12JobOptions-XX-YY-ZZ for this, so it's up to [~jboyd] and [~onofrio] to approve this request for N {type} events in M datasets at priority Y"    
  * Now click "Group sign-off" --> "In review"
  * "In review" --> ask Physics Coordination for approval:
         * Use standard template (below) - information can be taken from spreadsheet.
<verbatim>
Subject: SUSY <MCCampaign> <CoMEnergy> Request: <description>

Hi Andreas,

The SUSY group would like to request ...<brief description (JIRA title)>... samples with <Generator(s)> at <CoMEnergy>. These samples are ...<brief justification> . The full request is for <N> <type> events in <X> datasets at priority <Y>.

The JIRA thread is here: ...

Technical details:
- The DSIDs are ...
- The spreadsheet for the request is at ...
- <type> priority <Y>
- <jobOptions tag/info>
- DESDs to be kept
</verbatim>
         * Include in cc: SUSY convenors, subgroup convenors, atlas_csc_mcprodman
      * Once approved, click "PC approved" --> "Approved"
      * "Approved" - now marked as done in JIRA

   * Request to MC prod people
      * In JIRA "create issue" 
         * Project: default
         * Issue type: Bug
         * Summary: description and DSIDs
         * Priority: Ignore this unless it is very urgent
         * Components: MCJobOptions
         * Labels: SUSY, MC12JobOptions
         * Attach tarball with jOs - needs format so that it could slot straight into MC12JobOptions
         * Description: "The SUSY group would like to request the attached files for inclusion in the next tag of MC12JobOptions.  They implement ………  There are a few dummy job options files, one modified control file, a couple of slha files, …..  Please let me know if you spot any problems."

   * If there are input files
      * Edit evgeninputfiles.csv
      * Tag updated version




   * Check if jobs are running - if not contact requester


---++ BIG Spreadsheet columns/colours
   * DS start-DS end, N samples
   * Color code: white - just came in, yellow - I need to do something, light green - jobs not submitted or still running, purple - problem with initial submission and not yet resolved, dark green - all done
   * SUSY: SUSY convenors approved?
   * ATLAS: ATLAS approved?
   * Prio: Priority in JIRA - colour RED if prio=high (3=minor,5=major,...)
   * PP: Production priority (from request spreadsheet)
   * nEe: N events EvGen only
   * nEG: N events FullSim only
   * nEF: N events FastSim only
   * SG: Which SUSY subgroup group



---++ Links
[[https://its.cern.ch/jira/browse/ATLMCPROD/component/14253][JIRA SUSY summary]]
[[https://docs.google.com/spreadsheet/ccc?key=0AiEl32nvxJogdEd5QzZ3T3FOUEw2M1lRY0U1b3VEOHc&usp=drive_web#gid=0][BIG spreadsheet]] %BR%
[[https://drive.google.com/#folders/0ByEl32nvxJogS3gxMmMxWXU5Yk0][MC spreadsheets - Google Drive]]
[[https://docs.google.com/spreadsheet/ccc?key=0AuDM1zlcTF8XdHBOSzYyeVVrdW9iajQ1UUpCaHVSMmc&usp=drive_web#gid=0][Example request spreadsheet ]] %BR%
[[https://svnweb.cern.ch/trac/atlasoff/browser/Generators/MC12JobOptions/trunk/share/evgeninputfiles.csv][evgeninputfiles.csv]] %BR%
[[https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/SUSYMcRequestProcedure][SUSYMcRequestProcedure TWiki]] %BR%
[[http://atlas-computing.web.cern.ch/atlas-computing/projects/releases/status/][Release status]] %BR%
[[https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/AtlasMcProductionMC12][AtlasMcProductionMC12 TWiki]] %BR%
[[https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/AtlasProductionGroupMC12a][AtlasProductionGroupMC12a TWiki]] %BR%
[[https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/PreparingLesHouchesEven][PreparingLesHouchesEven TWiki]] %BR%
[[https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/SUSYEvGenValidation][SUSYEvGenValidation TWiki]] %BR%
[[https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/MCProdJira][MCProdJira TWiki]] %BR%
[[http://atlas-project-mc-production.web.cern.ch/atlas-project-mc-production/register/][To register inputs/integration grids/config files]]


---++Misc

---+++Tags

   * For an Atlfast-II job the black magic of Atlfast-II happens in the reco step in this case (sim just does the ID, the calorimeter simulation actually doesn't happen until that second step) so the "recon" job gets given a second "_aXYZ" tag rather than a "_rABC" tag. E.g. e2880_a188_a171
   * For the spreadsheets the following convention should be followed - for the dummy tags  =e1111_a2222_a3333_r4444=:
<verbatim>
- e1111 goes in the "Evgen" column
- a2222 goes in the "Simul" column
- a3333 goes in the "Atlfast" column (even though it's a reco tag)
- r4444 goes in the "Atlf Merge" column (even though it's a reco merge tag)
</verbatim>

---+++Bugfix:
The usual procedure is if it is a *bugfix* and the old samples are *buggy* (read: will not be used by analyses) you can go with the same DSID and rename if necessary (Josh, make sure during the request that you ask them to mark the old ones as obsolete and schedule them for deletion).  If those old samples will be used by any analyses, then go with new DSIDs.

If it's a bug fix and it's small (20k-100k events) you don't strictly have to go through PC; you can directly iterate with the production team.  If it's more than about 100k then it's good practice to make sure PC is aware.

---+++Replacing existing jOs:
Always do a diff of new and old to make sure changes make sense

---+++New MadGraphUtils.py:
Need to request a new tag. Try to check how much has changed between new version and previous version via diff and/or ChangeLog if it's not too much then you probably Zach to) and request a new pcache with new tag. Add this new pcache to the release column of request spreadsheet.

---+++Sherpa integration grids
These are always required even for processes that are "quick" to generate.

---+++UEEE3 in 13 TeV
We have a proper energy extrapolation tune (as opposed to a tune, with adjusted parameters for specific CM energies) for 13 TeV with UEEE4(5) so UEEE3 is not an option for 13 TeV jobs (it is for 7,8,14 TeV).

%BR%
%BR%
%BR%
%BR%
%BR%
%BR%

-------
---+*Sherpa*
-------

asetup 17.2.12.7.10,MCProd,64,slc5,here...
tar czf input.tar.gz Process/ Results.db 


%BR%
%BR%
%BR%
%BR%
%BR%
%BR%


-------
---+*LondonFramework*
-------
LondonFramework

%BR%
%BR%
%BR%
%BR%
%BR%
%BR%


-------
---+*Direct stop 0-lepton 2012 8TeV data*
-------

---++Technicalities
   * Working from SUSYTools-00-01-03
   * Using p1032(1033) tags of NTUP_SUSY

---++Code
Code can be found here: https://svnweb.cern.ch/trac/atlasusr/browser/mcfayden/SUSY/rel17_8TeV/stop/trunk/
   * Object definitions applied in:
      * [[https://svnweb.cern.ch/trac/atlasusr/browser/mcfayden/SUSY/rel17_8TeV/stop/trunk/analysis/shared/NTAnalysisBase][analysis/shared/NTAnalysisBase/NTGet*.cxx.h]] (Physics objects)
      * [[https://svnweb.cern.ch/trac/atlasusr/browser/mcfayden/SUSY/rel17_8TeV/stop/trunk/analysis/shared/NTAnalysisBase][analysis/shared/NTAnalysisBase/NTAnalysisBase.cxx/.h]] (Tool initialisations)
      * [[https://svnweb.cern.ch/trac/atlasusr/browser/mcfayden/SUSY/rel17_8TeV/stop/trunk/analysis/shared/NTAnalysisBase/NTMasterFunctions_V1_rel17_SUSY.cxx.h][analysis/shared/NTAnalysisBase/NTMasterFunctions_V1_rel17_SUSY.cxx.h]] (Overlap removal and scale factors)

   * Kinematic variables defined in:
      * [[https://svnweb.cern.ch/trac/atlasusr/browser/mcfayden/SUSY/rel17_8TeV/stop/trunk/analysis/progs/rel17_8TeV_stop/NTKinematics.cxx][analysis/progs/rel17_8TeV_stop/NTKinematics.cxx]]
      * [[https://svnweb.cern.ch/trac/atlasusr/browser/mcfayden/SUSY/rel17_8TeV/stop/trunk/analysis/progs/rel17_8TeV_stop/FillHists.cxx][analysis/progs/rel17_8TeV_stop/FillHists.cxx]]
      * [[https://svnweb.cern.ch/trac/atlasusr/browser/mcfayden/SUSY/rel17_8TeV/stop/trunk/analysis/progs/rel17_8TeV_stop/rel17_8TeV_stop.cxx][analysis/progs/rel17_8TeV_stop/rel17_8TeV_stop.cxx]]


---++Main changes wrt 2011 object definitions

Jets:
   * Now using *AntiKt4LCTopo* jets rather than *AntiKt4TopoNewEM*
   * No calibration required anymore

b-jets:
   * No new recommendations afaik

Electrons:  
   * electron quality flags wrong in D3PDs (not sure which tags) -> define using functions, using:
   <verbatim>
   #include "egammaAnalysisUtils/IsEMPlusPlusDefs.h"
   #include "egammaEvent/egammaPIDdefs.h"
   </verbatim>

Muons:
   * Don't appear to have changed...

Triggers:
   * jet+MET:
      * EF_j80_a4tchad_xe100_tclcw_loose
      * EF_j170_a4tchad_EFxe80_tclcw
   * single electron:
      * EF_e24vh_medium1_EFxe35_tclcw
   * single muon:
      * EF_mu24_j65_a4tchad_EFxe40_tclcw

Initialisation of most tools now slightly different. See [[https://svnweb.cern.ch/trac/atlasoff/browser/PhysicsAnalysis/SUSYPhys/SUSYTools/tags/SUSYTools-00-01-03/Root/SUSYObjDef.cxx][SUSYObjDef.cxx ]].





%BR%
%BR%
%BR%
%BR%
%BR%
%BR%
%BR%
%BR%
%BR%
%BR%
%BR%
%BR%
-------
---+*trigEmulator*
-------

Emulation code for EF jet triggers in 2011 using !D3PDs. This tool should be quite versatile. Initially it was developed for emulation of EF_*a4tc* jet triggers from mc10a has only EF_*a4* type branches in the menu. From the inputted EF trigger name string it works out the type of trigger and thresholds to be applied.
The L2 decision is required as an input.

   * Emulates different jet algorithms, a4, a4tc, a10tc... etc.
   * Only requires L2 decision and trigger name as an input

---++Basic Instructions
---+++Setting up
You need to compile the .cxx file with the rest of your code (exactly how you do this depends on how you run on the D3PDs...). For example if you used a MakeClass in root you would want to do something like:
<verbatim>
gROOT->ProcessLine(".L trigEmulator.cxx+");
</verbatim>

before you start calling the function "Loop()"

Then in your main analysis file include the header:
<verbatim>
#include "trigEmulator.h"
</verbatim>
and in your main analysis function but outside the event loop you should do something like this:
<verbatim>
trigEmulator::trigEmulator myEFTrig;   
</verbatim>

---+++Feeding D3PD branches
Then inside the event loop you need to give the emulator the branches it needs something like this (again it depends on your setup how you call the branches etc.):
<verbatim>
myEFTrig.jetpt=(*trig_EF_jet_emscale_pt);
myEFTrig.jeteta=(*trig_EF_jet_emscale_eta);
myEFTrig.jetphi=(*trig_EF_jet_emscale_phi);
myEFTrig.type_a4tc=(*trig_EF_jet_a4tc);
myEFTrig.type_a4=(*trig_EF_jet_a4);
myEFTrig.type_a10tc=(*trig_EF_jet_a10tc);
myEFTrig.ptThresh=30.;
myEFTrig.etaRange=3.2;
myEFTrig.EF_metx=trig_EF_met_MEx;
myEFTrig.EF_mety=trig_EF_met_MEy;
</verbatim>
  
---+++Usage                     
Then again inside the event loop, you get the emulated EF decision by passing the L2 decision and the string of the EF trigger you want like this:                                                                                                                                                                   
 <verbatim>
bool testBit_EF_j75_a4tc_EFFS = myEFTrig.isPassed(L2_j70,"EF_j75_a4tc_EFFS");        
</verbatim>



---++Files:

   * [[%ATTACHURL%/trigEmulator.cxx][trigEmulator.cxx]]: trigEmulator class C++ file

   * [[%ATTACHURL%/trigEmulator.h][trigEmulator.h]]: trigEmulator class header file

   * [[%ATTACHURL%/steer.C][steer.C]]: Steering file for !MakeClass example

   * [[%ATTACHURL%/testClass.C][testClass.C]]: !MakeClass .C file

   * [[%ATTACHURL%/testClass.h][testClass.h]]: !MakeClass header file


<!--For significant updates to the topic, consider adding your 'signature' (beneath this editing box)-->
*Major updates*:%BR%
-- Main.JoshMcFayden - 16-May-2011

<!--Please add the name of someone who is responsible for this page so that he/she can be contacted if changes are needed.
The creator's name will be added by default, but this can be replaced if appropriate.
Put the name first, without dashes.-->
%RESPONSIBLE% %REVINFO{"$wikiusername" rev="1.1"}% %BR%
<!--Once this page has been reviewed, please add the name and the date e.g. Main.StephenHaywood - 31 Oct 2006 -->
%REVIEW% *Never reviewed*

%STOPINCLUDE%
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
text	sendJobs.sh.text	r1	manage	9.3 K	2015-04-30 - 15:00	JoshMcFayden	Pathena evgen job submission script
c	steer.C	r1	manage	0.7 K	2011-05-16 - 20:45	UnknownUser	Steering file for MakeClass example
c	testClass.C	r1	manage	8.5 K	2011-05-16 - 20:46	UnknownUser	MakeClass .C file
h	testClass.h	r1	manage	556.7 K	2011-05-16 - 20:47	UnknownUser	MakeClass header file
cxx	trigEmulator.cxx	r1	manage	6.0 K	2011-05-16 - 20:42	UnknownUser	trigEmulator class C++ file
h	trigEmulator.h	r1	manage	1.2 K	2011-05-16 - 20:43	UnknownUser	trigEmulator class header file
Topic revision: r38 - 2018-11-06 - JoshMcFayden
Account
- Log In
- Cern Search
- TWiki Search
- Google Search
Sandbox All webs
Edit
Attach
Copyright &© 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback