TWiki> Main Web>TWikiUsers>KennethDavidLong>KDLLogBook (revision 21)EditAttachPDF

Daily web log book


2014 04 22

  • Ran reweight on 100,000 event pp --> w+zjj --> l+l-l+vl with SM_LT012 model data set.
    • It appears that this process is not currently configured to run on Condor, and it is veeeery slow:
      • First reweight completed at 1:46 (AM) April 20
      • 31st reweight completed at 0:45 April 23
    • Files are located in ~/cms/Madgraph/generated_events/w+zjj_ewk_LT_reweights and in /nfs_scratch/kdlong/w+zjj_ewk_LT_reweights

2014 04 23

2014 04 25

  • Created a simple text file for easier submission of Madgraph results. I should have done this a long time ago!
    • Contents of survey_refine.cmd:
      • survey
      • refine 100000
      • refine 100000
      • refine 100000
      • refine 100000
    • Contents of combine.cmd
      • combine_events
      • store_events
  • Run processes as follows:
    • Initially edit file Cards/me5_configuration.txt to:
      • run_mode = 1 (cluster mode)
      • cluster_type = condor
      • cluster_queue = None
    • Run: nohup ./bin/madevent survey_refine.cmd > survey_refine.out &
      • Note: nohup is roughly equivalent to using C-z then bg (which is equivalent to &, sends processes to background) + disown. This completely detaches the jobs from the shell.
    • Edit Cards/me5_configuration.txt to:
      • run_mode = 2 (combine_events script gives error when run on cluster)
      • Run: nohup ./bin/madevent combine.cmd > combine.out &

  • Attempting to create a 100,000 event data set with the process pp --> w+l+l- QED = 6, w+ --> l+ vl. This allows the inclusion of zg interference and is more realisitic.
  • Generated 1000 events using generate p p > w+ l- l+ j j QED=6, w+ > l+ vl with default settings in the SM. Stored as wpz_zg_default_sm in /nfs_scratch/kdlong and ~/cms/Madgraph/generated_events.
    • I do get quite a few errors during the survey command, of the form: ^[[1;34mWARNING: resubmit job (for the 1 times) ^[[0m
    • See survey_refine.out of these respective directories for details. This doesn't seem to affect the refine process in this case, the events were still generated without apparent issue.
    • It seems that the error is related to submission to the condor cluster and not with madgraph. A similar problem is discussed here:
  • Ran 100 events with SM pp --> w+zjj process to see if this error is related to the process. I still get errors in the survey process of the same form. It again does not cause issues in the refine process.
    • Files are stored in nfs_scratch/kdlong/w+zjj_ewk_sm. See survey.out for error.

2014 04 26

  • I didn't do).
  • Run as follows:
    • ./bin/madevent
    • import model SM_LT012
    • define p g u c s d b u~ c~ s~ d~ b~
    • define j = p
    • define l+ e+ mu+ ta+
    • define l- e- mu- ta-
    • define vl ve vm vt
    • define vl~ ve~ vm~ vt~
    • generate p p > w+ l- l+ j j QED=6, w+ > l+ vl
    • output wpz_zg_ewk

2014 04 30

ERROR: Failed to set ClusterId=549192 for job 549192.0 (110)

ERROR: Failed to queue job.

ERROR: Failed to set ClusterId=549293 for job 549293.0 (110)

ERROR: Failed to queue job.

2014 05 06

  • Met with Dan Bradley to discuss the condor errors I have been consistently seeing, e.g. (2014 04 25). As expected, the issue comes from jobs running on machines that are not "properly configured" for MadGraph. Specifically, some machines do not have CVMFS. Dan Bradley fixed this by making CVMFS are requirement for all jobs submitted by CMS users.
  • It's not clear why Matt had not encountered this issue, but it should now be fixed for everyone.

2014 05 15

  • Matt has been investigating the differences between the CMS recommended card (which does not converge for Z-gamma interference processes) and the default Madgraph card. It appears that the convergence errors are linked to dRll = 0 (separation between leptons) and ptl = 0 (minimum pt for leptons). Using dRll = 0.001 and ptl = 5 does converge. The current run_card.dat, modified from Matt's, is attached.
  • Currently running zg interference process including b quarks and new physics with this run_card.dat, located in /nfs_scatch/kdlong/wpz_zg_ewk_all_aqgc_FT1

  • I now understand the solution to the "Failed to set ClusterId=549293 for job 549293.0 (110)", "Failed to queue job." problem, referenced, e.g. on 2014 04 30. The solution is embarassingly simple: The run_card.dat file has the code path directory hardcoded in. The errors arise when I try to copy another run_card.dat and use it!
  • Investigating running reweight feature on condor cluster. Running 31 reweights took ~2 days (see 2014 04 22).
    • Parellelizing the reweight process is addressed here:, but the solution is unclear.
    • Matt suggest splitting the 100,000 event data file into 100 pieces and reweighting the full sequence of events seperately. This would be easy to implement, and is likely the best option, though it does come with a fair amount of inconvenience, such as file transfers and the large amount of files which would be produced.
    • I have submitted a question to the authors asking for more information about how they suggest to do this.
  • Registered for Fermilab computing account. Will attend a hadron callorimeter upgrade discussion/class tomorrow:

2014 05 16

  • Attended HATS class on hadron calorimeter upgrade at Fermilab
  • Recieved an answer from the Madgraph authors. They suggest that Matt's prescription is easiest, but also suggest how one would parallelize code.
  • 100,000 event zg interference process from yesterday appears to have successfully run survey/refine.

2014 05 19

  • The 100,000 event zg interference process only created 660 events after the combine_events stage. Not sure why this is, apparently I did not run enough refines. Trying again.
  • Discussed reorganizing the WpZ_ana.C code with Matt.
    • He agrees that a large reorganization of the code is a good idea and worth the effort.
    • Looking into creating a make file to run independently from root. Getting errors from Shared Object files. Possible more of a hassel than I realize.
    • See GitHub for for log of edits.

2014 05 20

  • 100,000 event zg interference process ran and combined successfully. Located in /nfs_scratch/kdlong/wpz_zg_ewk_all_aqgc_FT1 and ~/cms/Madgraph/generated_events/wpz_zg_ewk_all_aqgc_FT1
  • Sucessfully compiled, linked, and ran the WpZ_ana.C file from a make file. The path of the file has to be given to LD_LIBRARY_PATH so it can be found at run time.
    • Added LD_LIBRARY_PATH= $LD_LIBRARY_PATH:~/WpZ_ana/lib to .bashrc file.

2014 05 21

2014 06 18

  • Attended HATS class at Fermilab on Jet substructure

2014 06 19

  • Attended second day of HATS class at Fermilab on Jet substructure
  • Spoke with Matt about physics details of project
    • Important Note: values in param card are in units of 1/M^4, with M in units of GeV. Thus a mass scale of 1 TeV gives param = 1 x 10^{-12}

2014 06 20

  • Day off in Chicago

2014 06 23

  • Working on root script for reading histograms from .root file created by WpZ_ana program that compares plots from a specified weight 'number' (it's number in reweight_card.dat with the SM plot
    • modified LHEWeights.cpp file to indentify the SM weight as set param = 0.0 and append _SM for this histogram

2014 06 30

  • The reweighting for 10000 events /nfs_scratch/kdlong/wpz_zg_ewk_all_LT012 seems to have run sucessfully.
    • Madgraph actually includes a (perl) script for combining N .lhe.gz files, in PROCESS_DIRECTORY/bin/internal
    • run as ./ file1.lhe.gz file2.lhe.gz ... fileN.lhe.gz combinedfile.lhe.gz banner.txt where banner.txt is an empty file
    • Obviously running 1000 cammand line arguments isn't feasible. Wrote to create a simple shell script for combining N files of the form unweighted_events_XXX.lhe.gz
    • Tested by comining the 100 .lhe.gz files from this reweighting
  • Finished reworking code to add the following plots:
    • pt and eta for each (3) leptons in an event, orded by highest pt
    • pt_ll and m_ll for each combination of lepton particle anti-particle in an event. For events with 3 same flavor leptons, these are ordered by the distance os m_ll from the Z mass.
  • Began 100,000 event processes in /nfs_scratch/kdlong/wpz_zg_ewk_all_LT012 using run_card.dat with mmll = 12.
Command "reweight" interrupted with error: AttributeError : 'Banner' object has no attribute 'lhe_version' Please report this bug on More information is found in '/nfs_scratch/kdlong/w+zjj_all_LT102_update/run_01_tag_2_debu g.log'. Please attach this file to your report.

2014 09 17

I've got to do a better job maintaining this.

  • Sorting through parton level processes generated in MadGraph
  • Define "signal" by total QCD + QED generated events - QCD only
  • In Madgraph:
    • QED = 6 QCD = 0 is "signal", but does not include QED/QCD interference. Instead use QED = 4 QCD = 2 as background and subtrack from QED = 6
    • Note that the order of QCD or QED is the maximux number of vertices allowed. Lower is also allowed, see "examples/format" at

2014 09 18

2014 09 19

  • Generated 1,000,000 event samples for QED = 6 and QCD = 4 QED = 2 pp > w+ l+ l- jj > l+ l- vl l+
  • Attempts at following the recipes I find online have been pretty frustrating. I haven't found anything yet that gives a specific example of taking in a .lhe file and converting it to EDM. My procedure keeps failing in the hadronization step with error:
    • [2] Calling event method for module BetafuncEvtVtxGenerator/'VtxSmeared'
    • Exception Message:
    • Principal::getByLabel: Found zero products matching all criteria
    • Looking for type: edm::HepMCProduct
    • Looking for module label: generator
    • Looking for productInstanceName:

2014 09 22

  • CMSSW integration continues to frustrate
  • Working on making plot of signal = (QED6) - (QED4 QCD2). Can make the histogram subtraction work in CINT but am having trouble getting it to work in my plotting script.

2014 09 23

  • Sucessfully made QED = 6 QCD = 0 and [QED6 - (QED4 QCD2)] "signal" comparison plot.
  • Presented at meeting and met with Matt
  • Matt points out that the errors in my CMSSW generation appear to be from the first (LHE->EDM) step not working correctly
  • Met with Robert (undergraduate student working with Matt) and passed on my version of the WpZ_ana code. Noticed a few updates that should be made:
    • Make a simpler command line argument interface to accept a root and lhe file at run time
    • Create a simple technique for making unweighted plots (no lhe file)
    • Make LHEWeights function compatible with weights that set multiple Lagrangian parameters.
  • A few other points from talking with Robert not related to analysis code:
    • Update condor reweight code with a simple user interface so that it can be passed to Robert
    • Matt, Robert, and I should be doing a better job making sure we are on the same page with our MadGraph release and run/configuration cards.
  • Wrote script to allow commonly edited parameters to be changed from default cards while keeping run_card parameters that we don't usually change untouched. Also takes care of setting run_mode in me5_configuration.txt so that survey/refine is done via condor but combine/store is not.
    • Script is located at /nfs_scratch/kdlong/Default_MadGraph_Run. Invoked as ./run_madevent <path_to_run> <num_events> <COM energy in TeV>
    • Tested on Tyler's username sucessfully. Hopefully will work out of the box for Matt and Robert.

2014 09 24

  • Emailed Matt and Robert about madevent script implementation
  • Sucessfully updated code to run with no LHE file
  • Trying to fix LHEWeights to cooperate with multiple parameters set. Getting the same mysterious erros with file reading hanging up for no apparent reason as at the beginning of the summer.
Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatdat run_card_14_5_15.dat r1 manage 15.8 K 2014-05-15 - 23:58 KennethDavidLong Current run_card.dat for full simulations at 13 TeV as of May 15, 2014
Edit | Attach | Watch | Print version | History: r24 < r23 < r22 < r21 < r20 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r21 - 2014-09-25 - KennethDavidLong
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback