See index in AmnonHarelDataMixing

Testing Mike's DataMixer, this time with the cmsDriver technology. Goal was to see a plot of CRAFT energies (here, probably a muon) overlayed directly on top of simulated signal (100GeV pion gun). See test #1 below. But while getting there, run into the odd looking overlay in "test #2".


  • mixing freshly generated MC with existing data runs almost "out of the box:"
cmsrel CMSSW_3_1_0_pre11
cd CMSSW_3_1_0_pre11/src
addpkg Configuration/PyReleaseValidation V01-15-10
addpkg Configuration/StandardSequences V01-17-00
addpkg Geometry/CMSCommonData V01-05-26
addpkg IOMC/RandomEngine V00-11-12
addpkg SimCalorimetry/HcalSimAlgos V02-06-07
addpkg SimGeneral/DataMixingModule V00-02-02
scram b SinglePiPt100.cfi -s GEN:ProductionFilterSequence,SIM,DIGI,DATAMIX,L1,DIGI2RAW,HLT -n 10 --conditions FrontierConditions_GlobalTag,STARTUP31X_V1::All --eventcontent FEVTDEBUG --no_exec

  • Can mix existing MC and data as described below

  • Interesting overlay test #1: data (HCAL towers & no rechit --> I can't show digis for this one) + MC (HCAL towers & digis) --> mixed (HCAL towers & digis)

  • The good news is that when adding significant energy (test #1), you get more than when adding insignificant (i.e. below RECO thresholds) energy (test #2)

  • The bad news is that in both cases the energy (digi charge, to be precise) is lower than that in the original MC
    • difference is about 4.2GeV or 82/440 photoelectrons (Shuichi: 0.2fc/GeV, Rick: 20 photoelectrons per GeV)

  • Rick found that the difference is an unlucky statistical fluctuation from the photostatistics, which use a Poisson distribution for the number of photons collected.
    • To turn it off in further tests (in the digi and mixing stages):
                 process.simHcalUnsuppressedDigis.he.doPhotoStatistics = False
                 process.mixData.he.doPhotoStatistics = False
    • After turning them off, Rick gets test #2 (@ HE +23,41,1)
      • Data: 0:3.42646, 1:1.38063, 2:2.33683, 3:1.64851, 4:3.42646, 5:1.38063, 6:1.23356, 7:3.86261, 8:2.31732, 9:1.38063
      • MC: 0:2.67812, 1:2.43128, 2:2.67594, 3:1.64185, 4:90.37, 5:44.0923, 6:16.0231, 7:7.26294, 8:5.27362, 9:3.55629
      • Mixed: 0:3.42646, 1:1.38063, 2:2.33683, 3:2.20218, 4:96.6625, 5:38.6091, 6:13.5729, 7:9.10315, 8:4.75369, 9:2.43804
    • So it looks like I was bit by the photo statistics in the MC digitization stage

Minor Quibbles

  • cmsDriver's help messages don't mention choosing sequence(s) for a step

Mike's Instructions


Mike: you need the latest version of EventFilter/RPCRawToDigi. Only use "mixData" as the InputTag.

Warren Clarida got it working in /uscms/home/wclarida/SIM_PROGS/CMSSW_3_1_0_pre11/src (with the GFlash option).

Warren: The only thing that I changed was the global tag from STARTUP_31X to STARTUP31X_V1

New package list

SimGeneral/DataMixingModule V00-02-02
Configuration/PyReleaseValidation V01-15-10
Configuration/StandardSequences V01-17-00
Geometry/CMSCommonData V01-05-26
IOMC/RandomEngine V00-11-12

First round of instructions

you need to add these tags on top of a CMSSW_3_1_0_pre10

cvs co SimG4Core/GFlash
cvs co -r testGflashAndDM Geometry/CMSCommonData
cvs co -r testGflashAndDM Configuration/PyReleaseValidation
cvs co -r testGflashAndDM Configuration/StandardSequences
cvs co -r V02-06-07      SimCalorimetry/HcalSimAlgos
cvs co -r V00-02-02      SimGeneral/DataMixingModule
and then you can use these commands

For DataOnSim: SinglePionPt100.cfi -s GEN:ProductionFilterSequence,SIM,DIGI,DATAMIX,L1,DIGI2RAW,HLT -n 10 --conditions FrontierConditions_GlobalTag,STARTUP_31X::All --gflash --eventcontent FEVTDEBUG --no_exec
(all one line)

you may still need the HcalSimAlgos.

How I got there

  • data area: /uscms/d2/aharel/
  • work area: /uscms/home/aharel/CMSSW_3_1_0_pre11
  • setup: see below under technical details
  • non-collision data: same as in
  • simulated collisions: FullMC.root which I created from the 100GeV pion gun config as detailed below
    • key point: need to keep the digis. I just used "keep *"
    • dumping the information I need from the MC with --> dumpGun.root
    • redid with 500 events MC500.root, --> dumpMC500.root
    • redid with 5K events and 50GeV pions MC5K.root ...
    • redid, dumping all recHits (bug in my configs that do so fixed Aug 10) with
  • mixed events: hacked the cmsDriver-produced mixer config file to use the existing MC file. Details below.
    • key point: rerunning the digi step
    • dumping mixed events into my format with --> dumpnm.root

ls StandardSequeces.* | awk -F. '{system("ln -s "$0" "$1".StandardSequences."$3)}'

Specific mixing test(s)

Test #1

Found (*) the location (HCAL tower) with the highest overlap between MC500 and the data file: ('25, 41', 8.9916669999999996, [84], [384])

  • Copied that data file with --> D1event.root
  • checked it's the right event with -> dumpD1.root --> this fails because reco wasn't run (see gdd1.log).
  • reco & dump with (earlier attempts, and aren't quite right)
  • The data event (#348) is from Run: 68021 Event: 14881518
  • This check agrees with the digis I got from the full data file: HCAL towers & digis
  • Mixed the events with (fixed the secondary input file, failed to turn on the needed debug statements)
  • dumped the mixed digis with
    • Results are a bit confusing. Doesn't help that I filtered some recHits. Rerunning with no thresholds (dumpnmt1a, and dumpMC500a). Fixed a bug in my to do so (Aug 10).
  • added debug prints and realized the wrong data event is used --> I haven't managed to switch the secondary source.
  • corrected a typo (fileNames, not filenames) and did which takes the CRAFT event I wanted.
    • dumped with

An alternative approach to try: take the most energetic MC hit, and aim the pion gun at it at various energies.

Test #2

This is what actually does, in contrast with what I meant it to do: it mixes data event 68021 event: 7823771 on top of MC event #84.

  • data (HCAL towers & no rechit --> I can't show digis for this one) +

Note that the hardest digi, at time slice 4 (i.e. the fifth one), went DOWN by a lot. Considering the non-linearities, this is a lot of energy being subtracted. Is that a reasonable thing to do?

Some answers:

  • Shuichi: the fC are already linearized. This is ~4.2GeV.
  • Rick: This is about 4 sigma, so not just a statistical fluctuation

Technical details

(*) found by (a) root macro loops over file, updates an std::map, outputs highest energy seen per tower location and the event number where that happened, (b) macro run over data and MC files, (c) python script finds locations that appear in both, and prints them sorted by energy.

  • Was surprised to realize it's exactly the same data input files used in CMSSW_2. It also works for CMSSW_3.

  • setup:
cmsrel CMSSW_3_1_0_pre11
cd CMSSW_3_1_0_pre11/src
addpkg Configuration/PyReleaseValidation V01-15-10
addpkg Configuration/StandardSequences V01-17-00
addpkg Geometry/CMSCommonData V01-05-26
addpkg IOMC/RandomEngine V00-11-12
addpkg SimCalorimetry/HcalSimAlgos V02-06-07
addpkg SimGeneral/DataMixingModule V00-02-02
scram b
  • (Selected) error messages, all due to missing configs
    • compiing src/Configuration/StandardSequences/: GflashDigi_cfi, GflashSim_cfi, mixLowLumPU_mixProdStep1_cfi, mixProdStep2_cfi
    • compiling src/SimGeneral/DataMixingModule/python/supplementary/: StandardSequeces. DataMixerDataOnSim_cff, StandardSequeces.DataMixerSimOnSim_cff, ...
  • Note: Warren also has some head version of SimG4Core/GFlash, but he doesn't have EventFilter/RPCRawToDigi
  • SinglePiPt100.cfi -s GEN:ProductionFilterSequence,SIM,DIGI,DATAMIX,L1,DIGI2RAW,HLT -n 10 --conditions FrontierConditions_GlobalTag,STARTUP_31X::All --gflash --eventcontent FEVTDEBUG --no_exec --> ImportError: No module named GflashSim_cfi
  • SinglePiPt100.cfi -s GEN:ProductionFilterSequence,SIM,DIGI,DATAMIX,L1,DIGI2RAW,HLT -n 10 --conditions FrontierConditions_GlobalTag,STARTUP_31X::All --eventcontent FEVTDEBUG --no_exec ---> produced config file --> Failed to run due to unknown global tag.
  • SinglePiPt100.cfi -s GEN:ProductionFilterSequence,SIM,DIGI,DATAMIX,L1,DIGI2RAW,HLT -n 10 --conditions FrontierConditions_GlobalTag,STARTUP31X_V1::All --eventcontent FEVTDEBUG --no_exec ---> produced config file --> cmsRun it --> SinglePiPt100_cfi_GEN_SIM_DIGI_DATAMIX_L1_DIGI2RAW_HLT.root output file and the log file SinglePiPt100.log ---> works.

  • Controlling the inputs:
    • take1: added package Configuration/Generator, add file, that reads QCD3events.root from my data area
      • notes symbolic links don't work, must place a copy of the file there AND it must end in, plain .py won't work frown AND probably need to run scram b again.
      • Using it instead of SinglePiPt100.cfi: ReadTest_cfi -s GEN:ProductionFilterSequence,SIM,DIGI,DATAMIX,L1,DIGI2RAW,HLT -n 10 --conditions FrontierConditions_GlobalTag,STARTUP31X_V1::All --eventcontent FEVTDEBUG --no_exec
      • Can't hack it. ProductionFilterSequence must not be: missing, empty, pointing to a source, or pointing to a different EDAnlyzer (tried EventContentAnalyzer)
    • take2: hack the working config instead (as suggested by Mike). --> Converged to config from take3 below. Problems with MC file? Tried hacking it to just make the MC file -->
      • now keeping all fields with --> FullMC.root. Dumped to gdFullMC.log: found the following
      • HCAL rechits are now in: HBHERecHitsSorted + "hltHbhereco" + "" "HLT"
      • ECAL rechits are now in: EcalRecHitsSorted + "hltEcalRegionalJetsRecHit" + "EcalRecHitsEB" "HLT"
      • HCAL Digis are now in: HBHEDataFramesSorted + "hltHcalDigis" + "" "HLT"
    • take3: get cmsDriver to produce the new MC for me? SinglePiPt100.cfi -s GEN:ProductionFilterSequence,SIM,DIGI,L1,DIGI2RAW,HLT -n 10 --conditions FrontierConditions_GlobalTag,STARTUP31X_V1::All --eventcontent FEVTDEBUG --no_exec --> --> cmsRun it --> SinglePiPt100_cfi_GEN_SIM_DIGI_L1_DIGI2RAW_HLT.root & AutoMakeMC.log
      • dumped with to see what's there --> gdMC.log --> jets are there. Digis are there (e.g. HBHEDataFramesSorted + "simHcalDigis" + "" "HLT"), but where are the recHits?
    • Variants of above, starting with the FullMC files from take2
      • running without the generation, sim, and digi steps gave the error in nm_no_sim_no_digi_fails.log
      • putting the digi step back in ( - it runs! (nm.log) --> nm.root
        • dumped content ( --> all the fields names now contain "MIXTEST" (chosen in, while the original are still available with "HTL" (chosen in E.g. mixed digis are in HBHEDataFramesSorted + "hltHcalDigis" + "" "MIXTEST"
      • dumping mixed events into my format with

Old work area

  • /uscms/home/aharel/CMSSW_3_1_0_pre10
  • setup: Didn't need the HcalSimAlgos so far.
  • running:
    • Got an error error: no such option: --gflash ---> trying without this option
    • Looks like the config name is SinglePiPt100.cfi (Pi not Pion)
    • Got an error ValueError: Step DATAMIX unknown

-- AmnonHarel - 02 Jul 2009

  • MC digis (test #1):

Topic attachments
I Attachment History Action Size Date Who Comment
GIFgif sad4_210.gif r1 manage 17.3 K 2009-08-10 - 16:40 AmnonHarel CRAFT towers (test #2)
GIFgif sad4_384.gif r1 manage 16.9 K 2009-07-16 - 11:44 AmnonHarel CRAFT towers (test #1)
GIFgif sam5_84.gif r1 manage 15.3 K 2009-07-16 - 11:43 AmnonHarel MC tower (test #1)
GIFgif sax1_2.gif r1 manage 15.7 K 2009-08-10 - 16:42 AmnonHarel Mixed towers (test #2)
GIFgif saxb2.gif r1 manage 15.8 K 2009-08-10 - 18:12 AmnonHarel Mixed towers (test #1)
PNGpng sxd4_384h441.png r1 manage 28.5 K 2009-07-16 - 11:43 AmnonHarel CRAFT digis (test #1)
PNGpng sxm5_84h441.png r1 manage 32.0 K 2009-07-16 - 11:39 AmnonHarel MC digis (test #1)
PNGpng sxx1_2h441.png r1 manage 26.2 K 2009-08-10 - 16:42 AmnonHarel Mixed digis (test #2)
PNGpng sxxb2h441.png r1 manage 26.4 K 2009-08-10 - 18:12 AmnonHarel Mixed digis (test #1)
Edit | Attach | Watch | Print version | History: r23 < r22 < r21 < r20 < r19 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r23 - 2009-08-27 - AmnonHarel
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback