-- AmnonHarel - 21 Feb 2009

Relevant hyperlinks

• AmnonHarelDijetCentralityRatio - a newer page for the publication push
• AmnonHarelCLsInDCR
• CMS AN-2010/126; PAS EXO-100-02: v8; CDS entry of ICHEP preliminary
• analysis management page - with PAS link, PAS (link broken?), Official analysis Wiki page - EXO10002, Exo-10-002 hyper news
• Bora's trigger turn on curves and More information for 2.84nb-1; Jet 50 curves: inner & outer. New jet 50 curve with 0.83pb-1
• Jim's input fits:
  • For publication push - 2.875pb-1: Jim's data plots, data in .root file, data with various models
    • Signal fits (after inner mass fix), plot for Pythia ratio fit, extended contact interaction MC and fit, demonstrating need for turn offs there, plots of total uncertainties.
  • 2.96pb-1 (from an unofficial dataset): data summary plots, extended MC
    • plots underlying contact interaction signal fits, Signal fits (after reviewing fit ranges, with 44 lambdas).
  • 254nb-1: data summary plots, root file
  • ICHEP - 120nb-1: data summary plots, Overwritten: Signal fits (after reviewing fit ranges, with 40 lambdas)
  • 50 nb-1: data summary plots. Updated contact interaction models (two more lambdas) added in the same place.
  • june 17 data - 11nb-1: data summary plots,detailed plots ("postPreApproval"), and (overwritte:SM and contact interactions (lambda>=350GeV) functions and parameters (corrected NLO))
    • signal uncertainties
  • june 14 data: SM and contact interactions (lambda>=500GeV) functions and parameters (corrected NLO), but there's a bug in the contact interaction fits, detailed plots ("postPreApproval"), and data summary plots,
  • older: Functions and numbers, contact interactions inner fits, contact interaction turn on plots and additional plots, may 20th plots
• data quality plots from Chiyoung (via Jim)
• Pythia effects study for the non-perturbative corrections (from Jim)
• Kostas' run info text file
• Selda's MC fits
• Dan's summary plot for relative JER variation
• Discussion of choosing a normalization region
• AmnonHarelStatusReportForStatisticsBoard (outdated by now - this was from before the first meeting with the SB)
• AmnonHarelFutureLimits, AmnonHarelUnderstandingJESuncertainties (somewhat outdated, Cousins viewpoint via systematics on statistics-only limit is more powerful)
• Eta cut optimization for resonance from Jason

Inputs to statistical analysis

Contact interaction models @3pb

• The SM line is not visible since it is behind the other lines - all contact interaction models are set to SM above lambda

Technical tests

SM models @3pb

Effects of systematic variations

• Correlation coefficients of the LLR with various nuisance parameters from lambda=500 (total counts from June 14 dataset of 10.5nb-1). Only 1000 PDSs per ensemble, will redo with more statistics.
  • in the contact interaction case, the LLR is dominated by the JES. This is explained below

	JES	relative JES	flat shift	scale
SM	0.07	-0.34	0.04	0.15
lambda=500	0.66	-0.13	0.07	0.01

• Complete systematic variations from likelihood-based code .root format):
  

Side band fit

• The SM ratio before and after the fit - sideband_fit_50nb.txt
• The function is implemented in C++. It's "fittedSM_evaluate_ratio" which is the first function in fittedSM.c
• Automated sideband fit output with 50nb-1:
  

JES variations

• Visualization of JES systematics in lambda=500 contact interactions:
  • this is, in effect, a huge signal modeling uncertainty
    
• Test of unbinned technique used for SM and contact interaction in [4].
• Test of binned JES systematics used in q* models (rough models from 2010-Apr-18):
  

q* modeling plots

• Test of q* interpolation:
  
• Visualization of why we use only the core of the q* shapes - there are fits created by Dan with:
  • extended tails (0.18<x<1.82)
  • upper tail fitted to smooth out statistical fluctuations (fit and its chi^2 look great, but I don't have a plot handy to demonstrate that)
    

string resonance modeling plots

Talk drafts

plots for talks

3fb push

Selected LLR plots:

Pythia QCD model
	Lambda=2.4TeV	Lambda=3.0TeV	Lambda=4.0TeV
.png
.root

Wiki not ready yet - all plots etc. are in a tar file on lxplus...

Feel good statistic:

• All numbers here are not yet final...
• The p-values are for the "with systematics" histogram. They include interpolation within the key bin, and supersede the older p-value number.
• the fit is not shown for the NLOcorr plot, as it is of very low quality (chi^2/n.d.f. ~74, fails to capture the large upper tail)
  

SM model	NLO (with non perturbative corrections	Pythia with k-factor	Pythia	Flat ratio
plot		not updated

120.0 nb-1, contact interaction results

	with systematics		without systematics
	Normal view	zoomed in	Normal view	zoomed in
.png
.root	clim1_4KI0_12d.root	N/A	clim2_4KI0_12d.root	N/A

Notes:

• The prediction and limit curves all used the observed total counts. So the natural expectation for a smooth curve is false - data fluctuations are apparent here.
  • in particular the "peaks" in LLR for 1150, 1400, and 1600 GeV are related to similar peaks in the total data counts of the 120nb-1 dataset.
  • the LLR jumps from 1700-2000 may be due to the discrete jumps in the turn_on, as in that region, each two lambda (e.g. 1700 and 1750) share the same turn on bin and value. However, this does not full explain the structure
    • these are not statistical fluctuation from the limiter ensemble sizes! Those are basically sqrt(4000)=63 times smaller than the 1sigma band =~0.022 in LLR, which is smaller than the marker size.

Feel good statistic:

• The p-values are for the "with systematics" histogram. They include interpolation within the key bin, and supersede the older p-value number.
• the fit is not shown for the NLOcorr plot, as it is of very low quality (chi^2/n.d.f. ~74, fails to capture the large upper tail)
• plots were overwritten. See AN...

Selected LLR plots:

lambda	1.7TeV	1.85TeV	2.0TeV
.png
.root	psd17001I0_12.root	psd18501I0_12.root	psd20001I0_12.root

50.0 nb-1, contact interaction results

with systematics	without systematics

Some LLR plots with systematics

Lambda=1050	Lambda=1100

Feel good statistic:

• The p-values are for the "with systematics" histogram. They include interpolation within the key bin, and supersede the older p-value number.
• the fit is not shown for the NLOcorr plot, as it is of very low quality (chi^2/n.d.f. ~74, fails to capture the large upper tail)
• plots were overwritten, can be recreated if needed

11.4 nb-1, contact interaction results

• Contact limits w. data, from June 23:
  • the purple line is the CLs line - at high lambdas the separation is low and no CLs limit exists
    
• feel good statistic @ 11.4nb-1:
  • the fit is to the no statistics curve, and of the form [0]*TMath::GammaDist(x,[1]/2,0,[2])
  • the numbers in parenthesis are the uncertainties on the last digits. However, this is a fairly rough estimate (not using Bahadur representation here)
  • the plots with the p-values of the feel good statistics were overwritten, and can be restored if need be.
• LLR plots at various interesting lambdas:

lambda	With systematics
400
450
500
600

Older plots for talks

Are exclusion probabilities linear near the limit?

Two examples:

dataset	with systematics	without systematics
11.4nb-1 data
100nb-1 predictions	not made

• contact limit plots

integrated luminosity	full plot	zoom in	without systematics
100nb-1
300nb-1

• x-sec limits for 700GeV q*:
  

integrated luminosity	full plot	zoom in	without systematics
300nb-1

• PLOT IS SUSPECT!
  x-section limits summary for 700GeV q* @ 100nb-1: PLOT IS SUSPECT
  
• limits on 500GeV q* @ 300nb-1, without systematics:
  

Performance with the different "modes" of limit setting

• LLR for lambda=1TeV, "old style"=each ensemble used only its hypothesis, so that in the NP ensembles the total event counts came from the NP hypothesis.
  • This is a valid analysis, but no the one we want and perform in "data mode" (below). This kind of limit setting implicitly uses the total counts, spoiling our orthogonallity to the dijet spectrum measurement.
    
• LLR for lambda=1TeV, predictions mode (both ensembles use the SM total event counts):
  
• LLR for lambda=1TeV, data mode (both ensembles use total counts from data), fake data statistics happen to be low (3012 events):
  
• LLR for lambda=1TeV, data mode (both ensembles use total counts from data), fake data statistics happen to be high (3426 events):
  

Verifying the choice of minimal stat. uncertainty - 5%

• New study - mq* = 500GeV at 100nb-1 - the signal starts at the very first bin!
• Each shaded band represents half a sigma (from -2 to 2) on the SM distribution.
• The 5% criterion leads to starting with bin #9 (starting from the bin at 156)
• the absolute statistical errors on the first bin ranged from <1% (bin #3) to >11% (bin #13). Note that bin #13 starts at 489GeV, so about half the signal region is excluded.
• the statistical errors shows in the summary plots are the sum in quadrature of the statistical uncertainties on the 95% quantile (approximated using the "Bahadur representation") and the statistical uncertainties on the fraction of SM below the nominal cut.

first bin	LLR distributions	first bin	LLR distributions
summary plot		no syst.
1		3
5		7
9		11
13

• conclusions:
  • when the systematics are held constant, the statistical analysis is not overly sensitive to the choice of the sideband region. In this example the 5% criterion led to a decent choice of cut off, though a lower value may be slightly better.

Sensitivity to uncertainty on shifting

• same scenario as above, but the first bin is #3.
• the "1.3" number comes from adding in quadrature the expected uncertainty on a shifting constant fit to the two lowest bins in the data, i.e., in the corresponding sideband. This is for 100nb-1.

shifting uncertainty (in %, absolute)	0	1.1 (nominal)	1.3
LLR distributions

• Conclusions: no significant sensitivity, can afford using only these two bins as the sideband region.

What do things look like with a badly fluctuating dataset?

• Study done at 100nb-1. Found a pseudo dataset with a shift ~2sigma from expeced: -0.0130, fitted uncertainty: 0.0067.
• ratio of a PDS with a particularly Bad Shift (-0.0130 +/- 0.0067):
  
• contact summary for a PDS with a particularly Bad Shift (-0.0130 +/- 0.0067):
  
• contact summary for a PDS with a particularly Bad Shift (-0.0130 +/- 0.0067), but with the normal ensembles:
  

7.2 nb-1, contact interaction results

plots for lambda=500,750GeV, and summary plots with and without systematics were formerly available, but since deleted

• only the bins above 230GeV are used (optimized for this lumi).
• the plotting macros don't account for weirdness that happens at low statistics, such as negative predictions. We expect to further process plots for a publication...
  • the odd predictions are since we will only publish with enough data and enough simulated lambda values such that a linear interpolation, such as used to find the limits, suffices.

2.84 nb-1, contact interaction results

• these plots were mostly overwritten in the wiki page - can be uploaded again if needed.
• only the bins above 230GeV are used. This was chosen for 10nb-1, and isn't quite optimal for 2.84nb-1.
• the plotting macros don't account for weirdness that happens at low statistics, such as negative predictions. We expect to further process plots for a publication...
  • the odd predictions are since we will only publish with enough data and enough simulated lambda values such that a linear interpolation, such as used to find the limits, suffices.

Code

• toDan20100412.tgz: Tar file for Dan - for qstar models

• dijet resonance signal class (1st version), based on code from Sertac and Dan: resonance.h, resonance.c, also requires this utilities file to compile as is.

• gzipped tar file for source codes of binom statistics, model, & basic null-hypothesis ensembles tests
  • rBInom.c contains the poisson ratio as a binomial (per-bin) statistics functions (including ROOT wrappers)
  • model.h & model.c contain the model class, which also calculates the multi-bin statistics
  • ensembles.c contains the ensemble_tester class (just basic NULL-hypothesis tests for now).
  • usejr.c loads everything (do root -l usejr.c or from CINT .x usejr.c)
  • rjetUtil.c so far contains only plotting utils, used in rBinom's built in visual tests.

• gzipped tar file for source codes of discreteDistribution & poisRatDist class
  • discreteDistribution is a generic class, independent of ROOT.
  • poisRatDist analysis specific
    • calculation of R can be optimized further (comments in code on how to do it)
  • can calculate R(1000, 2000) in 17 sec and read it from a file in 0.3 sec
  • improved "effective binning" option
  • pri.c contains the continuum limit, testPRs.c tests it and makes plots with it

• rj_macros_20100415.tgz: Statistical analysis macros

Random info

• plot_sm_jes.png:
  
• Contact limits w. approval data, without systematics:
  
• dcr_rf1.tgz: gzipped tar file of code and workspace for mock RooStats analysis

Attachments

• LLR distributions for lambda=3TeV, 3pb-1 (QCD from Pythia):
  

• LLR distributions for lambda=3TeV, 3pb-1 (QCD from corrected NLO):
  

• 3pb-1 contact limits, without syst., corrNLO, zoomed in:
  

• 3pb-1 contact limits, without syst., pythia, zoomed in:
  

• plot_smr_syst.root: Canvas of Systematic variations of the SM ratio (update RooStats like code, 3pb-1 dataset)

• LLR distributions for lambda=4TeV, 3pb-1 (QCD from Pythia):
  

• 3pb-1 contact limits, corrNLO, zoomed in:
  

• 3pb-1 contact limits, Pythia QCD, zoomed in: