Trigger software validation

In case of difficulties with the following instructions please contact J.Klukas so that they can be updated or improved.

In order to test out layouts, render plugins, reference histograms, you will need to setup your own personal copy of the DQM GUI. You can set this up from any machine, but these instructions will assume that you using an lxplus machine (just be sure to close down the GUI when you're done). If you are accessing a GUI on lxplus from outside the cern.ch domain, you must use a proxy so that your browser can access the lxplus machine. You must change your browser settings as described on the OnlineDQMTestBed page, and create a tunnel by running the following command in a terminal window from the same machine where you'll be running your browser (modifying to fit your username and the desired lxplus machine):

• ssh -ND 11080 your_username@lxplus000NOSPAMPLEASE.cern.ch
  • It will ask for your password, and then hang. This is good; the tunnel has been established and will stay open as long as this terminal window stays open.

Now, to set up a testbed, you will need to follow sections of the documentation on the DQMTest page, but use the list below as a guide for which sections to run for our purposes:

1. Run instructions for "Installing the GUI server", with the following changes:
   • Be sure to modify the very first line of code (you only need to run this once, although it appears in every section) to point to /tmp rather than /build, generating something like this:
     DQMV=5.0.0 ARCH=slc4_ia32_gcc345 DEV_AREA=/tmp/$USER/dqmtest DATA_AREA=/tmp/$USER/dqmdata
   • Also, the default instructions have you check out from CVS anonymously, so if you would like to be able to commit the changes you make later on, be sure to leave out the following command:
     export CVSROOT=:pserver:anonymous@cmscvs.cern.ch:/cvs/CMSSW
   • For the cvs co commands, you will probably want to use the tag OFFLINE_CONFIG_PRODUCTION rather than ONLINE_CONFIG_INTEGRATION.
   • These commands fetch a large number of files, so it may take a few minutes to complete.
2. Run instructions for "Starting the GUI server", remembering:
   • You must ignore the first line, since you've already set up these variables to point to /tmp. Changing these will break the instructions.
3. Run instructions for Setting up the collector
4. Copy a file of interest to the $DATA_AREA
   • This must be a harvested output file with a name following the DQM conventions (DQM_V*_R*_DbsPathOfDataset)
   • For testing, you can copy one of the official harvested CMS.RelVal files from castor:
     • A list of the files harvested by the DQM group can be found online:
       https://cmsweb.cern.ch/dqm/offline/data/browse/ROOT
     • Copy the desired file to the $DATA_AREA:
cd $DATA_AREA
source /afs/cern.ch/project/gd/LCG-share/sl5/etc/profile.d/grid_env.sh
voms-proxy-init
curl -O -L --capath $X509_CERT_DIR --key $X509_USER_PROXY --cert $X509_USER_PROXY https://cmsweb.cern.ch/dqm/offline/data/browse/ROOT/RelVal/$release/$dqmRootFile
5. Run instructions for Registering DQM output files to the web server
6. Again, run instructions for Starting the GUI server, which causes a restart so the GUI can recognize the new file
7. Again, run instructions for Setting up the collector, to complete the restart

Now, you are ready to browse through your stand-alone GUI. The step of starting the GUI server should have generated a message on your terminal with a link to the location of the server. Access that page (in the browser where you've set up a proxy if you're outside of CERN), and you have access to your private GUI. To learn how to access the dataset you uploaded and navigate around, take a look at the Central GUI section of this page.

In case of difficulties with the following instructions please contact J.Klukas so that they can be updated or improved.

To enable the CMS.RelVal layouts, you must modify $DEV_AREA/config/server-conf-devtest.py to load hlt_relval-layouts.py and shift_hlt_relval-layouts.py. Add "hlt_relval" to the two LAYOUT definitions, like this:

LAYOUTS = ["%s/%s-layouts.py" % (CONFIGDIR, x) for x in
          ("csc", "dt", "eb", "ee", "hcal", "hlt", "hlx", "l1t", "l1temulator", "rpc", "pixel", "sistrip", "hlt_relval")]
LAYOUTS += ["%s/shift_%s_layout.py" % (CONFIGDIR, x) for x in
           ("csc", "dt", "eb", "ee", "hcal", "hlt", "hlx", "l1t", "l1temulator", "rpc", "pixel", "sistrip" , "fed", "hlt_relval" )]

Again, follow the instructions from DQMTest for "Starting the GUI server" and "setting up the collector", so that this change is picked up.

Now, you can modify shift_hlt_relval_layouts.py to define a short (5-10 histogram) layout for your subsystem. The changes should automatically be reflected in the GUI whenever you save. To get a sense of how to write these layouts, take a look at the EGAMMA section of the file, then navigate to HLT/HLTEgammaValidation/Zee Preselection/doubleEle5SWL1R to see the output that this code produces. When you click on the plots, text should appear in the floating "Description" box, corresponding to the content defined in the python file.

Example:

muonPath = "HLT/Muon/Distributions/HLT_IsoMu3/"
muonDocumentation = " (HLT_IsoMu3 path) (<a href=\"https://twiki.cern.ch/twiki/bin/view/CMS/MuonHLTOfflinePerformance\">documentation</a>)"
def trigvalmuon(i, p, *rows): i["00 Shift/HLT/Muon/" + p] = DQMItem(layout=rows)
trigvalmuon(dqmitems, "Efficiency of L1",
            [{'path': muonPath + "genEffEta_L1",
              'description': "Efficiency to find an L1 muon associated to a generated muon vs. eta" + muonDocumentation}])

HLT layout validation files:
?DQM/Integration/config/shift_hlt_relval_layout.py
?DQM/Integration/config/hlt_relval-layouts.py

Integrating Layouts Into the Offline GUI

If you are happy with your layouts, you can commit the changes directly from this folder, as long as you did not check out the files from CVS anonymously. All CMS.RelVal developers should already have been given developer access for the DQM/Integration package. Once you've tested and committed your changes, make a new tag and contact Nuno Leonardo to announce it. Once the new tag is picked up, your layouts will be visible in the Offline GUI.

In case of difficulties with the following instructions please contact J.Klukas so that they can be updated or improved.

Some quality tests simply compare against numerical thresholds (like ContentsWithinExpected), but others compare against reference histograms (like Comp2RefChi2) which must be included in your harvested file. To try out embedding such reference histograms, you create a configuration by using the following cmsDriver command (changing the CMS.GlobalTag as needed):

cmsDriver.py harvest -s HARVESTING:validationHarvesting --mc --conditions FrontierConditions_CMS.GlobalTag,STARTUP31X_V7::All --harvesting AtJobEnd --no_exec -n -1

To this, you should add the following content, replacing the workflow and fileNames with those of a GEN-SIM-RECO dataset, and changing the referenceFileName to the file you wish to use for reference histograms:

## Additions for Reference Histograms 
process.dqmSaver.workflow = '/RelValZMM/CMSSW_3_3_0_pre3-STARTUP31X_V7-v1/GEN-SIM-RECO'
process.source.fileNames = cms.untracked.vstring(
  '/store/relval/CMSSW_3_3_0_pre3/RelValZMM/GEN-SIM-RECO/STARTUP31X_V7-v1/0015/EA5A8C1C-1A9F-DE11-BA5D-000423D992A4.root',
  '/store/relval/CMSSW_3_3_0_pre3/RelValZMM/GEN-SIM-RECO/STARTUP31X_V7-v1/0015/EA4ED1BC-1D9F-DE11-B9CF-001D09F24F65.root',
  '/store/relval/CMSSW_3_3_0_pre3/RelValZMM/GEN-SIM-RECO/STARTUP31X_V7-v1/0015/B4FE952C-1F9F-DE11-B8D4-001D09F2441B.root',
  '/store/relval/CMSSW_3_3_0_pre3/RelValZMM/GEN-SIM-RECO/STARTUP31X_V7-v1/0015/62BF4CE2-209F-DE11-BD34-001D09F24489.root',
  '/store/relval/CMSSW_3_3_0_pre3/RelValZMM/GEN-SIM-RECO/STARTUP31X_V7-v1/0015/A8F75952-7A9F-DE11-B70B-001D09F24763.root',
)
process.DQMStore.referenceFileName = 'rfio:/castor/cern.ch/user/n/nuno/relval/harvest/CMSSW_3_3_0_pre2/DQM_V0001_R000000001__RelValZMM__CMSSW_3_3_0_pre2-STARTUP31X_V7-v1__GEN-SIM-RECO.root'
process.dqmSaver.referenceHandling = 'all'

When you run this configuration, the output file should include a top-level folder name "Reference", which will include the same tree of plots as the "DQMData" folder, but taken from the reference file. Now, you can use this configuration as a starting point for including comparison-based quality tests.

Take a note also of ongoing developments for db-based references (see eg pdf).

In case of difficulties with the following instructions please contact J.Klukas so that they can be updated or improved.

To define DQMQualityTests, you must add the QualityTester module in your path (in the same configuration as you used to define reference histograms), and point it to an xml file containing definitions of quality tests you want to run. You should consult the DQMQualityTests page for documentation, and look at an example xml file in CVS, or search for other examples. After you create your file, be sure to point the qTester module to it through the FileInPath parameter.

## Additions for Quality Tests                                                  
process.qTester = cms.EDFilter("QualityTester",
    qtList = cms.untracked.FileInPath('HLTriggerOffline/Muon/data/QualityTests.xml'),
    reportThreshold         = cms.untracked.string('black'),
    prescaleFactor          = cms.untracked.int32(1),
    getQualityTestsFromFile = cms.untracked.bool(True),
    qtestOnEndJob   = cms.untracked.bool(True),
    qtestOnEndLumi  = cms.untracked.bool(False),
    testInEventloop = cms.untracked.bool(False),
    verboseQT =  cms.untracked.bool(True)
)
process.qtAndDqmSaver = cms.Sequence(process.qTester * process.DQMSaver)
process.dqmsave_step.replace(process.DQMSaver, process.qtAndDqmSaver)

You can now run your updated configuration, and browse through the output file. The QualityTester module will dump strings into the folders alongside any histograms it checked. Consult the DQMQualityTests page for information on using C++ code to read these strings. Also, on the GUI, you can click on the drop-down menu for alarms to view only histograms which failed a quality test.

Integrating Quality Tests Into Automated Production

To include your quality tests in the automated production, you must place your xml file in your validation module's data area (example from muon HLT) and place the QualityTester configuration in your python area (example for muon HLT). To include these in the automatic validation, you must add your QualityTester module to the central QT validation sequence. Once you've tested everything, contact Nuno Leonardo and point him to your updated version of the central QT validation sequence so that he can commit the change.

In case of difficulties with the following instructions please contact Z.Gecse so that they can be updated or improved.

General rendering

For modifying how histograms get displayed in the gui, one may define what's referred to as 'dqm render plugins'. See CMS.DQMGuiPlugins. In such module one can set root-style aspects such as color for line/marker/fill, log scales, and so on. See a nice example from heavy flavor HLT.

Special feature: superimpose graphs

See example from heavy flavor HLT. The procedure involves: (i) combining 1D histos in one 2D histo during postprocessing; (ii) recovering and ploting them in the same canvas via the render's postDraw function.

Automated validation results are shown in the dqm offline gui. To view them:

1. install grid certificates (one time)
2. open the gui
3. click on Run #
4. in the search box, enter relval ttbar 3_3_0
5. click on Any and confirm the dataset
6. in Workspace, select HLT
7. navigate the folder structure from subsystems to desired plots
8. for accessing the summary performance plots, from Workspace select: Shift, HLT
9. for comparison of releases: click on View details in the upper-right of the window, then in the reference tab enter the intended reference dataset(s)