Online/Offline Tracker DQM Operations for 2009 DQM

The following documentation is intended to include instructions for both the SiStrip and Pixel DQM shifters

Before becoming a DQM shifter

Each shifter have to accomplish these tasks well in advance his/her shift. Different types of the Tracker Shifts are described here. Please read that page, than subscribe using the TRK-P5 (for online SiStrip shifters), Pixel-P5 (for online Pixel shifters) or TRK-Offline (for combined SiStrip and Pixel offline shifters) tag of the Central Shift Subscription Tool.

DQM Tutorials

DQM tutorials for shifters are held weekly. You can find the calendar of Tracker DQM tutorials here. Please add your name in the People subscribing column Slides can be found here

Tools for a DQM shifter (general)

Here the tools that an on/off-line shifter should master are described. Specific usage for on/off-line tasks are detailed below.

Tracker Elogs

You need to log in with your NICE account.

To add a new entry, choose New , then the Type of the entry and Subject, then type a short note below.

Central DQM GUI

Monitor of Tracker-related quantities occurs via a web application with a Graphical User Interfaces (GUI), that accommodates histograms, quality test responses, detector synoptic views, etc.

The web page to access DQM views and histograms during data-taking is:

http://cmsweb.cern.ch/dqm/online

and for off-line it is:

https://cmsweb.cern.ch/dqm/tier-0

Central GUI servers require the use of an authorized personal Grid certificate installed in your browser (only terminal screens at CMS Centers are exempt from the need). You have to:

  • be sure to be a member of CMS VO (You can register here)
  • have a valid personal grid certificate installed in your browser. Users with CERN account can request and retrieve a personal grid certificate via this link.
In case of trouble, please check how to solve them on this page.

Menus on the left bar can be accessed by left-double-clicking. Then a collection of histograms is normally shown in the main view. Left-double-click on a histogram to show it enlarged. By a new single-click you can go back to the multiple view. Do not use Back button in the browser.

Twiki Whiteboard

An additional information repository is provided here in the form of a Twiki page. That is intended as a tool to inform on/off-line shifters about temporary conditions of the Tracker that may affect what they will see. Take care of entries on the whiteboard but do not modify it, since only experts are allowed to do so. Note that whiteboard is a dynamic tool and most of past entries could disappear or not to be valid anymore.

Run Registry

The Run Registry is a web-based archiving tool for DQM. It is the main place where the most important information about all the runs, and the status of each subdetector are registered. It can be found here:

http://pccmsdqm04.cern.ch/runregistry/

Runs where the Tracker is in can be found looking at the STRIP and PIXEL tabs at the top, with a status different from EXCL. Run lists can be displayed according to the different tabs in the second row.

DQM Instructions for Online shifts

Online Tracker shifts are held in the CMS Control Room at P5. Among the tasks of the online shifter is the monitoring of the detector status using the tools of the DQM.
For shift and tutorial subscription, please read this page.

During the shift turnover the previous shifter will update the new one on the status of the runs and discuss about changes in shift instructions and/or applications (if any). He/she will also show the previous shifter where the browsers for the different applications (GUI, e-log, etc.) are.

The Tools for the Online shifter

The Tracker Online shifter is expected to have several tools under control.

Tracker Elogs

Shifter should report in Elogs only in case of problems.
  • Tracker General ELOG . Report here general problems of the Tracker (DAQ, DCS, Cooling...), if any
  • Central DQM ELOG . Report here general problems which might affect DQM histograms. This would allow Central DQM shifter to become aware of this. There is no need to post the same message twice. If a message is already posted in the above ELOG one can just add the link here. The Central DQM shifter would never check the Tracker ELOGs.
Check also the Whiteboard where the status of hardware, software, Tier-0 etc in a specific moment could be noted down by experts.

Central Online GUI

The web page to access DQM histograms during data-taking is: http://cmsweb.cern.ch/dqm/online
On desktops in P5 (that online shifters are supposed to use), Grid certificates are already installed.

In the Run field on top of the page, you can choose the ongoing run with Live, or a past run. Then use the available options from Workspace menus:

  • Summaries
    • Summary : overall status of all sub detectors in CMS can be checked from this summary page. snapshot
    • Report : report from sub-detectors used in certification snapshot
    • Shift : page dedicated for CMS Central DQM shifter
  • Tracker
    • SiStrip : histograms defined in SiStrip system can be viewed from this page which has different options
      • Quick Collection : A few sets of histograms selected from full list of SiStrip histogram. All these sets must be checked by the shifters. snapshot. The description of these sets and corresponding histograms can be found here
      • Layout : a few extra sets of histograms which might be consulted in case of unusual behavior snapshot. The description can be found in the same link as above
      • Detailed Browsing : If needed a detailed browsing is also possible snapshot
    • !Pixel : histograms that are booked and filled for the Pixel detector can be viewed from here. The shifter should check the following page for every physics run:
      • Quick collection: A set of pre-defined canvases with key histograms. The shifter must check them all for all runs. A description of these histograms and instructions on what to do if they don't look like expected, can be found here: instructions on online Pixel DQM histograms for shifters. In case problems show up in one or more histograms, the shifter is expected to diagnose the problem by navigating down the detector structure in the GUI:
      • !Pixel: detailed browsing of the detector structure available, including access to all histograms that are booked and filled for the various levels of reconstruction, down to module level.

Tracker Expert GUIs

In the Tracker DQM we have specific expert level GUIs to investigate problems once they are identified in the general purpose CMS GUI. This GUI is also web based and accessible as long as a run is active.The advantage here is that it is specially designed for the Tracker Experts and one can dig down the histogram structure up to the individual module level in an easy and efficient way. Specific GUIs are available for the SiStrip and Pixel DQM systems. They are also used to launch so-called TrackerMaps, which provide an easy overview of the entire SiStrip and Pixel detectors in one page.

Run Registry

An on-line shifter can find very useful information in the Run Registry, as: start time of a given run, trigger type and number of triggered events, B field value. You can double-click on a run entry and look at Online comments or on Comments tab for the STRIP and PIXEL subsystems. No specific action is expected by the shifter, since the tool should be reserved to central DQM shifters and subsystem experts.

The Tasks for the Online shifter

The main task is to monitor histograms, in order to assess the Run status and provide feedback to detector experts. Any inconsistency should be pointed out immediately. List of detailed tasks:

  1. Check pre-defined sets of histograms from the online central GUI for every run
    • Select from the Workspace menu the SiStrip or Pixel option. Then select the histogram groups from the Quick collection options on the left.
    • Follow the histogram descriptions here for SiStrips or here for Pixels and check carefully all defined sets under the Quick collection
    • Observed problems should be reported in the Tracker General ELOG
  2. Spot problems inside the Tracker using the SiStrip Expert GUI
    • At the beginning of a run make sure to start the local DQM process which would allow to access the Expert GUI and the TrackerMap. Use the run control interface to start the process using the link. The steps to be followed are explained in the snapshot here.
    • Open following links in two browser windows. Get in contact with experts if the process cannot start or halts
    • Keep it running as long as the run is ongoing. At the end of the run the process will terminate automatically
    • Once a problem is found in the Central GUI (histograms showing unexpected behavior) the Expert GUI and the TrackerMap should be used to further investigate the problem. These tools are especially designed to locate problems in different mechanical sub-detectors or even at module level following the complex structure of the Tracker.
    • Finally identified problems should be reported in the SiStirp Tracker General ELOG

BEWARE:

  • Reconstruction in Tracker takes generally a larger time than in other subdetectors, so it could happen that sometimes Tracker histograms in the GUI do not update as frequently as the others. This is a normal situation and central DQM has NOT to be contacted for this
  • temporary problems related to DQM plots can be reported only to the Tracker DQM experts, tel 16-2389

Histograms and Errors in the online GUI

For both SiStrip and Pixel there are pre-defined groups of histograms which should be checked carefully. The description of the shift histograms and the actions to be taken in case of unexpected behavior are described in the following links :

NOTE : Automatic Quality Tests (QTest), based on statistical parameters, are performed on a sub set of the available histograms. Alarms are raised accordingly when deviations are observed. They can be visible as a frame around each histogram when a histograms directory is opened. GREEN blinking frames indicate that everything is consistent with expectations. RED permanent frames correspond to quality test failed or not performed for some reasons.

Check the SiStrip Tracker-Expert GUI in case of problems

A number of options are available in the SiStrip Tracker-Expert GUI to locate a problem. The description and the usage are described in following links

The Pixel Tracker-Expert GUI is currently used only to launch the Pixel TrackerMap. The usage is described in the following link:

Book results

  • Information for problematic/bad runs identified from pre-defined histograms in Central GUI, Tracker Expert GUI or TrackerMap should be inserted in the !SiStrip Tracker General ELOG (for SiStrip problems) or !Pixel Tracker General ELOG (for Pixel problems) respectively. Each run should be entered separately. The reason to classify the run as bad should be clearly written. There is NO need to insert good runs.
  • The status of the run should be checked from the Run Registry and the comments of central DQM shifter should be cross checked
  • At the end of the shift a summary report should be inserted where the full list of runs should be included there.

Instructions for DQM Offline shifts

For shift and tutorial subscription, please read this page.

Offline Tracker DQM shifts are normally held in CMS Center at Meyrin. The possibility to held shifts also in remote CMS centers (FNAL, DESY) could be used from time to time. They are in common between SiStrip and Pixel subdetectors.

The Tools for the Offline shifter

Run Registry

The Run Registry is the main source of general informations for an off-line shifter. Look at start/stop time of a given run, trigger type and number of triggered events, B field value, general run feature. Tools like Plot Chart (top page) may used to visualize the trend of acquired events and trigger rate. You can click on a given run to retrieve general informations from central DQM shifters on STRIP subdetector status.

Tracker Elogs

In case of problems during the run, on-line shifters should have already filled the proper Elogs:.
  • Tracker General ELOG . Check if general problems of the Tracker (DAQ, DCS, Cooling...) have occurred during the run
  • Central DQM ELOG . Check if general problems which might affect DQM histograms have occurred during the run
Check also the Whiteboard where the status of hardware, software, Tier-0 etc in a specific moment could be noted down by Tracker experts.

Central Tier-0 GUI

The GUI to be used by Offline shifter is available at: https://cmsweb.cern.ch/dqm/tier-0

In order to see something select workspace Everything and pickup the dataset you need:

/Cosmics/Commissioning09-Prompt-Reco-v3/RECO.

Example of direct link:

https://cmsweb.cern.ch/dqm/tier-0/start?workspace=Everything;dataset=/Cosmics/Commissioning09-PromptReco-v3/RECO;runnr=101045

The main space for the Tracker will be SiStrip->Layouts

The Tasks for the Offline shifter

Main purpose for the off-line DQM shifter is to re-assess the tracker status of each run after the processing at Tier0 and possibly spot reconstruction, calibration or other unexpected problems.

An off-line shifter should keep in mind which is the standard dataflow steps:

  • data are taken at P5 and streams produced
  • data are copied at Tier0 and repacked into root files (several Primary Dataset)
  • reconstruction starts at Tier0
  • histograms harvesting is launched when reco of a given run is finished.
  • harvested histograms are published on the web GUI:

https://cmsweb.cern.ch/dqm/tier-0

Which runs to investigate

ALL the harvested runs when the Tracker was in Global Run have to be analyzed by offline shifters. The shifter can find the list of already analyzed runs by checking documentation and discussing with the previous shifter and shift-leader.

Follow the Run Registry to control the list and the status of all CMS runs.

It is useful to check also the Tier0 monitoring page and check for the list of available runs. They should have a CloseOutExport status and at least one green belt has to show up in the Reconstructed column. Few minutes after the harvesting, the DQM web server should pick the histograms and show them on GUI. A run can take up to a few hours in normal processing to be published after its end.

Launch the scripts for historic DQM plots

TO BE COMPLETED

How to look at histograms and errors in the offline GUI

For both SiStrip and Pixel there are pre-defined groups of histograms which should be checked carefully. The description of the shift histograms and the actions to be taken in case of unexpected behavior are described in following links :

NOTE : Automatic Quality Tests (QTest), based on statistical parameters, are performed on a sub set of the available histograms. Alarms are raised accordingly when deviations are observed. They can be visible as a frame around each histogram when a histograms directory is opened. GREEN blinking frames indicate that everything is consistent with expectations. RED permanent frames correspond to quality test failed or not performed for some reasons.

TO DO: layout for offline GUI to be introduced check with Lassi

Book results

Each run processed at Tier0 should be checked to assess the quality of data. List of histograms defined to flag a run as good or bad for SiStrip is available at SiStrip Offline Shift Histogram Description List of histograms defined to flag a run as good of bad for Pixel is available at TO DO: include link for Pixel histograms

A report with the full list of analysed runs should be sent via email to the Offline shift leader. For each bad run a detailed investigation (see next section) should be done and a short description of the problem should be given to the Offline shift leader.

TO DO: detailed explanation about the way comments should be written

When something seems wrong

If you spot a problem in one of the plots, or you follow up on a problem reported by the online shifter, etc. report your findings in the documentation. In additon:
  • try to understand from online ELOGs if something went obviously wrong
  • if the automatic good run flag is BAD, investigate why, what cut failed, what fraction of the modules failed, etc.
  • discuss with the offline shift-leader your findings.
Make sure to communicate with following shifters via the documentation tool and via the ELOG if necessary, so not everybody has to re-discover the same problem.

Offline shift-leader tasks

The shift leader is the reference contact for off-line shifters, and is usually available at CMS Center. His/her responsabilities are:

  • keep trace of list of checked runs
  • coordinate the runs quality certification
  • check motivations for bad runs and discuss with experts
  • report the list of good/bad (+reasons for bad runs) to central DQM

Offline shift-leader daily report

TO DO: table to be created

List of bad/good runs

Temporary page

Useful Phone Numbers

SiPixel DQM On Call   162203
SiStrip DQM On Call   162389
Green Barrack phone   77528
TK phone at CMS Control room   75501
CMS Shift Leader at CMS Control Room   75200
CMS Centre (Tracker Desk)   73459
Offline Shift Leader at CMS Center   73261
Suchandra Dutta   Suchandra.Dutta@cernNOSPAMPLEASE.ch, 74253
Laura Borrello   Laura.Borrello@cernNOSPAMPLEASE.ch

Topic attachments
I Attachment History Action Size Date Who Comment
GIFgif ClusterStoNCorr_OnTrack_landauPeak_superimposed.gif r1 manage 11.7 K 2009-06-18 - 13:58 LeonardoBenucci  
PNGpng NumberOfDigiProfile__TOB__layer__6.png r1 manage 27.4 K 2009-06-18 - 13:59 LeonardoBenucci  
GIFgif NumberOfRecHitsPerTrack_mean_superimposed.gif r1 manage 12.0 K 2009-06-18 - 13:59 LeonardoBenucci  
PDFpdf SiStrip_ExpertGui.pdf r1 manage 472.9 K 2009-06-24 - 23:49 SuchandraDutta SiStrip Expert GUI
PNGpng Summary_ClusterChargeCorr__OnTrack__TEC__side__1__wheel__1.png r1 manage 22.1 K 2009-06-18 - 14:00 LeonardoBenucci  
PNGpng Summary_ClusterCharge_OffTrack_in_TOB.png r1 manage 22.2 K 2009-06-18 - 14:00 LeonardoBenucci  
PNGpng Summary_ClusterCharge__OffTrack__TOB__layer__6.png r1 manage 24.4 K 2009-06-18 - 13:59 LeonardoBenucci  
PNGpng Summary_ClusterStoNCorr_OnTrack_in_TOB.png r1 manage 23.4 K 2009-06-18 - 14:00 LeonardoBenucci  
PNGpng TkMap_channel.PNG r1 manage 56.4 K 2009-06-18 - 14:00 LeonardoBenucci  
PNGpng TkMap_channel.png.PNG r1 manage 56.4 K 2009-06-18 - 14:03 LeonardoBenucci  
PNGpng TkMap_digi.PNG r1 manage 106.1 K 2009-06-18 - 14:01 LeonardoBenucci  
PNGpng TkMap_digi.png.PNG r1 manage 106.1 K 2009-06-18 - 14:03 LeonardoBenucci  
GIFgif TotalNumberOfClusters_OnTrack_mean_superimposed.gif r1 manage 11.6 K 2009-06-18 - 14:01 LeonardoBenucci  
PNGpng cmsgui_detailedbrowsing.png r1 manage 204.0 K 2009-06-28 - 20:06 SuchandraDutta CMS GUI detailed browsing page
PNGpng cmsgui_quickcollection.png r1 manage 171.5 K 2009-06-28 - 20:07 SuchandraDutta DQM GUI quick collection for Sub-detector shifters
PNGpng cmsgui_subdetlayout.png r1 manage 116.6 K 2009-06-28 - 20:10 SuchandraDutta CMS GUI sub-det extra layouts for shifters and experts
PNGpng cmsgui_summary.png r1 manage 275.4 K 2009-06-28 - 20:06 SuchandraDutta CMS GUI summary page
PNGpng expertgui_alarmview.png r1 manage 93.9 K 2009-06-24 - 23:52 SuchandraDutta Alarm View of Tracker Expert GUI
PNGpng expertgui_detailview.png r1 manage 98.4 K 2009-06-24 - 23:52 SuchandraDutta Detailed View of Tracker Expert GUI
PNGpng expertgui_shiftview.png r1 manage 64.2 K 2009-06-24 - 23:20 SuchandraDutta Shift View in the SiStrip Expert GUI
PNGpng expertgui_summaryview.png r1 manage 128.7 K 2009-06-24 - 23:51 SuchandraDutta Summary View of Tracker Expert GUI
PNGpng expertgui_tkmap.png r1 manage 160.3 K 2009-06-24 - 23:53 SuchandraDutta TrackerMap options in the SiStrip Expert GUI
PNGpng expertgui_tkmap2.png r1 manage 143.2 K 2009-06-24 - 23:54 SuchandraDutta Tracker Map and it's features
GIFgif mean_number_of_tracks_per_event.gif r1 manage 10.1 K 2009-06-18 - 13:58 LeonardoBenucci  
PNGpng reportSummaryMap.png r1 manage 18.5 K 2009-06-18 - 13:37 LeonardoBenucci  
PNGpng runcontrol.png r1 manage 172.0 K 2009-06-25 - 20:44 SuchandraDutta Runcoltrol Steps
Edit | Attach | Watch | Print version | History: r37 < r36 < r35 < r34 < r33 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r37 - 2010-03-08 - AndrewYork
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback