DQM Online Shift Instructions

Page Under Construction

<< DQM Shift Home

General Information

Here you will find the complete DQM Online Shift Instructions:

0. Check the DQM Sign up and Requeriments

Make sure you have done the sign up procedure and you meet all the DQM Shifts requeriments.

See the DQM Shifts Sign up wiki for more information.

1. Check the CMS Commissioning Hypernews

Check the commissioning hypernews. If there is a cancellation of a DQM Online Shift, that would be announced via commissioning hypernews (hn-cms-commissioning@cernSPAMNOTNOSPAMPLEASE.ch) by either RFM (Run Field Manager) or Run Coordinators. They also have other useful daily P5 operation news, in the relevant periods. Please note, a cancellation may come-in with rather short notice. During stable beam run operation, cancellations are highly unlikely, while during LHC machine study and technical stop periods, cancellations maybe possible (especially owl shifts). But in any case, default is "ON".

See the DQM Links List for more Interesting Hypernews links.

2. Read the DQM Online Shift Instructions and the DQM Elog

Read the following before coming on shift and at the beginning of each shift:

2.1 DQM Online Shift Instructions: these DQM Online Shift instructions carefully and completely. Instructions are subject to change and may have changed since your last shift.

2.2 DQM Elog: DQM Elog entries made by the previous DQM Online shifter.

2.3 DQM Online Shift histogram description: DQM Online Shift histogram description for each subsystem.

2.4 DQM Online Shift Short-Term Instructions: DQM Online Shift Short-Term Instructions for each subsystem.

In case you have a question regarding these instructions, please create an entry in the DQM Elog with following format:

- Title: "Question about DQM Online Shift instructions" .

3. Make a DQM Shift Handover

Make a DQM Online Shift handover. Make sure to arrive 5-10 minutes early in order to get relevant information from the previous DQM Online shifter including:

  • Information about the data taking current status and what happened during the previous DQM Online Shift.
  • Information about current performance of the DQM Tools.

If anything with your tasks is not clear, please ask at that moment.

There are 3 DQM Online Shifts at P5 a day for 24 hours coverage, they take place at: | 23:00 - 7:00 | 7:00 - 15:00 | 15:00 - 23:00 |. On the first day of global running DQM Online Shifts start at 9:00.

4. Set up your DQM Online Shift environment

4.1 Login: Login into the DQM Online console.

4.2 Open the Mozilla Firefox browser: Kill the Firefox browser before starting your DQM Online Shift:

  • Open a Linux terminal.

  • Type and execute: killall firefox

  • Type and execute: ps x

This will list all the processes running on the machine. There should be no Firefox processes running. If there are, follow the procedure above one more time.

Finally, execute the Firefox browser by clicking on the desktop Firefox icon. The next tools and wiki will be available through Firefox tabs.

4.3 Open the Run Registry tool:

  • Open the Run Registry tool on one of the screens by clicking on the Run Registry tab.

  • Login to Run Registry tool using your NICE username and password.

  • You should see your name on the top right corner of the Run Registry.

If you see "UNKNOWN" for name or place, or the previous DQM Online shifter's name, you have NOT logged in successfully. In this case, "Clear private data" in Firefox Tools menu (or press Ctrl+Shift+Del) . If the problem persists try the step 4.2 again. If necessary try other online computer (left or right side).

  • If required open the Message Board in the Run Registry: click on "Tools" (top right corner) and the click "Message Board". The Run Registry provides a Message Board to report important events or action. This is the central communication tool for chats between the shift persons and possibly connected experts.

4.4 Open the DQM GUI Online tool:

    4.4.1 Open the DQM GUI Online "Summary": Open the DQM GUI Online tool on one of the screens by clicking on the DQM GUI tab. The default screen should be the DQM GUI Online Summary (go to "Workspace" -> "Summary"). Keep the "Summary" view always open, and look at it frequently in order to be able to spot sudden problems immediately.

    4.4.2 Open DQM GUI Online "Error": Open the DQM GUI Online Error Folder on one of the screens by clicking on the DQM GUI Error tab or using the DQM GUI Online go to "Workspace" -> "Shift" -> "Error" folder. Keep the "Error" folder always open, it shows plots that could indicate errors, please read how to evaluate them in Hcal and SiStrip Short-Term instructions. Inspect them for Collisions11 and Commissioning11 runs too !!!!

    4.4.3 Open the DQM GUI Online "Shift": Open the DQM GUI Online Shift Folder on one of the screens by using the DQM GUI Online go to "Workspace" -> "Shift" folder. Keep the "Shift" folder always open in order to inspect the histograms.

Note: access to the DQM GUI from outside P5 only works with appropriate certificates installed.

4.5 Open the DQM Online Shift instructions wiki:

  • Open the DQM Online Shift instructions on one of the screens by clicking on the Online Instructions tab.

4.6 Open the DQM Online Shift histogram description wiki:

  • Open the DQM Online Shift histogram description for each subsystem on one of the screens by clicking on the Online Histogram Description tab.

4.7 Open the DQM Online Shift Short-term instructions wiki:

  • Open the DQM Online Shift Short-term instructions on one of the screens by clicking on the Online Short-Term Instructions tab.

4.8 Open DQM Elog tool:

  • Open the DQM Elog on one of the screens by clicking on the Elog tab.
  • Login to Elog tool using your NICE username and password.
  • Now you can start to create your DQM Online Shift Summary (see the step 8) at the beginning of your shift and save it periodically disabling email notification.

4.9 Check Snapshot Service tool:

  • To start it (either if it is not working or if requested by the DQM experts) click on the icon (shaped as an eye) on the bottom bar on the right.

5. Interact with Shift personal

5.1 Interaction with Shift Leader, Trigger Shifter and DAQ Shifter:

  • This is a MUST! The Shift Leader depends on input from DQM so you must report observations or problems (sat on your left).
  • Also if you are in doubt about something (for example, whether to sign off a run), talk to the Shift Leader.
  • For the trigger configuration for each run, talk to the Trigger Shifter (sat on your right).
  • In case of DAQ related questions (eg. which subsystems are in the run), talk to the DAQ Shifter (sat behind you).

5.2 Interaction with other shifters (from subsystems and DQM Offline):

  • This is also VERY IMPORTANT! The subsystem shifters can also depend on you.
  • If you have doubts about a subsystem's histograms, talk to the shifter of that subsystem.
  • The subsystem shifters are in the area on the left as you enter the door of the control room.
  • For communication with other central DQM Shifts (CC-Meyrin, FNAL, DESY) use the phone, polycom or the Run Registry Message Board.

6. Certify the runs

The Run Registry tool has two views that can be toggled/switched back and forth by clicking on the tab "Runinfo" and "Runs Table" (listed at the top of the RR).

6.1 At the Beginning of each new run:

  • Turn the Run Registry view to "Runs Table".
  • Decide which runs are significant enough to create and edit. A significant run is one which has more than 10,000 events or has been running for more than 5 to 10 minutes.
  • When a new significant run appears, select the run (in "Number" column), click and select in the menu "Create Global".
  • On the edit page: add your name, the "Global Name" and the "L1 Trigger" configuration. Look for "Global Name" in the DQM Online Shift Short-Term Instructions. Ask the Trigger shifter about details of the trigger configuration.

  • For Run Group naming, read the instructions below:

    • Depending on data taking conditions, Group Name for individual runs are defined. It is automatically done in the Run Registry following the filter condition. The list is provided below with corresponding filter condition in the Run Registry. You should check the Group Name of the runs while registering it in the Run Registry and make sure correct one is provided. In case it does not match with the data taking condition inform the Shift Leader and DQM on-call shifter (see the step 7.3.2 below).

    • You should be able to modify the Run Group name while the run is OPEN. If you cannot, please contact the DQM on-call shifter (see the step 7.3.2 below).

    • Only the experts can change the Run Group name once the run is signed off (SIGNOFF). Note that the Collisions## group name does not become available until both beams have been declared stable, but this should be fine since the run needs to be significant before you try to create the new Collisions row in the Run Registry.

Group Name Short Description Filter in Run Registry
Collisions11 Colliding Beam (both the beams should be stable) StableBeam = true
Cosmics11 Cosmic data taking and no beam in the set up, at least SiStrip should be there in DAQ with HV switched ON and at least one muon system (CSC, RPC or DT) switched on and in DAQ. Cross check with Shift Leader that the L1 and HLT keys are appropriate StableBeam = false
Commissioning11 Unstable beam in the set up / no beam in the set up but insufficient conditions to mark it as Cosmics11 run (e.g. Tracker out/off, no muon system included or no cosmics HLT key) StableBeam = false
SpecialRuns11 Special Runs to be certified, instruction should be provided by Run Coordination in Elog (This kind of gorup has no precise definition and has to be used only in case Run Coordination explicitely requests it and not during usual operation) StableBeam = anything

  • In the online comments, for those subsystems not in the readout, enter reason why.
  • Press "save" to make the intermediate entry of this run available to others.

6.2 During the run:

  • Turn Run Registry view to "Runs Info" and then:

    • See in the list of runs that the last entry has a dataset created by you, with the dataset name correct.

    • For edit the run and start the certification, click on the dataset name corresponding to the significative run and select "Edit" or "View"-> "Edit". This will take you to the editing page again. Check that the entries you made are there and then move to edit the right column. Add general comments if you have some.

    • Look at the DQM GUI Online Summary to get an overview. Check the "HV and Beam Status vs. Lumi" (Info) histogram to check which subsystem is on STANDBY (i.e: High Voltage being down). You can also ask the DCS (Technical Shifter). This information is crucial for evaluating the histograms of each subsystem. If everything is green (or grey) it is likely that the run will be very good.

    • Check the DQM GUI Online "Reports" page that the update times for the ongoing run are below 3 mins.

    • For each run check that the HV information displayed in the "Info" histogram on the DQM GUI Online Summary and the information in the "LumiSec" table of the Run Registry are consistent. It happens that the Run Registry does not get by-lumisection information correctly. In such cases the column "LS" in the Runs table shows 0. If this happens:
      • Make an Elog entry (see the step 7.3.3 below) indicating the run number.
      • Ensure that the DQM on-call shifter is aware of the problem (see the step 7.3.2 below).
      • Put the information manually into the general comments section of the Run Registry. Use the following format example:

          LS 0 = CASTOR, Strips, RPC, Pixel, and DT with HV OFF.  All others with HV ON.
          LS 50 = DT HV ON
          LS 51 = RPC HV ON
          LS 80 = Pixel and Strip HV ON 
    

    • In case of doubt ask the Shift Leader or subsystem expert about the actual true status of the HV for a given subsystem.

    • Go to the DQM GUI Online "Shift" page to assess the quality of each of the subsystems participating in the run. Follow the DQM Online Shift histogram description for each subsystem looking at those first that show red areas on the top level DQM GUI Online Summary.

    • If for some reason (e.g. talk further with subsystem experts about a red situation, you would like to discuss the goodness of a run) you need help during the certification, try to share this task with the remote shift person, i.e. agree with him/her which subsystems you are going to look at and which ones he/she is going to do.

    • If you find a problematic distribution which is not described in the DQM Online Shift Short-Term Instructions:
      • Contact the Shift Leader directly and immediately and report the problem. The problem may require immediate remedy (run stop/restart and/or subsystem expert consultation).
      • Mark the subsystem as GOOD or BAD in the Run Registry, according to the decision with Shift Leader and subsystem experts. From time to time click "apply" or "save" to save the changes made. "apply" will just save the changes, while "save" will save the changes and take you back to the main Run Registry view.
      • If the same problem persists over several runs or shifts, request the DQM contact (by email) to update the DQM Online Shift Short-Term Instructions accordingly.
      • For each problem post a problem report to the Elog (see the step 7.2.2 below).
      • For insufficient or unclear instructions, feel free to contact the subsystem DQM contacts via (friendly) email.

    • If you do not find a problematic distribution: mark the subsystem as GOOD.

    • Mark the system as BAD if one of the following conditions is fulfilled:
      • The system is malfunctioning, requiring fixing.
      • The HV is on, but the data is corrupted or otherwise invalid for analysis.

    • Do NOT mark the system as BAD, just because the HV is varied during the run. Mark it bad if the HV stays OFF throughout the run entirely. HV conditions for subsystems may vary during the run, especially in case of preparation for Collisions.

    • Save the Run Registry page intermediately and proceed to looking at the next subsystem in due time (few minutes).

6.3 Soon after the end of the run:

  • If necessary complete the DQM assessment as long as there is no new run starting. Stop work on the previous run as soon as a new (significant) run is started.
  • When you are done, get in contact with the Shift Leader and have him/her double-check the information that you have entered. The Shift Leader should explicitely confirm the GOOD and BAD flags and the comments.
  • Wait for the run to stop and cross-check, that stop time and reason appear in the Run Registry. Fill in the stop reason manually if needed. "stop reason" field (central left) and possible comments.
  • If all this is done, go back to the "Runs Info" view, click on the "dataset name" of the run you were certifying and press "move to SIGN-OFF".
  • If the same problems reappear in each run, ask the expert to update the DQM Online Shift Short-Term Instructions. Make a note about this persistent situation in the DQM Online Shift Summary Elog message (see the step 8 below).

7. Deal and report problems

7.1 Report general problems and observations in a timely fashion to:

  • The P5 Shift Leader.
  • The Subsystem Experts at P5 (keep the Shift Leader up-to-date about your findings).
  • The DQM contact persons via a (friendly) email.

See the CMS Subdetectors wiki for DQM Subdetector twikis links.

7.2 Deal and report certification problems following the next steps:

    7.2.1 Register the problems in the Run Registry: Subsystems with apparent problems should be marked as BAD, unless the Shift Leader or the Subsystem Experts instructs you otherwise. Please include all known relevant information in the corresponding fields for subsystem comments in the Run Registry.

    7.2.2 Register the problems in the Elog: Provide a concise description of the problem and the runs affected. Create an entry in the DQM Elog with following format to record problems:

    - type : Problem Report

    You may refer to the Run Registry entries for all details.

7.3 Deal and report specific DQM tools problems following the next steps:

    7.3.1 Check the DQM Online Shift F.A.Q.: In case of problems check the DQM Online Shift F.A.Q. site for a possible solution.

    7.3.2 Contact the DQM On-call Shifter (DQM Expert On-call): If a problem persists more than 15 minutes, please contact to the DQM on-call shifter.

    See the DQM Contact List wiki for contact data of DQM on-call shifter and other DQM personal.

    7.3.3 Report the problems in the Elog: Create an entry in the DQM Elog with following format to record problems and incidents:

    - type : Problem Report

    Each different problem should be reported in a separate entry. Use a subject that clearly indicates the type of problem (e.g. "GUI not starting", "All histograms empty", etc.). Please include detailed information that could help to the experts in order to solve the problem.

8. Create a DQM Online Shift Summary

Create only ONE entry in the DQM Elog with following format example to record your DQM Online Shift Summary at the end of your shift.

- type : Shift Summary

********************** 
registered runs 
********************** 
129762 - night run, inherited from previous. Circulating beams 
129778- ECAL out to investigate calibration problems. Circulating beams. 
129787- ECAL back in. Half of CSC HV off. Varying beams, varying rates. 
129793- No beams. Still CSC- off.

  • Do not forget to enter the run numbers of the runs you have processed.
  • More detailed information - including information produced by remote shifts - should be entered in the Run Registry tool.
  • Enter a VERY BRIEF description stating the reason for run stop, status of beams, etc. as in the example above.
  • The DQM Online Shift Summary should NOT contain a detailed listings of runs.
  • Use the Elog for this information only in case the Run Registry stops working. Use the Message Board, NOT the ELOG, for exchange of information between shift persons.
  • If there is nothing else or new to report, just make a posting with a "nothing to report" body.
  • Please remember to active the email notification before save your final DQM Online Shift Summary.

9. Close your DQM Online Shift environment

Logout of the Run Registry and DQM Elog. Close Mozilla Firefox browser and finally logout from the Linux operating system.

10. Report at the daily Run Meeting (optional)

The DQM Online shift person may be asked to give a 1-minute DQM report on the Global Runs taken in the last 24 hours or since the last Run Meeting. Please ask the field run manager for details.


<< DQM Shift Home


Your feedback is very important in order to improve our documentation, please send your ideas and suggestions to the wiki responsible:

-- EdgarEduardoRosalesRosero - 14-Feb-2012

Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r8 - 2012-02-17 - EdgarEduardoRosalesRosero
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback