L1 Prompt Feedback Expert (formerly Offline Trigger Shifter) Guide

Quick review of the shift: QUICK START

Make sure you have gone through the section "Before your shift"

The shifter must finish the certification of the express datasets every day by 9:00 am CET for those runs recorded the day before. Remember to submit your report always before 9 am CET on the next day.

You can find more information on each following step further below

COMMON FOR ALL RUNS:

1. READ THE NEWS SECTION FOR UPDATED INSTRUCTIONS.
2. Find out what runs are to be certified, start from the oldest run that has not yet been certified. Remember to add run numbers to your report. Add also the time stamps of the first and the last run that you certified.
3. If needed, prioritize the runs as instructed.
4. Find out the DQM certification tags for the run; add the L1tcalo and L1tmu tags to your reports.
5. Check the physically meaningful LS range.
   • Check what subsystems were excluded from the DAQ, if any
   • Check if the relevant subsystems were powered on during the run
   • Form the physically meaningful LS range

COSMIC RUNS:

1. Check that the trigger keys have no mistakes
2. Check that the overall L1 rate is stable
3. Check the rate of individual triggers:
   • L1_SingleMuCosmics
   • L1_SingleMu7
   • L1_SingleEG5
   • L1_SingleJet35
   • L1_SingleJet20er3p0_NotBptxOR
   • L1_SingleMuOpen_NotBptxOR
4. Inspect L1T shift histograms in online DQM GUI
5. Decide the certification tags for L1Tmu and L1Tcalo, and apply as instructed

1. Check that the trigger keys have no mistakes
2. Check that the overall L1 rate is stable. If there are any issues, proceed as instructed below.
3. Find out the rates of individual triggers for Proton-proton / Pb-proton / Pb-Pb runs as a function of pile up.
4. Check the timing (pre/post-firing fractions) for L1 seeds.
5. Check data-emulator comparison.
6. Inspect L1T shift histograms in online DQM GUI
7. Decide the certification tags for L1Tmu and L1Tcalo, and apply as instructed

News

Always read this section before starting your shift! It contains the latest changes.

Update 28.7.2017: The instructions considering the dead time have changed: runs should not be automatically marked bad simply due to a large dead-time. However, large dead time often indicates that there has been a problem in some subsystem, which might affect the quality of data. Thus the DQM plots should be checked very carefully in these cases. Also added some information about beam scans and how they affect the trigger rates.

Update 7.7.2017: If you encounter a run which is not 100% BAD, but contains some LS that are clearly bad (e.g. peaking rate) but also more than 10 LS that could be considered GOOD, please do as follows: Mark the run (L1Tmu or L1Tcalo or both) as BAD in Offline Run Registry, but add a comment where you explain the reason, and also clearly mention the LS ranges for good and bad parts of the run in the comment. This way experts can later mark part of the run as good and part of the run as bad.

Update 5.7.2017: Reminder to all shifters: If some/all trigger rates are unstable, and you do not understand the reason (even after searching for the reason in Run Registry, Elogs etc.), please notify the the experts by submitting a JIRA ticket and wait for expert opinion about L1Tmu and L1Tcalo flags before marking these flags in Offline Run Registry.

Update 29.6.2017: Plots in L1TEMU directory of the Online DQM GUI are working again and the new instructions are in place, so please check them for all Collisions17 runs.

Update 28.6.2017: For recent cosmics runs, L1_SingleEG5 rate has been higher than nominal (even up to 800 Hz). This is a known problem and it does not need to be reported. If runs look good otherwise, L1Tcalo can still be set as GOOD. A quote from Ecal expert: "The slightly elevated rate comes from one region in the negative endcap. This causes a hotspot in the ECAL TP occupancy plot at ieta = -25, iphi ~40. This is low energy noise. There is no rate excess in EG10 or above. We have kept this region unmasked because it does not have any impact in collisions runs. It is tolerable in terms of rate for cosmics data taking. (...) If this is the only issue with the runs, I would mark them as GOOD for L1TCalo."

Update 27.6.2017: Instructions for timing studies (pre/post-firing fractions) updated.

Update 27.6.2017: For recent fills, rate vs. PU curves for L1_ETM100 and L1_HTT300er can be outside the reference curve for the first runs in the fill. This is expected and can be ignored ( see this JIRA ticket).

Update 18.5.2017: From now on, the shift reports will be presented on Mondays (the last day of shift) at L1TDPG meeting (starting usually 15:30 CERN time).

General: It is very important that you finish the certification of the express datasets every day by 9:00 am CET for those runs recorded the day before. The deadline for first report of the shift is 9:00 Wednesday morning, and the deadline for the last report of the shift is 9:00 Tuesday morning. Even if you have not certified any runs for some reason, please send a short report where you update the situation. For example, on Wednesday by 9:00 am, you should have certified all the express datasets recorded on Tuesday (if possible). This will allow run coordination to better coordinate the activities at P5, and for L1T experts to be promptly notified in case of problems. Every morning at 9:30 am L1 DOC will look at the certification in the run registry in the express stream and report to run coordination. Trigger experts will also be notified.

In case of any problems, do not hesitate to submit a JIRA ticket! Instead of writing an email with a question (or question "Should I submit a JIRA ticket about X?"), just go on and use JIRA - it is equivalent to sending an email, except that the questions, answers and discussion are better archived to help future shifters.

Introduction

Offline trigger shifts are an important part of data validation. There are two different offline shifts: L1 Prompt Feedback Expert (PFE, formerly known as Offline Shifter) is responsible for the certification of the express datasets: the shifter analyzes the performance and the data quality of the L1 trigger within ~24h from the data taking and decides the L1T certification flags for the analyzed runs. The shifter reports the findings in the Elog, and if needed, directly to experts. L1 Trigger Expert Analyst (TEA) certifies the PromptReco datasets (delivered with longer delay than the express datasets) and takes care of other important tasks such as release validation and L1NTuple production. This page contains the instructions for the PFE shift. Instructions for the TEA shift can be found here.

A PFE shift can take up to 4 hours and can be done at any time of the day. The shifter is supposed to review all the runs considered by the DQM system taken since the last run certified by the previous shifter (details follow below). Shifts can be done from anywhere using web-based tools, they only require a computer with internet connection.

The offline trigger shifts are coordinated by Pamela Klabbers (Pamela.Renee.Klabbers@cernNOSPAMPLEASE.ch). Registration for 2017 shifts is now closed.

These instructions are primarly updated by Santeri Laurila (santeri.laurila@cernNOSPAMPLEASE.ch). If you have any questions or suggestions for improvements, do not hesitate to contact him!

Workflow of the PFE shift

A shift week starts on Tuesday and ends on Monday (inclusive). Every day, the shifter certifies the express datasets appearing in the offline run registry. The shifter must finish the certification of the express datasets every day by 9:00 am CET for those runs recorded the day before.

Every day, the shifter writes an Elog report, containing a general summary of run conditions and run-by-run observations. If you think your observation is important for current data taking, e.g. a problem appeared and was not resolved, please notify L1 Detector On-Call and also write a message in the corresponding Elog (Trigger or DQM). *The first Elog should be sent before Wednesday morning 9:00 am CET, and the last report should be sent when the shift is over, before Tuesday 9:00 am CET. Please follow instructions at Reporting section to properly organize your report!

After the shift week, the shifter also gives a short report in L1TDPG meeting on Monday , the last day of the shift week (the meeting starts usually at 15:30 CERN time). This means that after the meeting on Mondy, the shifter still need to certify the last runs and provide the last Elog report.

Before your shift

1. Make sure you have a working GRID certificate.
2. Request shifter role in RunRegistry:
   • Go to RunRegistry offline version
   • Select "Workspace" -> "L1T"
   • Select "Tools" -> "My roles"
   • Click on "New role" and fill in:
     • Role: shift
     • Location: (either "pc" or your geographical location)
     • Reason: L1T offline shift
     • Due date: leave open
   • Finally click on submit. The request will be handled by one of the admins. In case of problems, create a JIRA ticket.

Starting your shift

1. When reading the instructions for the first time, the vocabulary might be useful to decipher the acronyms.
2. To be able to open DQM GUI and CMSWBM, you need to have a GRID certificate (you can get one from CERN CA)
   
3. Open your favorite text editor to write a report of the shift (an example is given at the Reporting section)
4. Open this page in a separate window when starting your shift; use the following links to open new tabs/new windows:
   • Trigger offline elog (for reporting (Subsystems -> Trigger -> Trigger Offline) and for checking what other shifters have written)
   • RunRegistry offline version (for finding out which runs to consider and making the L1T certification)
   • RunRegistry user version (for checking DCS status)
   • CMSWBM Run Summary (for checking L1T rates)
   • List of valid trigger keys (2017) (for checking if the keys in the run are valid; the old list of 2016 keys is here)
   • CMSWBM Fill Report page (for checking the rates as a function of pile-up for collision runs)
   • DQM GUI online (for online DQM histograms)
   • L1TEMU DQM histogram instructions page (for checking what are the criteria of L1T DQM data vs. emulator comparison)
   • L1T DQM histogram instructions page (for checking what are the criteria of L1T DQM)
   • DQM short term instructions (for checking the known abnormalities in DQM plots)
   • Known issue with RunRegistry and DQM GUI: if page gives an error message or does not show all information, reload the page (sometimes necessary to reload twice). With Run Registry, you can also try logging out and then logging in again.
5. In addition, check also Elog for information relevant to put into your shift summary (trigger problems, DQM problems, general CMS problems, special beam/running condition, etc...). You can search run information from all Elogs using the "Find" option in Elog, writing the run number in "Text" field and choosing "Search all logbooks" option. Especially the trigger and DAQ Elogs can be helpful.For example, the following Elogs categories can be useful for understanding the runs:
   • General -> Shift leader
   • Subsystems -> Event Display and DQM
   • Subsystems -> Trigger -> Trigger
   • Subsystems -> DAQ -> DAQ
6. Find out the run numbers of the physics runs taken after the previous shift.
   • To see what are the latest runs certified by the online DQM, go to RunRegistry offline version and L1T workspace. Look at the column "Dataset State". Any runs, which are in state "OPEN" are waiting for certification by the offline trigger shifter. The runs already considered by previous shifters should be marked as "SIGNOFF" or "COMPLETED".
   • Look at the report of the previous shifter in the Trigger offline Elog and find out (by default at the beginning of the report) what are the timestamps and run numbers of the latest Collision/Cosmics/HeavyIon run that were considered. Your task is to consider the runs that are newer than those considered by the previous shifter.
   • For collision runs, certify the Express datasets only. The TEA shifter takes care of the corresponding PromptReco datasets. For cosmics runs (and ONLY for cosmics runs), add the L1Tcalo and L1Tmu certification flags and your comments also to the corresponding PromptReco datasets when they appear in the run registry.
   • Start from the oldest run first and work your way to the top of the list (i.e. to the newest) one run at a time. Make sure you include the run numbers and timestamps of the latest Collision/Cosmics/HeavyIon run you considered (separately for each; if you did not consider runs of certain type, copy the run numbers and timestamps from the previous shifter's Elog report)!
   • Prioritize the runs as follows:
     • Collision / Heavy ion: highest priority (because your report is needed to choose the good runs for analysis!)
     • Cosmics: medium priority (consider only, if no Collision / Heavy ion runs are available)
     • Commissioning: consider only if specifically asked
   • Don't be afraid to skip tens of unconsidered Cosmics runs if there are Collision / Heavy ion runs to be considered!
   • It can happen sometimes that a cosmics run is first classified as commissioning, and even after this is corrected, the ”commissioning” version is still visible in RR. In this case, you can just ignore the commissioning version and certify the cosmics version as usual.
   • Take note of each run that you consider for the shift
7. Find out the DQM certification and if some subsystems were excluded from DAQ or on in standby
   • DQM certification
     • In the RunRegistry user version, choose from table dropdown menu “Datasets” and click apply. Scroll down the list or filter by run number to find the desired run.
       Show tutorial figure Hide tutorial figure
       
     • Each run that you consider should have two entries for the run. You can ignore the one where Dataset name is /Global/Online/ALL only look at the other one (PromptReco). There can be a delay before the PromptReco dataset appears - during this time only /Global/Online/ALL version is visible. While waiting for PromptReco to appear, you can start the certification process using /Global/Online/ALL dataset. Just remember to check afterwards that it is in agreement with the information in PromptReco dataset.
     • By moving the mouse cursor on top of the coloured box of L1T of the desired run, any comment written by the online DQM shifter will become visible.
     • Take note of the L1T flag (to be included in your shift report) and any comments that could be useful for you later in the certification process.
   • Were some subsystems excluded from DAQ in run?
     • In the RunRegistry user version, choose from table dropdown menu “Datasets” and click apply. Scroll down the list or filter by run number to find the desired run.
       Show tutorial figure Hide tutorial figure
       
     • Take note also of the subdetectors (columns Csc ... Strip) that are yellow (STANDBY) or gray (EXCLUDED, i.e. not included in DAQ).
       Show tutorial figure Hide tutorial figure
       
     • If you are considering a Collisions / Heavy ion and Pixels and Strip are yellow (STANDBY) or gray (EXCLUDED), the corresponding trigger rates should be zero.
     • If you are considering a Cosmics run and Strip and/or all of the muon systems (DT/CSC/RPC) are yellow (STANDBY) or gray (EXCLUDED), the corresponding trigger rates should be zero. If part of the muon systems are EXCLUDED or STANDBY, the muon trigger rates are expected to be lower than the normal values.
     • If a subdetector is marked as BAD, it is still important to know if the L1 trigger was ok or not.
   • Were some subdetectors not powered during some part of the run?
     • In the RunRegistry user version, choose from table dropdown menu “Run Lumi Sections”. (If the page is not working, you can also use "Dataset Lumi Sections" page.)
       Show tutorial figure Hide tutorial figure
       
     • Filter by the column Run number and make sure that all of the lumisection ranges are visible (i.e. smallest "Section From" entry is 1) Show tutorial figure Hide tutorial figure
       1. Click on column "Run Number"
       2. Type in desired run number
       3. Press enter or click on sort
     • The green symbols mean that the detector control system was ok for the mentioned lumi section range. Be warned, that a green symbol can appear also for systems, that were excluded from DAQ, so these tho things must be checked separately!
   • Take note of the lumi section range that is physically meaningful to check for the run
     • For Collisions / Heavy ion runs:
       • Consider the lumi section range, where beam is stable (Beam1 stable and Beam2 stable are green) and tracker is on (Bpix, Fpix, Tecm, Tecp, Tibtid, and Tob are green) Show example Hide example
         • In the example, stable beams are declared for LS 1-1277 and Pixel and Strip are both active during LS 1-1269 (need both pixels and strips to do seeding and tracking). I.e. a physically meaningful LS range to examine in the run is LS 1-1269.
     • For Cosmics runs:
       • Consider the lumi section range, where at least one of the muon systems (CSC, DT, RPC) is completely green, where the silicon strips are on (Tecm, Tecp,Tibtid, and Tob are green) Show example Hide example
         • In the example, at least one muon subsystem (CSC/DT/RPC) is active during LS 1-322 and Strip is active during LS 1-308 (Pixel is not needed for seeding/tracking in Cosmics runs). I.e. a physically meaningful LS range to examine in the run is LS 1-308.
     • Physically meaningful lumi section range is defined as the range where the above conditions hold, the relevant subdetectors are included in DAQ and “L1A Physics” rate is >0. E.g. for a cosmic run, we require that the silicon strips and at least one muon system are on, that these systems are also included in DAQ, and that L1A Physics rate is non-zero. Please indicate the physically meaningful LS range for each run in your Elog report.
     • Datasets from runs with silicon strips and at least one muon system should be labeled as "Cosmics" (instead of "Commissioning") in Run Registry. However, in rare cases a run with e.g. no muon systems, they can still get labelled as cosmics. If this is the case, e.g. there are no physically meaningful lumisections in the run, mark both L1Tmu and L1Tcalo as BAD in run registry (see instructions below) and mention this in your Elog report.

Checks related to L1 trigger information for each run

1. Open the L1Summary page for the run:
   • From the CMSWBM page click on the RunSummary link.
     Show tutorial figure Hide tutorial figure
     
   • In the Run Summary page, fill in the run number (or range of run numbers) and click SubmitQuery.
     Show tutorial figure Hide tutorial figure
     
   • From the run list, click on the desired blue run number (hint: it saves time, if you open the run in a new tab in the browser).
     Show tutorial figure Hide tutorial figure
     
   • For collision runs, take a note which fill the run belongs to. This will be needed in later while inspecting the trigger rates and timing.
   • In the following page, click on the link next to caption “L1 key”.
     Show tutorial figure Hide tutorial figure
     
2. Check that the trigger keys have no mistakes:
   • In the L1Summary page, look at the L1 Summary key. It should be of the same type as the run. If it is not, the run should be marked as BAD (for example: if a collisions run has L1SummaryKey ...cosmics..., the run should be marked BAD).
     Show tutorial figure Hide tutorial figure
     
   • In the L1Summary page, check GTKey and the keys on the lines CalorimeterTriggerKeys and MuonTriggerKeys and compare them against the list of valid keys in the page of valid trigger keys. Report any missing or invalid keys in your report (do not copy all trigger keys to your report!).
     Show tutorial figure Hide tutorial figure
     
   • The page listing the valid trigger keys is updated quite often (see time stamp at top of page to see when was the last update). If you see that the valid trigger key page has been updated, please recheck if the list of trigger keys you have not found has appeared in the list of vaild trigger keys.
3. Check that the overall L1 rate is stable:
   • In the L1Summary page, find label “L1A Physics” and click on the adjacent blue number showing the trigger frequency.
     Show tutorial figure Hide tutorial figure
     
   • Examine the “L1A Physics” rate as a function of lumisections in the LS range that is physically reasonable (flat top and tracking included in DAQ and active for collisions/heavy ions; at least one muon subsystems and strips included in DAQ and active). The number of saved LS might be smaller in CMSWBM than indicated by RunRegistry (this is a known feature, just consider those LS that are available in CMSWBM and that are physically reasonable).
   • Describe in the report the behavior of the “L1A Physics” rate as a function of lumi sections. Report discontinuities/spikes that differ by >15 % from the general behavior. (Occasional dips/peaks (of ~1-3 lumi sections) are acceptable, but all larger unstabilities should be investigated. ) Report, if possible, also the reason for discontinuities/spikes/anomalous behavior. “L1A Physics" rate is a sum of individual trigger rates, so you should try to trace which individual trigger(s) is the source of anomalous behavior.
   • If there are sudden jumps in the L1A Physics rate, check first if they are explained by prescale changes (see below for instructions) or sub-systems going off/out (information you should already have seen in Run Registry User version).
   • If you need to check the reason for unstable behavour or strange rate, for normal Collision runs, you can check e.g. the following triggers (NB! Reporting rates of each trigger is not required anymore for collision runs!):
     • L1_ZeroBias (this might be the only active trigger for first Collision17 runs in May 2017!)
     • L1_SingleMu22
     • L1_SingleJet180
     • L1_ETM100 (Missing transverse energy, calculated from jet momenta)
     • L1_HTT300er (Total tranverse energy, calculated from jet momenta)
     • L1_SingleEG40
     • L1_SingleIsoEG34
     • L1_DoubleIsoTau32er2p1 (can be prescaled to zero at very high lumi) -->
   • For Cosmics runs, check through the list of individual triggers given below.
   • For low-pileup Collision runs (such as TotemRun), you can look at the following triggers:
     • L1_SingleMuOpen
     • L1_SingleJet20_BptxAnd
     • L1_SingleEG5
     • L1_TOTEM_0
     • If the mentioned trigger path combination does not exist, take the lowest unprescaled path of the same type.
   • For Pb-proton / Pb-Pb runs, report the behavior of the following triggers:
     • L1_SingleMu5_BptxAND
     • L1_SingleJet36_BptxAND
     • L1_SingleEG14_BptxAND
   • Example of Collision run with start and end of collisions (7e33 menu, prescale column index 2, peak luminosity ~6e33) Show example Hide example
     • • The overall trigger rate decreases exponentially as collisions take place (unless compensated by changing prescale values for the triggers).
       • There is always a short delay from declaring stable beams and turning on the power to the tracker (the powering on results into increased dead time visible as a drop in the rate when DAQ is obtaining sync).
       • In a longer run, it is normal to have small (up to -15 %) dips caused by short periods of increased dead time (usually some subsystem is busy or there are sync issues in the DAQ)
       • Note that one can toggle the x-axis to show either time or LS (the table below the figure can be used to find the exact LS, if necessary
     • Example of Collision run with discontinuity from prescale index change (7e33 menu, prescale column index 2->4) Show example Hide example
       • The discontinuity at ~7:40 (LS 521) is caused by the change of prescale column index, i.e. some (but not all) triggers are allowed to have increased rate by reducing the L1T precale. This takes place usually when the L1APhysics rate has dropped sufficiently high. Sometimes such discontinuity takes place at the beginning of the run, when a too high overall rate is obtained and one has been forced to reduce the rate.
       • Note that one can toggle the x-axis to show either time or LS (the table below the figure can be used to find the exact LS, if necessary)
4. COSMIC RUNS ONLY: Check the rate of individual triggers:
   • Find the desired trigger by scrolling the list. If the name of the trigger appears in bold letters, then the trigger is active. If the desired trigger is not active, do not check it's rate, but simply mention it is not active.
     Show tutorial figure Hide tutorial figure
     
   • For Cosmics runs, report the behavior of the following triggers:
     • L1_SingleMuCosmics (average rate around 110-130 Hz): report the average rate (for physically meaningful LS)
     • L1_SingleMu7 (average rate around 20-25 Hz): report the average rate (for physically meaningful LS)
     • L1_SingleEG5 (average rate between 10-270 Hz): report the average rate (for physically meaningful LS)
     • L1_SingleJet35 (average rate between 0-100 Hz): report the average rate (for physically meaningful LS)
     • L1_SingleJet20er3p0_NotBptxOR: report if the trigger is active (bolded in L1 Summary page) and whether the rate is larger than 0
     • L1_SingleMuOpen_NotBptxOR: report if the trigger is active (bolded in L1 Summary page) and whether the rate is larger than 0
       • To see the rates for cosmics, click on the blue number in the “Post-DT Rate, Hz” column (hint: it saves time if you open the rate of the interesting triggers to a separate tab in the browser)
         Show tutorial figure Hide tutorial figure
         
       • If the trigger is not available/active, use the lowest-threshold unprescaled trigger of the same type (for unprescaled triggers, typically the number in InitialPrescale column of L1 Summary page is 1, more detailed instructions for checking the prescales are below). Notify the the experts of this kind of trigger menu changes by submitting a JIRA ticket.
       • Note which of the individual triggers demonstrate the similar spikes or discontinuities as those observed in L1APhysics rate or if all of them change at the same time. Dips are often caused by subsystems coming online or going standby or sync problems in DAQ.
   • Qualty criteria for trigger rates
     • For each run, you have to label the muon triggers (L1Tmu) and calorimetric triggers (L1Tcalo) as GOOD or BAD. These flags are reported in Elog entry and marked in Offline Run Registry (see instructions below).
     • L1Tmu/!L1Tcalo flag can be marked as GOOD if 1) the muon/calo trigger rates are stable for physically meaningful LS, 2) the rates do not exceed the given upper values by more than 10% 3) If the rates are smaller than the above values, the difference is understood (e.g. part of the sub-systems are off) and 4) L1T and L1TEMU DQM plots are satisfactory (see below).
     • Also if you see abnormally high rates for many runs, please notify the the experts of such changes by submitting a JIRA ticket.
     • If the above criteria are not met, first try to understand the changes/abnormalities yourself (search Elogs etc.) If you are not sure about which flag is the correct one, leave the run in OPEN state in the run registry (do not sign off), notify the experts by submitting a JIRA ticket and wait for experts' instructions for finalization of the certification.
   • Tips for understanding trigger rates
     • Always make sure that you are looking at post-deadtime rates.
     • Also for individual triggers, occasional dips/peaks (of 1-3 LS) are acceptable but all larger abnormal patterns in rates should be investigated and reported.
     • If all individual rates have the same dip, it is probably due to a DAQ problem or an LHC beam scan. In these cases, the L1T trigger is still considered as performing normally. Thus it is important to try to track down the reason of strange behaviour of trigger rates.
     • If a peak in some rates corresponds to a dip in other rates, it is probably due to a trigger problem: one trigger is sending at very large rate, blocking others from working normally.
     • You can check for beam scans, such as luminosity and emittance scans, by searching through Elogs (e.g. Shift leader Elog and BRIL Elog). Beam scans correspond to sudden changes in instantaneous luminosity and pileup, and thus affect also all trigger rates in the same way. You can check instantaneous luminosity and pileup for a given fill on the Fill Report page: when you scroll down the page, you can see the plots "CMS: Fill XXXX Instantaneous Luminosity" and "CMS: Fill XXXX Pileup monitor" for a given fill. If there is a peak/drop in these plots corresponding to a peak/drop in trigger rates, we know that L1T is performing fine and the rate changes are simply due to changes in beam conditions.
     • If a rate looks abnormal, check also the dead time by ticking the "Pre Dead-time rates" on the page that shows the rate plot. The relative difference between the two rates is normallyless than 10%. A high dead-time indicates a problem with some subsystem, but does not neccessarily mean that the quality of data would be bad. If you observe high dead-times, please mention also the pre dead-time rate in your Elog report.
     • L1Summary page shows also a number called "Overall DeadTime: Total". Also this number should be less than 5% (for physically meaningful LS range). For >5%, please report the dead-time on Elog and your summary slides. NB! This number should be calculated only taking into account the physically meaningful lumi sections (with L1A Physics rate > 0). You can enter the LS range in L1Summary page and hit enter, and the numbers get updated.
     • If a calo trigger rate, such as SingleJET* or SingleEG* is abnormally large, you can check from DQM GUI online if the reason for this is a hot tower in ECAL or HCAL. Select the correct run, go to L1T workspace and look plots "Calo Layer1 ECAL Input Occupancy" and "Calo Layer1 HCAL Input Occupancy". Hot towers shold be visible as excess occupancy (green/yellow/red). If you see hot towers, they probably explain the large rate and you should mark L1Tcalo flags as BAD and mention the hot tower(s) in your Elog report and summary slides.
   • Tips spesific for cosmic runs
     • For cosmic runs, a rate can be considered stable if the variation is within ~20%
     • If the rate is very small(<= 1 Hz), even larger variation can be considered normal (due to noise in detectors).
     • Muon trigger rates are lower than nominal when part of the muon systems are excluded or when all FEDs corresponding to CSC, DT and RPC are not active. Active/excluded FEDs are listed on Run Summary page.
     • DT is the dominant muon system for cosmic muons, as they are coming from the sky.
   • Checking for prescaled/unprescaled triggers and prescale changes:
     • To check for prescale indeces and changes, go to L1 Run Summary page, enter the run number and after the page is showing run information, click "Prescale Changes". A new page opens ( example for run 297411). It shows the prescale changes as a function of time and lumi sections. "NewPrescaleIndex" shows the index for the prescale that is used after given LS.
     • To check prescales corresponding to different prescale indeces, go back to L1 Run Summary page, and from there, go to "Prescale Sets" page ( example for run 297411). This page shows which trigger paths are prescaled for different prescale indeces. If the number in table is 1, the trigger is unprescaled. All other numbers mean that the trigger is prescaled, e.g. number 160 means that only one event out of 160 passes the trigger.
       Show tutorial figure Hide tutorial figure
       
5. COLLISION RUNS ONLY: Find out the rates of individual triggers for Proton-proton / Pb-proton / Pb-Pb runs as a function of pile up:
   • In order to check the rates of the individual triggers, one has to first know which fill the run belongs to. Check the instruction "Open the L1Summary page for the run" to get the number of fill for the run you are inspecting.
   • Go to page CMSWBM Fill Report page
   • Enter the fill where the run belongs in "Specific fill" field on top left and click "GO". <!-- Show example Hide example <img alt="" src="/twiki/pub/Sandbox/TestTopic11111131/RateVsPileUp.png" /> -->
   • Click the "HLT rates vs pile-up" link below the "GO" button.
   • A new pages opens, where you can see links to "Stream Rates", "Dataset Rates" and so on. Click the link called "L1 Trigger Rates". <!-- Show example Hide example <img alt="" src="/twiki/pub/Sandbox/TestTopic11111131/RateVsPileUp2.png" /> -->
   • In this page you will see rates as a function of pile-up. <!-- Show example Hide example <img alt="" src="/twiki/pub/Sandbox/TestTopic11111131/L1rates.png" /> -->
   • Start inspecting the rates for the following triggers (you can search for plots with the text search of your web browser):
     • L1_SingleMu22
     • L1_SingleJet180
     • L1_ETM100
     • L1_HTT300er
     • L1_SingleEG40
     • L1_SingleIsoEG34
     • L1_DoubleIsoTau32er2p1
   • The plots describe pre-dead time unprescaled rates of the single L1 seeds used in the data taking as a function of pile up. The rates are divided by the number of colliding bunches such that differing number of bunches in the LHC does not affect the rates.
     • The plots include the data from all the runs collected so far associated to a specific LHC fill.
     • The different color represents the data coming from the different runs of a LHC fill.
     • The red line is a rate vs. pile up fit created using a reference fill. The dashed red area following the fit is the uncertainty related to the reference fit.
     • Check that rates of the run you are inspecting are within the error bands of the reference fit.
     • Read the pile-up axis from right to left. At the beginning of the run pile-up is at its maximum. As the length of the fill increases and the intensity of the colliding bunches decreases since the protons collide. This means that also the pile-up will decrease. Thus, as the run number arises in the same fill, you must see smaller and smaller pile-up and decreasing rate.
     • Make sure that for each run, the rate and pile-up decrease compared to the previous run.
     • Check that for each run, the rates of individual L1 seeds shown above are within the uncertainties of reference fit. If this is not the case or if the reference fit is missing, notify the L1 experts by submitting a JIRA ticket and mention this in your daily offline shift report.
     • For each trigger seed specified above, report the average pile-up and average rate in your daily report.
     • If the rate of an individual trigger is not within the errors of the reference fit, or their rate increase (and not decrease) check luminosity from the fill report for the give fill. Show example Hide example
     • If the rate change can not be explained with the information available in the fill report page, notify the experts by submitting a JIRA ticket and mention this in your daily offline shift report.
6. COLLISION RUNS ONLY: Check the timing (pre/post-firing fractions) for L1 seeds:
   • It is enough to do this timing study for the two longest runs (larges number of physically meaningful lumisections) for each fill. For these runs, you should mark the calculated pre/post-firing fractions to your Elog report. The instructions for calculating these numbers are given here.
   • First check the global BX for the isolated bunch on CMSWBM Bunch fill page. Enter the fill number and click "Submit". The page shows the filling scheme, i.e. the positions of the bunches as a function of global BX, for that fill. The bunches are shown as blue pixels in the summary table and as rows with bold black text in the list below. An isolated bunch should correspond to a single blue pixel in the middle of a white row of pixels, and a single bolded line with non-zero number in the middle of lines filled with zeros. For example, for fill 5864, the only isolated bunch is at BX=1539 (you can check yourself). Some fills contain a large numeber of isolated bunches (you can choose any of them) and some may not have isolated bunches at all (so you have to skip this study).
   • For some fills, it is possible that there are no isolated bunches. Then you should mention this in your report, but you cannot proceed with the following steps.
   • Go to DQM GUI online. Choose "L1T" workspace by clicking first "Workspace" and then "L1T". Show example Hide example
   • The timing plot that we are interested in can be foud in directory "L1TStage2uGT" (inside "L1T"). Click "(Top"), then "L1T" and finally "L1TStage2uGT" directory.
   • Choose the plot "algoBits_before_bxmask_bx_global" (#7 in the list) by clicking it once. Make the numbers stored in the plot readable by clicking "Customise", writing "text" to Draw options field and hitting enter key. You can make the plot readable by limiting the range of Y axis. To start with, you can set y-axis range from 51 to 53. You should look at the isolated bunch around bx=47: to do this, set X axis range from 42 to 52. Show example (NB! The example actually shows a different plot, algoBits_after_prescale_bx_global (#2), but otherwise it is still correct!) Close
   • Your task is to calculate pre/post-firing fractions for the isolated bunch for each trigger bit mentioned in the following list. For this, we consider five bins: the nominal BX (where the isolated bunch should be) and bins -2, -1, +1 and +2 w.r.t the nominal. The pre-firing fraction is defined as the number of events in bins at 1 or 2 BX before the nominal BX, devided by the total number of events in the five bins. The post-firing fraction is defined as the number of events in bins at 1 or 2 BX after the nominal bx, again devided by the total number of events in the five bins. (Don't worry, an example will follow.)
   • The numbers in the vertical axis are trigger bits: each number corresponds to a L1 trigger seed. The bits corresponding to different triggers can be seen for each run in "L1Summary Algorithm Triggers " list of L1 Run Summary page. (Go to Run Summary page, enter the run number and click the "L1_KEY".) For collision runs, you should report the pre/post-firing fractions for the following triggers (the corresponding trigger bits are also given, but you should double-check these):
     • L1_SingleMu22 (10)
     • L1_SingleEG40 (52)
     • L1_SingleJet180 (130)
   • For example, if the plot looks like the one in example, with isolated bunch at nominal BX=47, we can read the L1_SingleIsoEG34 events from the first row that is visible (bit 49). The total number of events in the five bins from 45 to 49 is 1+4+64+8+0=77, so the pre-firing fraction is (1+4)/77=6.4% and the post-firing fraction is 8/77=10.4%. Show example plot Hide example plot
   • If the pre/post-firing fraction is 0 for a given trigger, report also this. If all bins are empty (including the nominal BX), report also this.
   • These pre/post-firing fractions should not affect your judgement about the quality of the runs (L1Tmu and L1Tcalo flags). It is enough to report them in Elog. (However, when you check the other DQM plots (as described later in these instructions), there you will also find plots related to timing: they should have a clear peak at BX=0, and if they don't you should mark the run as bad!)
   • We recognize that this step is a bit inconvenient and time-consuming: we are working to automatize it. In the meantime, your contribution for timing studies is important!
7. Check data-emulator comparison:
   • In DQM GUI online, click "Workspace" and select the "Shift" workspace from the "Summaries" list on the left. Open the "L1TEMU" directory by clicking it. Then choose the correct run by clicking on "Run #" and then click on “Online runs”. Type the run number in the search field and after finding the desired run number, click the link. The run number should appear now on the black header. Follow the instructions given in L1TEMU DQM histogram instructions page.
     Show tutorial figure Hide tutorial figure
     
   • If the DQM plots for are missing for a run, you can skip this step. Please mention the runs with missing plots in your Elog report.
8. Inspect L1T shift histograms in online DQM GUI
   • In the online DQM GUI, select the "Shift" workspace and the desired run as above, but now open "L1T" directory. Follow the instructions given in L1T DQM histogram instructions page. Take also into account the short term instructions for online DQM.
   • Pay special attention to BX plots, that should have a clear peak at BX=0. If the don't, you should mark the run as BAD.
   • Again, if the DQM plots are missing for a run, you can skip this step, just mention this in your Elog report.

Submitting a JIRA Ticket

• First check from this page that there there does not exist a recent JIRA ticket about the same issue.
• Go to JIRA page and click "Create" button on the top corner of the page Show example Hide example
• You will now see a new window with several fieds. Many of the fields are filled automatically, but please check the following fields: Show example Hide example
  • Project: CMS Level-1 Trigger DPG (CMSLITDPG)
  • Issue type: On Call (or something else, e.g. "Bug" if that describes the problem better)
  • Component/s: L1TShiftReports
• Write the title of your finding/question/problem in the "Summary" field
• Describe the finding/question/problem in the "Description" field

Certification of L1T in RunRegistry

• A quick checklist to decide whether a run is GOOD or BAD
  • Is L1A Physics rate stable? If not, are the changes in rate understood (e.g. subsystems going to standby or excluded from DAQ)?
  • For cosmic runs: Are the rates of individual triggers within reference rates? If not, is the difference understood (e.g. part of subsystems excluded)?
  • Collisions: is the rate vs. pile-up plot normal for the fill corresponding to this run?
  • Collisions: do the data-emulation comparison plots (L1TEMU DQM) look normal?
  • Do the L1T DQM plots look normal?

• Editing the evaluation of the certification of L1Tmu (muon related L1T) and L1Tcalo (calorimeter related L1T) into RunRegistry under the L1T Workspace
  • In the RunRegistry offline version go to the L1T Workspace. Show tutorial figure Hide tutorial figure To change Workspace, click on the Workspace button on the top row and choosing L1T from the menu that opens up.
  • The active workspace is indicated in parenthesis after the words "CMS DQM Run Registry". It should read now L1T in the parenthesis.
  • Your name should appear left to the Workspace button with the word SHIFT in parenthesis. If instead of SHIFT there is null in the parenthesis, you do not have the rights to edit RunRegistry (follow instructions at "before your shift" and allow some time for the confirming of your request; if you did these steps already and the problem persists submit a JIRA ticket with a description of the problem.
  • For each run that you consider, do the following steps:
    • Navigate to the desired run and select the desired run from the list of runs by clicking on it. The dataset state for the run should be OPEN.
    • If you need to change the L1Tmu and L1Tcalo flags to something else than what you see on the screen, click on the Manage button and select Edit. A page opens up where you can edit the flags. If you change the flag, remember to add a short description of the reason why you change the flag. Remember to press the Save button at the bottom to save your editings. Finally, check in the list of runs that your editings have indeed been saved (if necessary, hover your mouse on top of a flag to see the comments for that flag). If necessary, redo the editing and saving (you may sometimes need several attempts if RunRegistry or network is under load). Show tutorial figure Hide tutorial figure
    • If you encounter a run which is not 100% BAD, but contains some LS that are clearly bad (e.g. peaking rate) but also more than 10 LS that could be considered GOOD, please do as follows: Mark the run (L1Tmu or L1Tcalo or both) as BAD in Offline Run Registry, but add a comment where you explain the reason, and also clearly mention the LS ranges for good and bad parts of the run in the comment. This way experts can later mark part of the run as good and part of the run as bad.
    • Sign off the run (if there are no open questions about L1Tmu and L1Tcalo flags): click on the Move button and choose the option "to SIGNOFF".
    • If you mark the run as "BAD" in offline RunRegistry, make a comment about the reason why you are doing that.

Reporting

Once every day, you post an Elog report summarizing the status of certification. Moreover, at the end of your shift week, you report on your week by giving a presentation in L1T DPG meeting on Monday (the last day of your shift):

1. Post your daily report in the trigger offline Elog. The report should start with:

• • Date and time of the latest considered collisions/heavy ions run (latest in terms of the creation of the offline dataset in RunRegistry offline version); if you did not analyse such runs during the shift, copy the date and time from the report of the previous shift
  • Date and time of the latest considered cosmics run (latest in terms of the creation of the offline dataset in RunRegistry offline version); if you did not analyse such runs during the shift, copy the date and time from the report of the previous shift
  • A general run summary of findings during the shift including relevant information from other elogs
  • A summary table of pre/post-firing fractions for different fills. (At least mention the fills+triggers with fractions much larger than normally.)
  • A summary list of runs considered with run number, type of run, certification (GOOD or BAD) for L1T at online and for L1Tcalo, L1Tmu at offline (the behavior of muon triggers should go to L1Tmu decision and the behavior of calorimeter triggers should go to L1Tcalo decision).
  • When writing the last report of shift week, mention clearly in the beginning of the report if there are any runs that you were not able to certify (or sign off) because they stayed in the Waiting list of Offline Run Registry. Then the next shifter knows to take care of these runs too (once they are moved from Waiting list to Datasets list in Offline Run Registry).

The deadline for first report of the shift is 9:00 Wednesday morning, and the deadline for the last report of the shift is 9:00 Tuesday morning. Even if you have not certified any runs for some reason, please send a short report where you update the situation.

Example of a report (this is from last year, check also latest reports in Elog for recent examples):

Show example Hide example

First run start time stamp, Collisions16: 2016.10.24 19:32:11      
Last run stop time stamp, Collisions16:   2016.10.26 12:24:37 

First run start time stamp, Cosmics16:    2016.10.19 01:15:54      
Last run stop time stamp, Cosmics16:      2016.10.12 08:57:19


Run            Group           L1T Online   L1Tmu Offline   L1Tcalo offline   Comments
-------------------------------------------------------------------------------------------------
283934    Collisions16,3.8T      GOOD           GOOD             GOOD              
284935    Collisions16,3.8T      BAD            WAITING          WAITING       (1),(2)
283471    Cosmics16, 3.8T        GOOD           GOOD             GOOD             
282934    Cosmics16, 3.8T        GOOD           GOOD             GOOD             

(1) Rates for triggers L1_ETM90, L1_HTT300, L1_SingleEG40, L1_DoubleIsoTau28er in the rate vs PU plots NOT within the reference rates.
(2) In the online DQM GUI, the plot "30 - EMTF Chamber Occupancy" has white holes. The run has been marked as BAD in offline run registry and the experts have been notified for both of these issues. 
Waiting for more instructions before signing of the run.
-------------------------------------------------------------------------------------------------

Missing keys (used for Collisions16 runs but not listed as valid):
- UGT_BASE_KEY/v64, Collisions_EGfix/v13

Run-by-Run Comments:

-------------------------------------------------------------------------------------------------------------------------------

## Run 283934 ---- L1Tmu GOOD, L1Tcalo GOOD

No detectors excluded or in standby
Physically meaningful LS range: 1-1297
During the run in a few tens of lumisections, HCAL forward was out of DAQ. 

L1 Rate: 74.0 kHz (few dips that mostly correlate with HF going out of DAQ)
Average pile-up for run : 37

L1_SingleMu22 : 2.3 Hz
L1_SingleJet180 : 0.9 Hz
L1_ETM90 : 3.0 Hz
L1_HTT300 : 1.7 Hz
L1_SingleEG40 : 3.5 Hz
L1_SingleIsoEG34 : 4.0 Hz
L1_DoubleIsoTau28er : 4.2 Hz

-------------------------------------------------------------------------------------------------------------------------------

## Run 284035 ---- L1Tmu GOOD, L1Tcalo GOOD

No detectors excluded or in standby
Physically meaningful LS range: 1-369

L1 Rate: 73.4 kHz 
Average pile-up for run : 37 

L1_SingleMu22 : 2.3 Hz
L1_SingleJet180 : 0.9 Hz
L1_ETM90 : 3.0 Hz
L1_HTT300 : 1.7 Hz
L1_SingleEG40 : 3.5 Hz
L1_SingleIsoEG34 : 4.0 Hz
L1_DoubleIsoTau28er : 4.2 Hz

-------------------------------------------------------------------------------------------------------------------------------

## Cosmic run 283471 ---- L1Tmu GOOD, L1Tcalo GOOD
   
No detectors excluded or in standby
Physically meaningful LS range: 1-83
L1 Rate: 3.1 kHz 

L1_SingleMuCosmics : 113 Hz 
L1_SingleMu7 : 20 Hz
L1_SingleEG5 : 254 Hz
L1_SingleJet35 : 75 Hz
L1_SingleJetC20_NotBptxOR : 2.7 kHz
L1_SingleMuOpen_NotBptxOR : 65 Hz

-------------------------------------------------------------------------------------------------------------------------------

## Cosmic run 282934 ---- L1Tmu GOOD, L1Tcalo GOOD 

No detectors excluded or in standby
Physically meaningful LS range: 2-62
L1 Rate: 391.82 Hz 

L1_SingleMuCosmics : 122 Hz 
L1_SingleMu7 : 24 Hz
L1_SingleEG5 : 270 Hz
L1_SingleJet35 : 0.5 Hz
L1_SingleJetC20_NotBptxOR : 67 Hz   
L1_SingleMuOpen_NotBptxOR : 76 Hz   

-------------------------------------------------------------------------------------------------------------------------------

Please submit a JIRA ticket if you find something which requires immediate fix and mention this in your shift report.

2. Prepare a short set of slides for the L1DPG meeting on Mondays summarizing your findings over the last week

• The slides should contain:
  • • The dates of the shift period.
    • The list of runs analyzed in the week.
    • A number of how many GOOD and how many BAD flags you put on the analyzed runs.
    • A list/explanation for the runs that were declared BAD.
    • A detailed explanation of every decision that differs w.r.t. the online DQM shifter for data quality of L1T/L1Tcalo/L1Tmu.
    • A list of recurring problems/anomalies (i.e. problems that are not limited to one run only).
    • Your feedback on the trigger shift itself such as list of things that can be improved. Feel free to report suggestions or anything else that you deem useful to the L1 community.
    • Example report could look like this: Example report
• Present the summary talk at the L1DPG meeting on Monday, the last day of your shift. The meeting is usually called "L1DPG meeting". If there is no such meeting on a Monday or you cannot make it to the meeting for some reason, please send a message to cms-l1t-prompt-feedback-task-force@cernNOSPAMPLEASE.ch, so we can agree on another way of presenting the results.

Glossary

Some Run Registry acronyms:

Contact information

In case of problems, submit a JIRA ticket.

%RESPONSIBLE% Santeri Laurila

-- JamalTildonRorie - 2017-08-23