%CERTIFY%

TRT Offline Shifter Instructions

Introduction

This page contains instructions for the TRT Offline Shifter (OTP task ID 530360). On this shift you have several primary tasks:

  • You inspect the TRT Data Quality (DQ) of recorded and processed ATLAS runs every day by looking at a set of monitoring histograms. You report about your findings, upload DQ defects to the central database and sign off runs for further processing.
  • You also inspect the automatic calibration output by looking at the PDF output generated when the calibration finishes. You should ensure good calibration, report your findings, and request help if the calibration is sub-optimal and needs to be updated.
  • You need to attend the weekly DQ Meeting on Wednesday at 16:00 CERN time and to answer any TRT questions in that meeting. NB: During initial turn-on periods these meetings may be held more frequently on MWF or even MTWTF.
  • You should give a report at the TRT Operation Meeting (Mondays) during the week of your shift period and the week following your shift period summarizing the week’s findings.
  • In all cases you are supported by the TRT Offline Expert (OTP task ID 530359).

Prerequisites

Before you can take shifts, make sure you fulfill the following technical requirements:

  • No special software tools are required.
  • You need a CERN account and you have to be a member of the CERN e-group “atlas-readaccess-current-physicists”. If you are not sure about this, go to CERN e-groups, and search for "atlas-readaccess-current-physicists" with Only groups I am on selected.
  • You need to be registered in OTP for booking TRT Offline Shifts (see below). Send an e-mail to Dominik.Derendarz@cern.ch and Anatoli.Romaniouk@cern.ch and ask to be include in the list.

You will only need to do this once.

Booking Shifts

Like all ATLAS shifts, TRT Offline Shifts are managed and booked through OTP.

  • Log in to OTP and sign up for shifts (OTP task ID 530360) under “Book My Shifts”.
  • Select line starting category ID # 570203 and then select your dates.
  • If you don't see category 570203, follow the instructions in Prerequisites.
  • If you are eligible for more than one type of shift be sure to select the correct line.
  • Don't use the search function (Magnifying Glass Icon) if you don't see a line labeled 570203.
    • This will lead to a page where shifts can be selected but they won't be saved.
  • Shifts should be booked in blocks of seven days, starting on Thursdays and ending on Wednesdays.

Shifts are generally distributed on a first-come-first-serve basis. Don’t miss your chance to be a TRT shifter!

The Expert Shifter

The TRT Offline Expert is there to support you, to answer your questions, and to investigate problems in detail.

  • Read the TRT Offline Expert TWiki to learn about the differences between the TRT Offline Expert and you, the TRT Offline Shifter.
  • Log in to OTP, go to “Tasks”, and search for task ID 530359 to find out who is the current expert.

Don’t hesitate to contact your expert by sending an email to the experts shifter e-group (atlas-trt-expert-shifters) whenever you need help! He/she is there for you. Please contact the expert before contacting anyone else – answering questions from you is part of the expert’s job!

For experts: TRTOfflineCalibrationTroubleshooting

Preparations

Take care of the following items before your shift:

  • Make sure you have read and understood everything below. Ask the expert if you have any questions.
  • Read the TRT DQ Defects TWiki to see which possible DQ defects are currently defined for the TRT.
  • Make sure you have viewed the TRT Offline Calibration TWiki to help understand what you are looking at for calibration and look at the TRT Offline Calibration History TWiki to know which calibration was used for ES1 (first pass called express) and BLK (second pass called bulk).
  • Make sure you have subscribed to the mailing lists mentioned below at least from three days before your first shift until your last shift. You can always unsubscribe when your shift period is over.
  • On the Monday before you shift, attend the TRT Operation Meeting (Mondays, 16:30 CERN time), either in person or remotely. Listen to the previous Offline Shifter’s report for the first half of the past week to catch up with the latest news. Attend the Data Quality Meeting on the Wednesday at 16:00 the day before your shift starts and listen to the previous shifter's report. After the DQ meeting arrange to talk with the previous shifter or the previous expert shifter about any issues that came up during the second half of their shift. Do not hesitate to ask the previous shifter or the previous expert shifter about any issues you don't understand.

Training

New for Run 3, we have started to organize training sessions; the first is in TRT days of June 2022 (see the Friday session): Run 3 TRT shift training.

Previous 2016 training: The Indico page for most recent session on 15 April 2016

Before your first Offline Shift, you should prepare two “shadow reports” in the week before your shift and send them to the Offline Expert to become familiar with your tasks. Although, there is no formal certification for Offline Shifters, you may begin to take the shifts only after the expert shifter approves you sample reports. You should also communicate with that week's TRT offline shifter who can also answer your questions and should explain to you the status of the TRT and what issues with the detector and the data quality reporting have been seen on their shift.

Daily Tasks

Inspection of Runs for DQ

Look at the TRT Offline DQ histograms to assess the data quality of the latest runs:

  • The list of runs to be checked is announced every day on atlas-dataQuality-automatic-notifications. Those e-mails also contain a link to the ATLAS Run Query. The query will display plenty of information about the runs, including the current DQ status of all detectors.
  • Open the DQ Shifter TWiki* and for first express pass (ES1) processing for each run to inspect “express_express” stream histograms. All relevant histograms for the different detector regions are listed on this webpage. One can also directly open the DQmonitoring tool DQ Web Display for each run and go to “ES1 → express_express → Entire Run → Overall Status → Inner Detector → TRT”.
  • If the histograms load slowly or not at all, one can also try the DQ Web Development Display, use Google Chrome, or even use RUCIO to download the histograms and look at them with ROOT. To download the histograms using run 271421 as an example (requires ~1 GB of space):
     rucio list-dids data15_13TeV:*271421*merge.HIST*
     rucio download name_of_the_data_set_you_are_interested_in

  • Look at the monitoring histograms under “Summary”, “TRTB”, “TRTEA”, and “TRTEC”. Histograms from other iterations (“BLK”) and other streams are not available or not relevant at this stage.
    • Note that you have to check the "include non-stable beam runs" box to list cosmic and test runs; if the box is unchecked (default) only collision runs will be shown.
  • Note anything that looks suspicious or unusual to you. This might be, for example:
    • Some runs might have very few statistics, in such cases you should consult with your expert shifter on how to evaluate such runs; eg. Low_Stat defect
    • Noisy or dead parts of the detector. These will typically show up in occupancy and track distributions. Particularly problematic straws or chips can later be masked by the expert shifter requesting a new calibration. If this happens, you should check further if things have improved after reprocessing.
    • Bad timing. This can be seen in event phase, leading edge (drift time), trailing edge, or the R(t) relation. Timing will look unusual if the solenoid magnet is off.
    • Distributions that differ significantly from their references, or that look strange in some other respect.
    • Angular distributions that are not flat in φ, particularly for collisions and for single beams. Certain angular distributions will not be flat for cosmics.
      • In case you see a change make sure this change is not related with the SCT and Pixel. In case the SCT or Pixels are not working in some sectors, the number of tracks vs φ will have a drop, to make sure this is not due to the TRT, check the number of hits on track vs φ and LL occupancy because a drop is expected there also if some TRT sectors are off or giving problems.
    • Changes in the calibration constants. This is seen by a shift or broadening in the time residuals (compared to the reference) or a broadening in the spatial residuals.
  • Automatic checks are run on most of the histograms. If you find distributions that are marked yellow or red in the DQ Web Display, you should pay extra attention!
  • The automatic checks are not perfect. If you think that a histogram looks bad even though it’s flagged green, it is better to trust yourself than the computer.
  • Finally, decide if a defect should be uploaded for the run following the TRT DQ Defects TWiki. If you are uncertain about this or there is no defect available that matches the problem you encountered, ask your expert. Keep in mind that you make the final determination of TRT DQ for the express stream processing.
  • As the bulk (BLK) processing becomes available for each usable run, inspect the histograms generated with the Physics_Main stream which use all of the physics events for the run rather than limited number of events processed in the ES1 processing. If as a result of inspecting the express stream the expert shifter has uploaded new calibrations, ensure that the new calibration has improved things (like reduced residuals and -if available- efficiencies as high or higher than the express stream). In any case the histograms of the BLK processing should be no worse than those of the ES1 processing. If you feel the BLK processing has changed the DQ determination, discuss this with the expert shifter before adding or removing defects. Please note that the express stream processing takes around 12 hours from the actual end of the run and the bulk processing which takes about an additional 48-hours after the express stream calibration loop finishes (will likely be reduced to 36-hours or even 24-hours). Note these findings and/or changes between the express processing (ES1) and the bulk processing (BLK) and inform the expert shifter.

Inspection of Runs for Calibration

  • If you have signed up for the correct mailing list below, you should receive an email as the express stream calibration job finishes for each run.
  • The calibration automatically runs using TMS (Tier0 Management System). TMS sequentially runs 5 sets of jobs for each run and each stage of job waits for the previous one to complete successfully (in case of any job failure contact the expert shifter). You can monitor the status of TRT calibration using The TMS monitoring page. The page will have many non-TRT tasks, but you can filter to choose only the TRT ones (user: trtcalib) using the boxes on the left-hand part of the screen.
  • The email summary sent from TMS are indicative for possible problems however to view the full calibration output, go to The calibration webpage (contact the expert shifter if you cannot access). If you want to check a run with finished calibration but don’t see it, find it in the drop down menu box and hit "Add". Once you find your run, click on the "i" button next to it and open the PDF file (this point will be checked with Alex).
  • As mentioned, most importantly you want to note the position residual width and time residual shift for all parts of the detector, but please look for anything else that is out of place. This can be a strange R(t) curve, deviation of the mean of the time residual plot from +0.5 ns by more than 0.3 ns or chips or boards with different timing (as happens when modules or cables in the TDAQ readout hardware are replaced or fixed.
  • If you think an update of the DB is needed, please don’t hesitate to ask your expert shifter. Once an update has been done, please be extra careful when checking the calibration jobs for subsequent processings to make sure that all looks OK.
  • Remember the TRT Validity Gate is at a fixed position, 4.5 DTMROC time bins (14.0625 ns) and has a fixed width of 9 DTMROC time bin (28.125 ns) which is set under the assumption that calibration is made so that an 18 ns drift time corresponds to a 1 mm drift radius. If the calibration is not updated occasionally, the hits from stiff in-time tracks will fall out of the validity gate and be rejected while hits from tracks in the out of time bunch crossing will fall into the validity gate and be considered. This same calibration convention and validity gate definition is used for all gas mixtures.
  • As mentioned above more details about the calibration can be found on TRT Offline Calibration Information for Shifters TWiki.

Inspection of Runs for Straw Status

  • On the calibration webpage you will also see next to each run a link that says "calibration". Click here then click on "view" next to TMS/run_$NUMBER/TRT_StrawStatusReport.$NUMBER.pdf. This will load a PDF which summarizes the status of straws in the run, including straws that are newly problematic in comparison to the reference (the reference should usually be the most recent stable run). Only newly problematic straws will be shown in the histograms with "number of additional" in the label (pages 4 and above). In most cases the label is hidden by the stats box but behind the stats box says "excluded straws". The numbers in each bin of these histograms correspond to the number of straws in that bin (this is basically just reading the z-axis for you).
  • Inspect this file and see if there are a large number of newly problematic straws. If there are >50 newly problematic straws in a given region, contact the expert shifter.
  • Also check the expert shifter's problematic straws status report (TRT_StrawStatusReport.txt) and make sure there are no incompatibilities between this and the pdf from TMS before the expert shifter uploads.
  • Note these reports will show straws that have high occupancy and straws that have HT issues, but these problems are not included in the conditions upload. Only straws with 0 hits or with <70% efficiency are included in the conditions DB.
  • See UploadTRTStrawStatus for more information about the definitions of problematic straws, the info in the straw status pdf plots, the formatting of TRT_StrawStatusReport.txt, and the uploading of dead and low efficiency straws.
  • For a good straw status reference run, see https://twiki.cern.ch/twiki/pub/Atlas/UploadTRTStrawStatus/TRT_StrawStatusReport.327057.pdf

Reporting

Report about your findings:

  • By 15:00 CERN time, daily: Send an e-mail with a short summary of the inspected runs (including run numbers and start/end times), a brief description of interesting features, and your decisions about defect uploads to atlas-trt-offline-shifts. You should also include the results from any finished calibration jobs: the position residual width, the time residual mean shift, whether any chip looks like they have bad timing, and whether the R(t) curves look okay. If there is something particularly strange about the DQ, if you are unsure about uploading a defect for a certain feature you encountered, or if you think the calibration has not completed entirely successfully, we will discuss your findings via e-mail or in a short phone meeting. Update your report afterwards if needed.
  • Paste your report to the DQ Logbook (make sure to tick “Systems: TRT”) and the TRT Offline DQ Reports TWiki. On the first day of your shift, create a new section for your week on the TWiki page.
  • By 16:00 CERN time, daily: Upload defects and sign off the run as described below. Don’t forget to sign off even if you didn’t upload any defects.
  • At 16:00 CERN time on Wednesdays: Attend the Data Quality Meeting, either in person or remotely. The Offline Expert will be there, too, but you should be ready to answer possible questions about defects you uploaded and any calibration results. The meeting can also help you to understand features that should be expected for upcoming runs. Please also check that all the runs of the previous week were signed off. It is possible that there are remaining runs to report on from the previous shift because of delays in processing the runs; e.g. if the calibration was not finished or new DB was uploaded and the BLK stream had to be checked.
  • Have a look at previous reports at TRT Offline DQ Reports TWiki to have better idea.

If an emergency comes up and you are unable to perform some of those tasks on a certain day please let the expert know at least 12 hrs in advance!

Setting Defects and Signing Off Runs

Upload defects and sign off runs for the TRT:

  • Go to the ATLAS DQ Defect Entry Page.
  • If you have to upload one or more defects:
    • Go to the “Upload” tab.
    • Select the defect you want to upload from the tree on the left side of the page.
    • Enter the run number.
    • Enter a luminosity block range. Usually defects in the TRT are valid for the whole run, so the default “1-End” should be okay.
    • Enter a meaningful comment describing what exactly you encountered in the data, possibly including a link to the ATLAS e-log.
    • Do not check the “expected recoverable” check box unless you were told so by the expert.
    • Upload the defect. You can ask the password from your Expert shifter.
  • To sign off the run:
    • Go to the “Sign off a run” tab.
    • Enter the run number.
    • Select “TRT” as System (if not selected automatically).
    • Enter a comment if you like.
    • Submit the form (same password as for the defect upload).
  • You can also access the upload page from the “Defect Database” link on the DQ Web Display page of the corresponding run. In that case the correct run is automatically selected but all systems are displayed, not only the TRT.
  • You can check the upload on the “Show defects in a run” tab. Enter the run number and the filter “TRT”, then click the “Load” button. After you are done only the defects you selected and uploaded should be displayed. Note that any successful change will immediately be reflected on the page.
  • To make an entry in the logbook -- needs to be done for ES1 and BLK:
    • Go to the DQ Logbook.
    • Enter the run number in the Search window
    • Click on the green + sign for the entry from author DQM System with Subject ES1 or BLK
    • Click the Reply button (not the edit button)
    • Click the System Affected menu and check TRT, uncheck Processing_Pass
    • Enter your report in the readout field, for example:
    • Click the Reply button
  • Make sure you also go through the procedures summarized on the 'ATLAS DQ Signoff Procedure' twiki!

Weekly Tasks

On the Monday just before your shift, you should attend the TRT Operation Meeting to familiarize with the status of TRT and on the first day of your shift you should attend the Thursday TRT SW Meeting to listen to the previous shifter's report.

On the Monday during your shift, you should summarize the relevant periods of your shift in a brief report uploaded to the relevant TRT Operation Meeting indico. You should also report about the whole period of your shift during the Thursday TRT SW Meeting just after your shift. This will also be helpful for the next shifter.

You should also report about TRT on the Wednesday DQ Meeting during your shift (note that there may be additional DQ Meeting calls as well).

TRT Offline DQ Shift Checklists - FOLLOW EVERY STEP!

Note that shifts are booked in blocks of seven days, starting on Thursdays and ending on Wednesdays.

Preparation - before starting your shift

  • Even if you took TRT shifts previously, attend an offline DQ shift training before your first shift this year.
  • Visit the Offline Shifter Instructions TWiki and make sure you are comfortable with all the content.
  • Subscribe to the Data Quality Notifications mailing list to receive daily emails about the runs to check.
  • Attend the Wednesday DQ meeting the day before your shift starts to be aware of the current problems. If needed contact the previous shifter for details.

Meetings and Reporting for your shift

      Shift Shift Shift Shift Shift Shift Shift  
Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu
OP   DQ SW       OP   DQ SW

  • Monday - Attend the TRT Operations meeting in the week before your shift starts - this is optional, but please try to attend
  • Wednesday Attend the DQ meeting the day before your shift period starts to be aware of the current problems. If needed contact the previous shifter for details.
  • Thursday - Attend the TRT offline Software meeting on the first day of your shift.
  • Monday - Report on your shift in the TRT Operations meeting during the period of your shift. Prepare a few slides listing the runs you have checked and the defects you have assigned to them.
  • Wednesday - Report on your shift in the ATLAS DQ Meeting on the last day of your shift (normally only Wednesday but there can be additional DQ meetings). Prepare a few slides listing the runs you have checked and the defects you have assigned to them.
  • Thursday - Attend the TRT offline Software meeting the day after your shift ends. Have at hand your report from yesterday's DQ meeting in case you need to show it.

Every day - during your shifting period

  • Check the runs mentioned in the daily DQ emails using the Data Quality Monitoring website.
    • Look for the runs (ES1 or BLK processing)
    • Go to express_express for ES1 and physics_main for BLK
    • Click Entire Run and navigate to InnerDetector > TRT
    • Check histograms Barrel (TRTB), and both endcaps (TRTEA, TRTEC)
    • If something is suspicious check the electronic logbook ELisA and contact expert shifter.
  • Check the calibration of the runs using the TRT Calibration website (ES1 only)
    If the calibration looks bad for a run, notify/consult the expert shifter.
  • Check straw status pdf files for the runs again found on the TRT Calibration website (ES1 only)
    If there is a large number of newly problematic straws, contact the expert shifter.
  • Email the TRT experts list (atlas-trt-expert-shifters@cern.ch) with new issues found or to request manual calibrations for runs needing them.
    - Specify the run number and the day the run was taken.
    - Include a link to a plot showing each issue you are asking about.
  • Enter any defects and sign off the run on the Defect Entry website
    go to Upload and change the flags.
  • Sign off the run(s); note that "ES1 sign off" and "BLK sign off" are separate:
    - "Sign off ES1 runs" clears the TRT_UNCHECKED defect.
    - "Sign off BLK reprocessed runs" clears the TRT_BULK_UNCHECKED defect.
  • Fill the TRT DQ Reports TWiki with your report on the defects observed or saying that all is OK.
  • Make an entry for each run (ES1 or BLK) in ELisA by replying to the corresponding email from DQM system, and informing about the sign off.
    In your reply, you can for instance include a link to your report in the TRT DQ TWiki.
  • Inform the TRT Offline Shift mailing list (atlas-trt-offline-shifts@cern.ch) when you have finished. Include in your email:
    - a link to your report in the DQ Twiki,
    - links to plots showing any new problems.

If you are unsure then contact your expert shifter or email the TRT experts (atlas-trt-expert-shifters@cern.ch) .


Old google docs - for temporary reference
Before your shift please fill the checklist at https://docs.google.com/document/d/1228R3RtVkIbAqQhspzgRCq27raqt3l0qQMnHsiJzU00/edit?usp=sharing
During your shift please fill the checklist at https://docs.google.com/document/d/1w-RpttZWQ5OqgUXtD5aA7nneEEen-qu7aaoF8_Z0jb4/edit?usp=sharing
The outdated 2016 checklist can be found here TRT_Offline_shifter_check_list.pdf

Mailing Lists

As the TRT Offline Shifter you have to subscribe to the following CERN e-groups:

You can subscribe to all groups (and unsubscribe from them) yourself by following the links on this page.

Useful Links

As the TRT Offline DQ Shifter, you will be using some of the following web pages and services often:


Major updates:
-- Main.SteffenSchaepe - 18-Mar-2011
-- Main.AdrianVogel - 13-Jul-2011
-- Main.SerkantCetin - 28-May-2015
-- Main.AndrewBeddall - 13-June-2022 (shifter checklist)

Edit | Attach | Watch | Print version | History: r32 < r31 < r30 < r29 < r28 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r32 - 2022-06-13 - AndrewBeddall
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback