MDT Shift Manual

Phone Numbers

Name Number Expertise
Muon Pager 160226 on-call expert
ACR Muon desk 71365 shifter (i.e., you)
Sotiris 163046 Muon Run Coordinator
Enrico 162197 DAQ
Alberto 162877 DAQ, Endcap
Ludovico 163664 Barrel
Fabio 164492 Barrel
Joao 162797 Endcap
Joerg 163397 Gas
Jianbei 162532 Gas (Endcap)
Henk 165351 MROD, JTAG
Shulamit 162815 DQMF
Gabriella 165628 GNAM
Eleni +41762393261 DCS

Accounts for new shifters

Before starting a shift make sure you have the 3 accounts you need:

Before you begin

Checklist

General --> checklist

checklist1.png

First thing you have to do when you start a shift is to complete the desk Muon 1 checklist and shifter Muon 1 checklist.

Follow these 2 checklists step by step, execute what they ask, and click on each check botton. When you reach the end click on OK. This sends a message to the run controller who will know that you are ready to start a combined run.

If you need some help on how to execute some step, you can click on the Help botton Help.png for each step and a window will open with some instructions.

Desk checklist

A reminder of the more important windows you have to open on your workspace. deskchecklist.png

Shifter checklist

A reminder of the more important steps you have to do to make the MDTs run on the combined partition.

checklist.png

The more important operations are:

  • Low Voltage power supply: are all the chambers taking part in the combined run on? Do they report a Vref = 2.5V?
    • Turn them on following the instructions.

  • Initialization: are all the chambers initialized?
    • Initialize the chambers which are in ERROR, FAILED, or UNKNOWN state.

  • Gas: do the chambers which are supposed to have High Voltage on have good gas? Is the gas pressure around 3bar? Is gas flowing in the chambers?
    • Check the gas status, and call Joerg (Barrel) and Jianbei (Endcap) to confirm that everything is good.

  • Turning_ON_HighVoltage: are all the chambers with good gas on? Are both multilayers of these chambers set to 3080V? Are currents negligible (below 1uA)?
    • After checking that these conditions are good, ramp up the High Voltage to 3080V. If chambers report currents above 1uA, make an entry in the Atlas elog. Make sure to indicate chamber, multilayer, and current value.

* MDT_Standalone Start a short STAND ALONE run and make sure everything is fine.

* MDT_Control_Panel To include new chambers or exclude some others that give problems, use the Mdt Control Panel (combined or stand alone partition) and add/remove chambers (remember to commit each change!). Before starting the combined run remember to compare the two control panels of the combined and standalone partitions.

Workspace

Sixteen full displays are available to simplify the work of a shifter. Shifters are free to organize as they please their working environment, here is presented a list of the most important application to keep under control:

MDT DCS FSM

Muons --> MDT DCS --> MDT DCS FSM

Shifters should be checking, at least every thirty minutes, the status of the gas distribution in MDT chambers. This is available in the framework of PVSS, which is also used to manage and monitor the status of power supplies. More information here and here. Use the same DCS windows also for the Initialization

DCS Alarms

Muons --> MDT DCS --> MDT DCS Alarms

In the list of messages, it is important to note on the elog the temperature of Power Supply boards, when a warning, or error, is issued. In order to be able to reconstruct the origin of the error, please indicate in the elog all the information contained in the DCS Alarm message.

MRS Monitor

TDAQ --> DAQ Panel --> Read Info --> MRS

The MRS Monitor returns all the messages issued by the run control application. By default, only messages regarding MDT, TGC, RPC, or CSC, are displayed. It is recommended to set, in the top-right corner of the window, the maximum number of messages to 2000, rather than the default value (20). The MRS Monitor is a primary source of information for expert. Feel free to copy and paste in the elog all the WARNING, ERROR, and FATAL messages that deal with Muon systems, for expert to look at. It is possible to change the MRS subscription, i.e., the filter to apply to messages before presenting them in the window. The subscription is changed by clicking on the MRS+ icon in the center of the MRS window. Expressions can be combined by using the following boolean operators: | (OR), & (AND), and ^ (NOT). See here for a non-exhaustive list of run control errors.

Spy IGUI

TDAQ --> DAQ Panel --> Read Info --> Spy IGUI

A Spy IGUI allows one to directly monitor the content of the partition that is run, without the permission to issue run control commands. The MDT shifter can use it to get a more detailed picture of which parts of the Muon system are included in a run, and provide these pieces of information to Muon experts by entering them in an elog entry, at each start of a run. As a side note, a Spy IGUI can be started from the same DAQ Panel opened in the previous step to start the MRS Monitor. It is also advisable to reduce to a minimum the open windows, to avoid cluttering the workspace. A confused desktop can lead to the loss of important pieces of information that could be important for experts to know.

The MDT System

In order to run the system successfully, it is necessary to ensure that the chambers satisfy the following checks:

Power Supplies

The MDTs require Low Voltage (5V) for the electronics and High Voltage (3080V) that is applied to the tubes. These are supplied by power modules located in miniracks in UX15 and controlled from USA15. A PVSS interface integrated into the general ATLAS DCS controls and monitors the power supply remotely, recording the history of changes in voltage and current.

From the control room, use the DCS GUI that may be accessed directly from the DCS Menu:

DCS --> DCS FSM Screen

A link is also available from the Muons Menu:

Muons --> MDT DCS --> MDT DCS FSM

DCStot.jpg

The system that will be used in the final configuration is the DCS FSM running on the machine cerntsatldcs01.cern.ch. If you select DCS FSM Screen, you should be connected remotely to the relevant machine.

The root account allows the user to view the status. However, in order to make changes, it is necessary to use the individual login and password. All shifters should contact Stephanie Zimmermann (zimmerst@cernNOSPAMPLEASE.ch) and make sure that they have a working account.

In the top left section of the GUI, a tree is displayed. Select MDT and then Barrel A, Barrel C, Endcap A, or Endcap C.

Turning ON Low Voltage

In the DCS GUI, there is a color scheme for the STATUS and the STATE : ( the box is the STATE, the dot on the middle of the box is the STATUS) :

Color Code Status State
green OK READY (HV+LV)
yellow WARNING only 1ML HV ON
orange ERROR -
blue - OFF
light blue (grey) - HV Ramping
red FATAL (e.g. unplugged) UNKNOWN or NO LV
turquoise blue - STANDBY (only LV)

When you log in and you cannot see any color about the state/status of the chambers, is because the lock on the left of the label MDT is open. You have to click on the lock and take the local possess of the panel (the lock gets closed).

The LV can be turned on by changing the state for the whole subsection next to the name of the subsection on the tree on the top left part of the GUI (this changes the state for all chambers in that subsection, so only chambers that should be modified should be included).

MDT --> You can choose between: Barrel side A: MDT_BA , Barrel side C : MDT_BC, Endcap side A MD_EA, Endcap side C: MDT_EC

DCS_1.png DCS_2.png

Then, under each of these selections, there are other tree options: power supply, Temperature, DDC (not available now). If you select the PS you can see the global situation of the power of the chambers, with the State box representing each chamber. For example the Barrel A is on the picture:

DCScolors.gif

Now you can change the Voltage of the chambers all together, for stations, for sectors or individually (clicking on the box representing the chamber):

GOTO_NOT_READY = LV, HV OFF:

GOTO_STANDBY = LV ON, HV OFF:

GOTO_READY = LV ON, HV ON.

Clicking on each chamber you can see also the Voltage, the current of each chamber and their history.

Turning ON High Voltage

Before turning on the HV, ensure that the gas situation is satisfactory. Please see the gas section in this manual for more detailed information.

Check with Joerg (Barrel) and Jianbei (Endcap) about the gas status of the MDT chambers.

If the HV is off and the chambers have adequate gas, the HV can be turned on using the DCS GUI. The trip current should be raised to 25uA while turning on the HV. Once the full voltage is reached, the trip current should be lowered to 5uA. These values should change automatically. The shifter should also ensure that the current is close to zero.

DCS-power-endcapC.png

In case the DCS GUI is unavailable

It is possible to access the PS mainframe directly from a terminal with the command:

muondaq@pc-atlas-cr-06> telnet 10.145.48.242 1527

When prompted to log in, use:

Login: admin

Password: admin

telnet.jpg

Use the tab button to open the Menu and select channels. Then, scroll down (use the up/down arrows) to the desired module and channels and turn them on/off using the space bar. To exit, use the tab button to open the Menu again and select Log Out. It is advisable to contact an expert who can give directions on how to use the terminal interface.

Monitoring of PS of chambers

If you find with the DCS that some chamber has problems with both Low Voltage or High Voltage, you can monitor its conditions (e.g the history of current (blue line) and voltage (red line) ) just cliking on the chamber you whant to minitor. It will appear a windows like this one:

<span class=BIS3A10_weird_stuff2.png" />

and OF COURSE!!! in this case there is some problem!!!!

Gas

The MDT chambers operate with a mixture of Argon (93%) and CO2 (7%) at a pressure of 3 bar. The gas is monitored in racks in UX15 that are connected directly to all the chambers, and these racks are then connected to the broader gas system. There is a PVSS interface in order to control and monitor the gas system remotely. This has been integrated into the general ATLAS DCS and may be accessed in the Control Room under the Muon Menu:

MDT --> INFRASTRUCTURE --> GAS

tree_GAS.png

You can see a general window with the situation of the Gas distribution.

tree_GAS1.png

It is important to keep under control chamber pressures, and gas inflow/outflow rates.

Initialization

The MDT chambers are initialized via DCS. The configuration parameters are read from the configurations database and input into a JTAG string that is sent out to the chambers. The chambers should be initialized once the electronics have Low Voltage ON (see the previous section on power supply).

It is now possible to use the DCS FSM pannel to initialize all the chambers. You should already have a DCS FSM pannel open from turning on the power supplies. (see previous section on power supplies).

Click on a node (partition, sector, chamber), then a window pops up allowing you to choose what to do. Please note that actions are only available if somebody has control of the FSM tree/nodes, in the usual way; you need to be logged in and be a member of the mdtoperators group (as for HV/LV) to issue any command.

You can initialize all chambers by going to the top level of each partition (BA, BC, EA, EC) and clicking on the STATE field on the right side of JTAG partition (first column).

tree.png

Actions are:

  • INITIALIZE initialize all selected chambers, irrespective of what state they are in

  • REINITIALIZE initialize only those chambers not in state INITIALIZED

  • RESET_STATE set the state of one or more chambers to NOT_INITIALIZED

  • ABORT_INIT only available in state loading, cancels the init request.

Example (Barrel side A):

MDT -->*MDT BA*--> JTAG BA

A panel appears with the JTAG Initialization Status:

initialization.png

You can also initialize by sector or chamber in a more detailed branch of the tree:

Example MDT --> MDT BA --> JTAG BA

By clicking on one of them you can see the situation for each sector in more detail.

Example MDT --> MDT BA --> JTAG BA --> INNER

initializationchamber1.png

It is possible to initialize a single chamber by selecting it and clicking * INITIALIZE *.

initializationchamber.png

Once you have initialized all chambers, check the color status:

  • GREEN: the chamber is ready to read out data.

  • RED or ORANGE: For any chambers which are not GREEN, check the error/warning messages by clicking on the chamber in the figure. The JTAG information appears in the right side of the panel. Sometimes during initialization, a chamber may fail. Try to initialize it again by clicking * RESET_STATE * and then * INITIALIZE * for that chamber. If it is still unable to initialize,

  • GRAY: the chamber is disabled, and will not be included in the read out. To include it, you need to click on the red cross next to the STATUS of the chamber. init_excluded.png click "Enable" init_excluded1.png Then initialize the single chamber by clicking * INITIALIZE *.

Taking a run

Once all possible chambers are initialized, they are ready to be included in the read out, and a run can be taken.

MDT Combined run versus Stand Alone Run

A feature of the system is that it is partitionable, meaning that it can be sub-divided into independent but fully-functional units. The MDTs can thus take data independently and in parallel with other sub-detectors (standalone mode) or combined with other parts of the detector (combined mode). During the commissioning milestone there are periods devoted to independent running, either to debug problems or to ensure stable and good quality data taking (e.g. calibration runs). However, since the goal is to run integrated with other parts of the detector, the combined software is typically run from the central station and the individual shifters may monitor the run in spy (observer) mode.

TDAQ Software

The Trigger and Data Acquisition (TDAQ) system allows for data-taking and selecting events using the trigger. During the commissioning runs, the shifter must configure, control and monitor the MDT system, and these tasks are carried out using the online software.

DAQ Panel Basics

You can open the DAQ panel by going to the TDAQ menu:

TDAQ --> DAQ Panel

On the left side of the DAQ Panel, specify the setup script used and the online database file used for the run. In particular for the M-combined runs select:

Setup scripts: /det/tdaq/scripts/setup_TDAQ_14.0.10.sh

Database File: /db/tdaq-01-09-00/combined/partitions/ATLAS.data.xml

Click on Get Partition, select the partition named muon_combined and then click Read In. Under the main menu, SpyIGUI opens a read-only copy of the Central TDAQ GUI.

DAQpannel1_generale.png

The OKS button opens the OKS Data Editor with the current partition to edit the configurations database. If settings in the database are changed and the TDAQ GUI is running, it is necessary to reload the database from the command toolbar of the TDAQ GUI (blue arrow on the right of the partition name) when all components are in NONE or INITIAL state. The MRS button opens the Message Reporting System (this is also available from the TDAQ panel). This monitor presents all the error messages from all components in the DAQ system and provides capabilities to filter messages.

In the monitoring tab, the OHP button opens the Online Histogram Presenter and OH opens a root TBrowser with the GNAM histograms (more information below under Monitoring). The DQMF button runs the Data Quality Monitoring Framework that consists of quality algorithms that applied to GNAM histograms (see below the section under Monitoring, and the DQMF Manual). The IS Monitor (Information Service Monitor) button shows detailed information for run parameters, run control, monitoring, and PMG. This feature is also available from the TDAQ GUI. The Data Flow (DF) key provides similar information as the TDAQ GUI Data Flow tab (see next section on TDAQ Panel). The IS Logger opens a GUI for IS. Do not click the icon! The IS Logger crashes often leaving a window which cannot be closed. Finally, the BUSY button opens the CTP Busy Panel.

TDAQ Panel Basics

The TDAQ panel is divided into three main section. The left section is the run control panel that displays the state of the system, the commands that are available (Boot, Config, Start, Pause, Stop, Unconfig, Shutdown). This section also contains a summary of Run Information such as the Run type, the Run number and the rates and number of events at different stages.

The bottom section is the MRS window that displays the (error, info and so on) messages. In the GUI toolbar the user can select to display the messages in the window in LONG or SHORT (this is the default) form. A separate MRS window may also be opened from the main DAQ panel.

On the top right side, the TDAQ panel has multiple tabs with information for running and monitoring. The tabs that are relevant for the MDTs are:

  • Run Control tab: This tab provides information about the run control. It is divided into two parts: the left side displays the tree of the segments in the partition and by expanding it the user can see the applications and controllers. The right side has three tabs on the bottom including the Status and Commands available to the user and information regarding the database configuration.
  • Infrastructure tab: This tab appears at the startup of the TDAQ GUI. Right-click on field to view the setup and application logs.
  • DataFlow tab: The information provided here is similar to Data Flow (DF) under the Information Service (IS) Monitor. All ROS segments show the number of L1 events. The Data Flow Manager (DFM) segment has the number of L2 accepts/rejects and L2 and Event Filter (EF) rates.
  • Run Parameter tab: The Filename Tag is used to tag groups of runs. Typically during M5 the tag m5-combined is used. Use tab to Enable/Disable recording of data.
  • MRS tab: This tab allows the user to configure filtering and formating of errors and warnings in TDAQ GUI lower screen.
  • PMG tab: The Process Manager tab allows the user to view and manually start/stop processes on online computers.
  • Segment &amp Resource tab: This tab can be used to enable/disable segments and subtrees by right-clicking on them. Note that the changes must be updated to the database (click on the button below) before running. Beware that it is crucial to check that the changes have actually been implemented in the database after updating since this operation sometimes fails without warning.

TDAQ.jpg

MRS Monitor

Monitors the messages produced during the run.

MRS.jpg

To have this, click on MRS on the main menu of the DAQ Panel and a windows will open. Set on the top right the number of lines you want to read (2000) and press OK. Then you can filter the messages you want to read. (For example only MDT related messages, or only WORNING, ERROR messages) clicking on the box MRS with a white space. A windows will appear windows where you can choose what kind of messages you want, spacing each other with the pipe | (OR) or with & (AND). MDT & ERROR | MDT & FATAL | MDT & WARNING should be an useful combination of filters... Expressions can be combined by using the following boolean operators: | (OR), & (AND), and ^ (NOT). See here for a non-exhaustive list of run control errors.

MRS2.jpg

Then click on * Set * and * Set new subscription *.

buisy.jpg

During the run you have also to monitor if MDT goes in Buisy state (that means that they will cause a stop of the run). To do this you can click from the Monitoring menu of the DAQ panel: Busy If you see that the run is stopped, have a look of this panel just to see if the stop is caused by MDT. If you see that or MDT B, or MDT EC are Busy 100%, that means that MDT caused the stop of the run, and you have to check the reason. Try to find on the messages of error if you can find the reason. (Sometimes you have just to initialize some chamber that loose the initialization.)

You need to do a Stand Alone run every time you change something on MDT Configuration (add some new chambers ore remove some others with problems). This is useful for debug. Whenever you reach a stable situation on Stand Alone run, you are ready to enter on the combined run.

In order to switch from a global run into MDT standalone mode running with the barrel, open a terminal as muondaq and type:

muondaq@pc-atlas-cr-06> tcsh

muondaq@pc-atlas-cr-06> mdt

muondaq@pc-atlas-cr-06> run

In order to run with the endcap, type instead:

muondaq@pc-atlas-cr-06> tcsh

muondaq@pc-atlas-cr-06> mdt2

muondaq@pc-atlas-cr-06> run

The Barrel and Endcap cannot run in combined standalone mode, but both these partitions can be run simultaneously. When the GUI has loaded, in the Segment &amp resources tab select the MDT RCD (select MROD channels), MDT ROS, TTC and EB (Event Builder) segments and channels that you want to run. Disable the other segments. Make sure that you have the MROD HW_InputChannels desired enabled. Also select the appropriate EB channels that are included in the run. The names of the enabled channels should match the MROD crate names. Commit the changes to the database. Then return to the Run Control tab and follow the steps boot, configure and start.

In order to enable/disable a chamber, the MDT Control Panel provides an extremely intuitive graphical interface. The user needs to select the MROD crate where the chamber is read out, mark the chamber of interest, and commit the changes made to the database.

MDT Control Panel

This panel allows you to quickly check which chambers are included/excluded from the current MDT partition, and to add or remove chambers from it. The MDT Control Panel reads and modifies the OKS database files located in the directory $DB_LOCATION/segments/MDT. The MDT Control Panel is a java application installed in the ~muondaq account in point1.

The MDT Control Panel is activated as follows (run standalone):

muondaq@pc-atlas-cr-06> tcsh

muondaq@pc-atlas-cr-06> mdt

By default, after the mdt command, the MDT Control Panel points to /atlas-home/1/muondaq/databases, which is the location of the files used in standalone runs.

muondaq@pc-atlas-cr-06> MdtCP

The command MdtCP is a shorthand for:

muondaq@pc-atlas-cr-06> java -cp ~abelloni/java/releases/MdtControlPanel_v7.jar MdtControlPanel

To change the database for the combined run:

muondaq@pc-atlas-cr-06> tcsh

muondaq@pc-atlas-cr-06> mdt

muondaq@pc-atlas-cr-06> setenv DB_LOCATION /db/tdaq-01-09-00/muons

muondaq@pc-atlas-cr-06> MdtCP

MdtCP.png

The interface is very intuitive. Each chamber is indicated by a button. White buttons represent chambers which are excluded from data taking, while dark gray buttons represent chambers which are included in the current set of online chambers.

The operator can include/exclude chambers by checking the "Include/exclude chambers" option and then continuing to simply press the chamber's corresponding button. A yellow border indicates that the status of a chamber (in or out) has been modified. Changes of the chamber configuration are committed to the database files in use by pushing the Commit button. A confirmation window will appear, and require to insert the name of the person who is committing a new configuration.

The Refresh button, on the other hand, is used to load the actual chamber configuration defined in OKS database files. This operation clears all modifications (indicated by yellow borders) currently shown by the MDT Control Panel. A scrolling menu allows for the selection of different MROD crates.

Data Quality Monitoring (DQM)

A variety of tools are available to monitor chamber behavior and detect problems with the detector hardware or in data-taking. Histograms are produced online using GNAM and an Online Histogram Presenter (OHP) facilitates their viewing. In addition, quality algorithms are applied to the GNAM output using DQMF, providing error and diagnostic messages. The monitoring information is displayed in separate interfaces.

GNAM

GNAM is an Online Monitoring structure that allows monitoring of detector status and running conditions. For the MDTs during M6, GNAM has been implemented at the ROD level and provides histograms such as TDC and ADC spectra, hits per chamber and tube and so on. In order to setup GNAM it is necessary to provide chamber and run information.

Before starting the COMBINED run check and/or correct the files (chamberlist.txt) located in the directory:

/db/tdaq-01-09-00/muons/data/gnam_setup/mdt

The filenames are chamberList- followed by the name of an MROD crate (e.g., chamberList-EC03.txt). In particular, chambers not included in the run should be commented out and the missing chambers should be added. It is critical that the user modifies the runtype flag depending on the type of run ( 1= noise; 2= cosmics). The GNAM root files were located in /data/gnam.

Before starting the STAND ALONE run check and/or correct the files (chamberlist.txt) located in the directory:

gnam_setup/mdt

It is possible to automatically update the list of chambers utilized by GNAM with the autognamupdate script, located in atlasgw:~abelloni/autognamupdate/autognam.

Online Histogram Presenter

The Online Histogram Presenter (OHP) is a stand alone application displays the GNAM histograms, enabling the user to browse the histograms and display selected groups of histograms to be checked during the run.

OHP.jpg

In order to configure OHP, in the main DAQ Panel select the file:

OHP Opt: ohp -c /db/tdaq-01-08-04/muons/segments/DQM/MDT.ohpconf.data.xml -e ohpcommands

Select Read Info and select OHP in the monitoring tab. The OHP panel is organized in browser mode and with tabs which provide overview information per crate and per chamber.

Data Quality Monitoring Framework (DQMF)

Getting Started

To have the DQMF implementation running properly and checking the correct histograms, it is essential to have the DQMF linked to the proper partition and to have the DQMagents running. However, this is best described by DQMF experts, and we defer here. We assume everything has already been setup and the DQMF agents are running. If the Java display is not yet running, it can be started with the command

dqmf display.sh $TDAQ PARTITION &

if the environment variable $TDAQ PARTITION has been set to the partition name.

The Display

In figure 1, we see a picture of the running DQMF java display. Note that the display may have changed slightly for M4, but the general features we discuss here will still be relevant. On the left hand side of the display is the set of histograms to be tested, organized in a tree-like structure. The highest region of the tree is MDT. One can then click on the small blue tab next to MDT to open the tree to the next level down, which should display the subdetector (EndCap or Barrel) along with the side (A or C). Again opening the tab for the subdetector reveals a list of the possible layers. If we continue opening the tree we find a list of the sectors, followed down in the tree by the chambers. Within a chamber is the list of histograms that are tested for the chamber. If you then click on one of the histograms (under the chamber regions on the left tree) the histogram will be displayed on the right hand side. Also on the rights hand side, under Configuration Parameters, one can see the name of the histogram and its reference histogram, the algorithm used to test this histogram, some of the input parameters to the algorithm (called algorithm parameters), and the thresholds which define a good and bad result. Under Results Produced, one can see the output values of the algorithm used to test the histogram. If a histogram has failed a given test, addition information as to why the histogram failed may often be present in the Results Produced section.

The Red, Green, and Yellow Lights

When the relevant histograms arrive on the OHS, DQMF will automatically check the histograms, and publish a result. This result can either be red, yellow, or green. Not surprisingly, red means a histogram has failed a test, yellow means the histogram has not yet failed but needs to be watched, and green means everything appears OK. If a result is simply gray, it means either the histogram never arrived on the OHS, or the histogram failed some initial test and a result was never produced. One example of a failed initial test is a failure of a minimum statistics requirement. An error message would be produced and sent to the specified output (probably a log file, but again, a question for a DQMF expert, it depends on some settings). The color of the result can be seen in the tree on the left hand side of the display. The small circle on the left hand side of the histogram name will turn to the result color. The display is set to the worst case summary, meaning that if any 1 histogram produces a red result, the entire chamber registers a red result, then the sector, and so on all, the way up to MDT. Thus you can simply track the red or yellow lights starting at MDT region all the way down to the offending histogram.

What to do when things change color

The general rule is that if a histogram turns red, you should seek an MDT expert and notify them. If the result is yellow, you should keep and eye on that result (a yellow light does not warrant the notification of an expert). If everything is green, you are in good shape. But, you still need to look around and keep and eye out for gray results. For this reason its a good idea to keep an eye on the file/output where the errors are being printed. If you see a gray result, its best to find out why and asses if one needs to contact an expert. If no error message is produced, the histogram never arrived, and its best to wait a bit to see if the histogram will come a bit later. If there is an error message, it may not be of immediate concern. For instance, if there is not enough statistics, you may need to wait a bit longer for a proper result. If a histogram never arrives after a significant period of time, or if the error message seems to be bad (or worse, incomprehensible) its probably best to contact a DQMF or MDT expert.

So what are we actually checking here?

I will describe briefly what we are checking on each chamber to give you a better idea of what is going on. The TDC histograms give the TDC spectrum of the chambers. We perform a fit to this spectrum and ensure that the rise time and drift time are close to their expected values. Next we check the number of hits per tube, and ensure the shape of this distribution is close to the expected reference histogram. If more than 24 bins deviate by more than 4$\sigma$ from the reference, or any one bin deviates by more than 7$\sigma$, a red result is produced. The exact same check is performed for the Noise Frequency per tube. Finally, we check that the mean and RMS of the ADC spectrum and Hit Multiplicity spectrum are close to their expected values.

Documentation/In case something goes wrong

MDT Run Control Errors

During a run, you may see warning or error messages in the MRS window. Some are compiled here:

Application Severity Msg Id Message What to do
MDTEndcapC-TTC-RCD Warning ROSException Relationship not set in database: MemoryConfiguration relationship for channel Id 2 Harmless, though annoying
MDT-EC2-RCD Warning ROSException MROD-EC2-15-T26: DB/OKS mismatch CSM(1-6) 1 EIS1C06: TDC mask 3fff vs ffff in OKS This message can be caused by the removal from readout of a mezzanine (typically one mezzanine is removed), or by the swap of two readout fibers between chambers with a different number of mezzanines, or a plainly wrong OKS configuration (as in this case, where OKS says the chambers has 16 mezzanines, while the hardware database reports that the chamber has 14; the database wins).
MDT-EC4-RCD Warning ROSException MROD-EC4-15-T30 Input 1 EIS1C14: No Data From TDC(0-17): 14 15 , Mezz Mask 0c000 Mezzanines 14 and 15 are not sending data. Mezzanines are counted from 0. Check that these mezzanines do exist (EIS1C14 has only 14!). If the mezzanine exists and was included in data taking, absolutely report to experts the message, by posting an entry in the elog, complete with the full error message.
MDT-BC2-RCD Warning ROSException MROD-BC2-11-T11 Input 4 BMS2C08: 1 CSM Link errors (Parity/LDOWN) This type of error is more dramatic than it seems, because when a chamber stops providing data, the run halts, and the MDT system (in this case, MDT B) issues a busy signal. The shifter in charge of MDT should reinitialize the chamber, as soon as the current run is stopped and brought to the INIT state.

ELOG

Any time something abnormal happens during a run, you must make an ELOG entry (informations about the run, error messages, problems encountered...).

You can open it from:

General --> ATLAS Logbook

elog.jpg

For read-only access, the following credentials can be used:

Username: atlas

Password: insider

In order to enter information request a password to Federico.Zema@cernNOSPAMPLEASE.ch.

To submit new items you have to clik on:

* New * and then you have to select the system (Muon for MDT) and the type of post ( Information, Warning, Fatal, Documentation, To Do). Then put the subject of your message, the run number you are referring and you can write the message on the white space on the bottom of the page. The click on * submit* .

elog2.jpg

The Elog can be accessed from outside CERN (read-only access) at:

https://pcatdwww.cern.ch/elog/ATLAS/ATLAS/

and from inside the CERN network at:

https://pc-atlas-www.cern.ch/elog/ATLAS/ATLAS/

Check the documented status and problems of chambers

You can access the database of chambers via the web. This is particularly useful if you you want to know why some chambers are out from the run, or why they have some problems. If you find some discrepancy, please report to the experts, in order to check it.

How to open a web page from Point 1 Network

To open a web page from pc of Point 1 you need before to exit from network of point 1. Do the following steps:

  • Open a terminal with your account of Point 1
  • Type ssh -X atlasgw-exp
  • it will ask you the Hostname of the pc you want to login. Put lxplus
  • press enter in the next step
  • insert your username and password for lxplus machines
  • now from lxplus you can open web pages typing for example konqueror&

To access the MDT status webpage, open:

https://linat05.grid.umich.edu/MDTchambers

were are displayed all MDT chambers with different colors depending what is their status (OK, HV Problems, LV Problem, Gas Leak, DCS Problems, Readout Problems). You are not aloud to modify anything (it`s a task for experts), but you can check the situation of the chambers.

MDTChamberStatus.png

A windows will pop up with the schema of all the chambers of each technology. You can choos between BI, BE, BM, BO, EE, EI, EM, EO and you can see the general situation. You can also go in more detail clicking on a chamber and will appear the list of problems 9if any) and how they are been solved of if they are still pending.

MDTChamberStatus1.png

TGC Shift Manual

What TO DO WHEN ?

When you come to the Muon Shifter Desk

  • OPEN Control & Monitoring terminals (or they have been opened on Muon Shifter Desk)
  • HV / LV failure check via TGC_DCS
    • tell the problem to TGC Shifter OR write down in e-Log (at least)
    • the problematic channel is colored as "RED"
  • Trigger Rate Monitor
    • total rate (A+C) must be around 100Hz

At the beginning of Run

When the error happens due to TGC system ...

  • Case0 : During M6, at least ONE person stays at ACR, so contact him/her
    • 09:00 - 16:00 : Takuya Sugimoto
    • 16:00 - 24:00 : Yasuyuki Okumura
    • 00:00 - 09:00 : Yuta Takahashi
  • Case1 : run has been stopped due TGC BUSY
    • 1-1 ; Rx Overflow
      • issue Reset SSW : ...
  • Case2 :
  • Case3 : ... it will be documented more with case study ...

Persons to contact

Number Name Expertise
Shikma, Shlomit 165842 DCS
Daniel 162431 ROD
Makoto 163614 Electronics
Takuya 162817 Electronics
Masaya 163043 Power Supply
L1 Trigger SCR 77683 TGC shifters (temp)

Preparation

Open Control & Monitoring Terminals

DCS

Trigger Rate Monitor

Front-end Electronics

  • login to
    • pc-tgc-daq-eca-01.cern.ch
  • open terminals for operation
    • pc-tgc-daq-eca-01> sh ~tgc07/start_localDAQ.sh A
    • pc-tgc-daq-eca-01> sh ~tgc07/start_localDAQ.sh C
  • then, these terminals are appeared on each side
    • pc-tgc-daq-eca-01
    • pc-tgc-cci-eca-01 (to configure FE on A-side)
    • sbc-tgc-ttc-eca-01 (to configure TTC setting on A-side)
    • pc-tgc-cci-ecc-01 (to configure FE on C-side)
    • sbc-tgc-ttc-ecc-01 (to configure TTC setting on C-side)
    • sbc-tgc-local-daq-01 (to open FE Status Monitor)

Electronics Status Monitor

  • @ pc-tgc-daq-eca-01 (A-side) or pc-tgc-daq-ecc-01 (C-side)
    • > cd /atlas-home/0/tgc07/MileStoneRun/P3/script/runstate
    • source setup.sh
    • java runstate
    • -> panel is appeared

List of Commands to change the SYSTEM STATUS

BE CAREFUL , YOU SHOULD KNOW VERY WELL WHAT YOU ARE DOING

NEED TO TYPE UNDER THE SUPERVISION OF TGC ON-SITE SHIFTER

Configure PS-Boards

  • @ sbc-tgc-cci-eca-01 (A-side) or sbc-tgc-cci-ecc-01 (C-side)
  • > cd /atlas-home/0/tgc07/MileStoneRun/P3/script
  • perl config_A.pl [Sector Name (A09,A11,C10,C11) OR " all "] [ASD or SLB or Cosmic]
  • perl config_C.pl [Sector Name (A09,A11,C10,C11) OR " all "] [ASD or SLB or Cosmic]
    • as sector name, A09, A10, A11, C09, C10, C11 are available

Configure CTM Trigger Boards

  • @ sbc-tgc-cci-eca-01 or sbc-tgc-cci-ecc-01 (C-side)
  • > cd /atlas-home/0/tgc07/MileStoneRun/P3/script/TRGCom
  • > ./M5setup_A.sh ( for all A-side sectors )
  • > ./M5setup_C.sh ( for all C-side sectors )
    • if you want to configure one sector, ...
      • > ./matomete_config_TRG_PIT.M5 [Sector Name (A09 or A11 or C10 or C11)]
  • > perl matomete_mask_trg_M1.M4.pl [M5A or M5C]

Configure Trigger Condition (set SLB Mask2)

  • @ sbc-tgc-cci-eca-01 (A-side) or sbc-tgc-cci-ecc-01 (C-side)
  • > cd /atlas-home/0/tgc07/MileStoneRun/P3/script/
  • > perl mask2_config.pl [Trigger Type(-->c.f. usage)] [sector# (9 or a or b)]

Configure LVL1 Depth

  • @ sbc-tgc-cci-eca-01
  • > cd /atlas-home/0/tgc07/MileStoneRun/P3/script/
  • Set Depth [sector# (9 or a or b)] [depth] | grep Result

Change Gate Width

  • @ sbc-tgc-cci-eca-01
  • > cd /atlas-home/0/tgc07/Milestonerun/P3/script
  • > perl s_bcid_config.pl [width] [sector# (9 or a or b)] > tmp
  • > perl w_bcid_config.pl [width] [sector# (9 or a or b)] >> tmp
  • > Set Param C09 3 tmp | grep Re
    • Here " C09 " doesn't mean setting parameter for C09, ...

Configure TTC setting

  • @ sbc-tgc-ttc-eca-01 (A-side) or sbc-tgc-ttc-ecc-01 (C-side)
  • > cd /atlas-home/0/tgc07/Milestonerun/P3/script
  • > ./ManualTtc [0xff1(board address)] [W or R (Write or Rerad)] [Bchan/TpTrigOn do] ; Test Pulse ON
  • > ./ManualTtc [0xff1(board address)] [W or R (Write or Rerad)] [Bchan/TpTrigOff do] ; Test Pulse OFF
  • > ./ManualTtc [0xff1(board address)] [W or R (Write or Rerad)] [Bchan/PsReset do] ; Reset PS-Boards
  • > ./ManualTtc [0xff1(board address)] [W or R (Write or Rerad)] [Bchan/DcsReset do] ; Reset Dcs

-- EmilyThompson - 30 May 2008

Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r3 - 2008-06-01 - EmilyThompson
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback