DCS (Detector Control System)

Controls and Monitors the basic detector hardware (power supplies, HV, LV, ) and infrastructure (gas, cooling, racks, ) DCS handles non-event based detector data.

There are 2 main tools used by the shifter, the DCS FSM (Finite State Machine) User Interface and the DCS Alarm Screen. A general introduction and functionality overview can be found here.

What to do in case of a DCS Alarm: General Procedures (Muons)

In case of a DCS alarm , stick to the following rules:

  • Check for 1 or 2 minutes if the alarm disappears by itself; if yes, still mention it in your shift summary. If not,
  • Check the severity of the alarm (WARNING, ERROR , FATAL ) and check if there are specific instructions for this particular alarm by right-clicking on the alarm entry in the alarm screen and selecting Alarm Help. If a alarm help exists, follow the instructions given there. If not
    • if it is a float type value or parameter, eg temperature, fan speed or similar, check the recent history by right-clicking on the alarm item in the alarm screen and selecting Trend. If the value just fluctuates around the limit, it's enough to make a note in elog/your shift summary.
    • if it is a WARNING during the day you can call the expert on call if in doubt. If it's during the night, make an elog entry.
    • if it is a ERROR or FATAL call the expert on call, then document the alarm in elog.
  • Acknowledge any alarm in WENT state only when instructed by the expert, by clicking on the red exclamation mark in the alarm screen
  • Mask and alarm to temporarily remove it from the alarm screen while under investigation when instructed to do so by the expert.

Low Voltage Operations

Dealing with individual LV channel problems

The following procedure holds for individual LV channel problems for the different muon sub-detectors (for LV failure of a large part of a detector, the cause is normally either a DCS or DSS interlock action or a problem with 48V. Additional steps are needed this case, see below).

CSC
  • Try to switch a failed LV channel back on once. Document in elog. If the problem persists, call the expert.
  • Please note that a sector whose LV was off intermittently does not give any data until the next run where CSC should be reconfigured! Tell the shift leader.
MDT
  • Try to switch a failed LV channel back on once. Document in elog. If the problem persists, call the expert.
  • JTAG reinitialize affected chambers
  • if during a run, reinclude chambers into the run, they will have been dropped as consequence of the LV failure.
RPC
  • Identify which type of LV failed (Vee, Vth, Vpd or Vpad).
  • In case of Vpad
    • Vpad can be switched back on only when no run is ongoing, since it will cause 20kHz fake trigger rate otherwise. Tell the shift leader and trigger shifter that a RPC trigger tower is off and will stay off until end of run. Call the RPC DCS/detector expert.
  • In case of LV other than Vpad:
    • First do a "Clear Alarms" action on the power system CAEN mainframes, by navigating in the FSM to the RPC node. The right click on "RPC" to open the RPC secondary panel in the left bottom part of the FSM UI. Click on "Clear Alarms".
    • Then try to recover the failed LV executing the command "GOTO_READY" on the sector where the problem is. If this fails, call the RPC DCS/Detector expert on-call.
    • Document in elog.
TGC
  • Do not try to act on a LV problem yourself; call the TGC expert on-call.

LV Interlocks

High Voltage Operations

Switching HV ON/OFF

In order to Switch HV OFF, for an individual HV channel or a larger part of a muon sub-detector, navigate in the FSM to the corresponding node, then execute the command SWITCH_HV_OFF / POWER_HV_OFF for it.
You should switch HV off

  • when told to do so by an expert, e.g. in preparation for an intervention;
  • for individual channels if the channel is behaving abnormally, in particular when it is showing a persistent OverVoltage alarm or instable output voltage.

In order to Switch HV ON, for an individual HV channel or a larger part of a muon sub-detector, navigate in the FSM to the corresponding node, then execute the command SWITCH_HV_ON / POWER_HV_ON for it.
Please note:

  • You should only switch HV on when told to do so by an expert , except in case of MDT/RPC HV trips. For the procedure of dealing with HV trips see below.
  • CSC HV can only be switched ON if LV has is ON, the command will otherwise fail.
  • There are a number of situations in which HV is switched off automatically by DCS, or HV voltages lowered. See the section on HV Interlocks/DCS Interlocks later in this manual for details.

Transition between HV READY and STANDBY

During beam operations, all muon sub-detectors ramp HV to nominal values ("READY") only when stable beams has been declared by LHC, implying no more manipulations/adjustments are done on the beams. Outside stable beams, HV is set to a lower STANDBY value, which can be the same value for the full sub-detector (CSC, RPC) or depend on the distance from the interaction point of chamber (MDT, TGC). Stable beams is indicated to ATLAS both by the beam mode, displayed on LHC page one, and the so called stable beams flag which is a hardware signal sent to the experiments.
Current Standby settings are
System Standby Voltage (V0) Nominal Voltage (V1) Detector Region
CSC 1300V 1800 V  
MDT 2500V 3080V BI layer; BEE; EM1,2,3; EO1,2,3; EI1,2
MDT 3080V 3080V remaining chambers
RPC 9000V 9600V  
TGC 2200V 2800V EIFI and forwards (F) chambers
TGC 2800V 2800V remaining chambers

Understanding if the stable beams flag is present or not
Whether the stable beams flag (hardware signal) is present/received by Atlas or not can be seen from the DCS LHC Widget always displayed in the middle top of the DCS FSM Ui. The widget's last 3 lines give the presence/absence of the stable beams flag, the status of the overall Atlas injection permit sent to LHC, and the DCS evaluated 'safe for beam' status.

Automatic transitions between Standby and Ready
Normally, all ramping between Standby and Ready depending on beam mode is handled automatically by the DCS. There are 3 DCS initiated actions plus a transition which is directly hardware driven.
  • Injection Handshake (DCS action): Upon reception of the Injection Handshake message from LHC, indicated in the control room by an audible signal, all muon sub-detector HV is ramped to STANDBY values if voltages were still at READY.
  • Adjust Handshake (DCS action): Upon reception of the Adjust Handshake message from LHC, indicated in the control room by an audible signal, all muon sub-detector HV is ramped to STANDBY values if at READY before. This transition is only relevant when going to "Adjust" from "Stable Beams", during the Adjust phase following the ramp and a squeeze detectors are still at Standby anyway.
  • Stable Beams (DCS action): Upon reaching stable beams after the sequence RAMP - FLAT TOP - SQUEEZE - ADJUST muon sub-detector HV is ramped to nominal voltages. The ramp up takes between 30 secs and a few minutes.
  • Loss of Stable Beam Flag, Beam Dump (Hardware action except for RPC): CSC, MDT and TGC ramp down to STANDBY voltages on the loss of the stable beams flag/signal. This is a direct coupling to the hardware signal received from LHC, and thus works even if there is a fault in DCS controls. Please note that on a sudden, unscheduled beam loss (beam abort) from stable beams the ramp down usually occurs a few minutes after the actual beam loss, when LHC operators reset the stable beams flag; in case of a scheduled dump the ramp down happens before the actual beam dump.
  • Loss of Stable Beam Flag, Beam Dump - RPC: The RPC situation is slightly different from the other muon detectors, here HV are kept at READY after a beam loss or dump (from stable beams!) for another 20 minutes to allow the study of after-glow effects; HV is ramped to STANDBY automatically either after the delay period or on the next injection handshake should it happen earlier.

Checking Status of Automatic Actions
You can check if all or some of the automatic actions are enabled or disabled by clicking on the MUON FSM top panel on the 'Advanced Panels' drop down menu, then select 'Common' and from there 'Autom. Beam Actions', which opens the panel shown here. Tick marks indicate if automatic actions are active or not. In case of an malfunction of the automatic actions, shifters can disable them (all together) by clicking on the 'Disable All Actions' button, then report this to the experts/system coordinator.

Manually going from STANDBY to READY
In case automatic transitions are not enabled, or fail, shifters can go to READY manually, if the stable beam flag is present or override mode is selected. To do so, execute in the FSM
  • the command GOTO_HV_READY on the MUON SYSTEMS top node of the Muon FSM tree; or
  • the commands GOTO_HV_READY on the MDT and TGC nodes plus the command GOTO_READY on the RPC node. CSC are coupled to MDTs and thus no command is needed here.

Manually going from READY to STANDBY
In case automatic transitions are not enabled, or fail, shifters can go to STANDBY manually. To do so, execute in the FSM
  • the command GOTO_HV_STANDBY on the MUON SYSTEMS top node of the Muon FSM tree; or
  • the commands GOTO_HV_STANDBY on the MDT and TGC nodes plus the command GOTO_STANDBY on the RPC node. CSC are coupled to MDTs and thus no command is needed here.

Please note that neither GOTO_READY/GOTO_HV_READY nor the corresponding Standby commands will switch on any HV channel which is OFF before. For this please refer to the instructions on HV turn on/off.

HV Override mode (allow nominal HV without stable beams)
For details on the override mode, please see the paragraph later in this manual.

HV Interlocks

Muon sub-detector HV, if in OFF state, is prevented from being turned on under certain conditions:

  • CSC HV interlocked in case of no LV: CSC HV is blocked from being turned on if LV for the sector is off.
  • CSC/MDT manually interlocked channels: CSC and MDT HV channels can be manually interlocked by the expert. In this case, the information "disabled" (MDT) or "interlocked" will appear on the FSM HV channel panel. In the Power System FSM panels, disabled channels are indicated by a little square around the status indicator. If you find any manually disabled channels not listed on the whiteboard, check with the expert !

  • MDT channels interlocked due to gas conditions: MDT HV is blocked if either the CO2 concentration, gas flow or gas pressure is not ok, or if there is no information from the gas system for an extended period of time. In the HV channel FSM panel the word "gasInterlock" will be displayed in this case.
  • MDT channels interlocked due to excessive HV board temperature: MDT HV is blocked if a board reports an abnormally high temperature. The word "tempInterlock" is displayed in the HV channel panel in this case. If you encounter this case, call the MDT/MDT DCS expert !
  • RPC voltage set points reduced due to gas conditions: RPC HV set points are set to a lower than normal safe value of 5000V in case of problems with gas flow or mixture. The safe voltage will continue to be enforced for some time, depending on the duration of bad gas conditions, after the gas problem clears !
  • TGC channels in Manual or Disabled Mode: Channels are blocked from being turned on if configured by the expert as in mode Manual or Disabled. The mode can be seen by navigating in the TGC FSM to the corresponding HV channel, an example is shown below. Channels in manual mode and OFF should normally be excluded (disabled) from the FSM tree.

In addition, HV when ON is automatically switched off for detector safety reasons when one of the non manual conditions above is reached. More details are given in the section on DCS Interlocks later in this manual. If you find any HV going to OFF state without a Trip or other Error, check if it was caused by an interlock action before you call the expert.

Dealing with HV Trips

The following procedure holds w.r.t. to HV trips for the different muon sub-detectors:

CSC
  • Tripped HV channels will be recovered automatically by DCS, no shifter action required.
  • Only in case you observe the same channel re-tripping constantly, please alert/call the expert.
MDT
  • Try to switch a tripped channel back on once;
  • If the channel trips again, leave it off; try to clear the trip alarm with the RESET_TRIP command in the FSM on the affected channel; if this does not work, you can mask the alarm from the alarm screen (right click --> Mask).
  • Disable the node in the FSM (by clicking on the the red cross next to it).
  • Post an elog entry. There is no need to call an expert for single channels tripping.
RPC
  • Do not act on a tripped channel yourself. call the RPC DCS/Detector expert on call.
TGC
TGC HV trips occur relatively often.
  • Recovery is automatic, you should not switch tripped channels back on yourself.
  • The automatic procedure attempts to recover a tripped channel up to 4 times; please note that there is a certain time (currently 10 minutes) waited between trip and recovery attempt; the channel appears in state "RECOVERING" in the FSM during this period.
  • You can check the number of automatic recovery attempts carried out so far from the table which appears in the FSM UI when navigating to the corresponding TGC sector wheel node (e.g. HV C 01 M3); look at the "#tr" column.
  • In case recovery fails for the 4th time, disable the channel node in the FSM and mask the alarm from the alarm screen (this does currently not work reliably for the summary alarm).
  • Post a separate entry to elog, channels will be looked at by the expert.

Dealing with other HV Errors

Threshold Settings

Front-end electronics thresholds are controlled by DCS for MDT, RPC and TGC.

MDT
MDT thresholds are loaded to the mezzanine card ASD chips as part of JTAG initialization. Threshold values are by default taken from the MDT configuration data base, the are different for each ASD chip. In addition thresholds for each chamber can be set to a user defined value via the JTAG FSM tree, for special calibration runs. When initialized in this mode, WARNING status is present in the FSM and 2 alarms appear in the alarm screen, one indicating custom threshold mode is selected, the second that the chamber is indeed initialized with the custom threshold and not the values from the configuration DB. Please make sure default thresholds from the database are reloaded before the next physics run if not explicitly instructed otherwise!.

RPC
RPC thresholds are generated by ADC modules which are part of the RPC LV system. Threshold voltages are referred to as Vth, trouble shooting is covered under Low Voltage Operations.

TGC
TGC front-end electronics thresholds are handled via TGC ELMBs (Embedded Local Monitroing Boards), which in turn are controlled from DCS via CanBus.
  • Threshold not responding: In case of alarms of type "TGC C THRESHOLD M1 sector08 phi1 E2 L2 thresholdw05 not responding" please execute the command REFRESH_CHANNELS on the relevant THRESHOLD_SECTOR in the TGC FSM tree. Only if this does not clear the problem call the expert.
  • Bad thresholds after LV power cycle: After any TGC LV operation thresholds may have to be reset to work correctly, please check with the expert immediately should threshold errors appear after a known operation on LV.

JTAG Initialization (MDT)

JTAG initialization is the loading of parameters to the MDT front end electronics (CSM and mezzanine cards) and their initialization. JTAG init is handled through the MDT FSM tree.

Dealing with lost JTAG initialization

JTAG initialization can be lost sponteneously in case of

  • loss of chamber low voltage
  • problems/glitches with the clock signal sent to the front end electronics

In case JTAG initialization is lost (nodes being NOT_READY/NOT_INITIALIZED ) in the FSM,

  • Reinitialize affected chambers. You can try several times if needed.
  • If a run is ongoing, once chambers are reinitialized make sure you reinclude them into data taking, in the same way as when recovering dropped chambers.
  • If reinitialization fails, make sure there is no LV problem nor problem with a MDT TTC crate being off.
  • In case reinit persistently fails, call the expert on-call.

The given procedure is for the case of JTAG init lost for some chambers. Should you loose a large part of the detector, in any case check with the expert before you reinclude them into an ongoing run!

Muon BIS (Beam Injection System) Interface. Injection Permit Logic.

The Muon BIS Interface handles the signals provided to LHC (injection permit, via Atlas BIS system) from muons and the signals received by muons from LHC (stable beams flag). Details on the status can be seen from the Beam Interlock Panel, which can be opened by navigating in the FSM to MUON SYSTEMS (FSM top node) --> MUON --> BEAM INTERLOCK.

The panel is spit in 2 parts (shown here: left -- no stable beams, STANDBY. right: stable beams, HV READY.

  • Injection Permit: The upper part of the Beam Interlock panel shows the status of the muon injection permit logic. The injection permit is a hardware signal common for all muon sub-detectors which must be provided to LHC, via the ATLAS BIS system, to allow injection. This signal is generated by the Muon BIS system when all HV is at STANDBY and muon HV is thus in a safe state. The signal is present, if the indicators circled in purple all appear in blue. The overall muon permit, as received by the Atlas BIS system, is shown in the right part circled in yellow. Should this be not the case when at STANDBY, please call the relevant sub-detector DCS expert straight away, you are holding up LHC injection.

  • Stable Beams Signal Handling: The lower part of the Beam Interlock panels shows handling of the stable beams signal. Please note that the signal from the LHC is not directly connected to muons' CAEN power system mainframes, but via a set of DCS controlled switches shown here. This allows to withheld the stable beams signal from the mainframe to go to STANDBY voltage (V0 set points) e.g. in case of the Adjust handshake by opening the switches.
    The shown situation is the one during beam operations without stable beams (HV Standby). When stable beams are reached, LHC Stable Beams Signal will become TRUE. On the GOTO_READY(_HV) command the 2 switches to the right for MDT and TGC are closed (will become red), the signal then reaches the mainframes which switch from V0 to V1 set points. The RPC switch is always closed since HV transitions are handled slightly different as described above.

HV Override Mode: Nominal HV without stable beams

Muons sometimes require nominal HV without stable beams, in particular in periods with no beam at all for an extended period of time (at least a few hours). Activating the override mode is an action reserved to the muon run coordinators and protected by corresponding access control. Please call the muon shift phione number in case any question on going to override mode comes up.
Please note that while either CSC/MDT or TGC are in override mode is active, the muon injection permit is blocked!.

Asserting Override Mode (Muon run coordinator instructions!)

To assert the override mode, navigate in the FSM to MUON SYSTEMS (FSM top node) --> MUON --> BEAM INTERLOCK. Click on the State field (reading 'READY') of the Beam Interlock node and execute the command GOTO_OVERRIDE_MODE. Override mode can be set also separately for CSC/MDT, RPC and TGC if needed, for this execute the same action separately on the sub-detector specific nodes.
Please note that activating override mode is a protected action reserved to the muon run coordinators!
Please note that while either CSC/MDT or TGC are in override mode is active, the muon injection permit is blocked!.

Clearing Override Mode (Shifter instructions)

The override mode can be cleared, ie the system made ready for beam injection by the muon shifter. To do so, please first put the HV to Standby as described above. Then in the FSM navigate to the Beam Interlock node, click on its state field (showing 'READY') and execute the command GOTO_BEAM_MODE. Verify the injection permit is given, this may take a minute.

DCS and DSS Interlocks

Gas System

Barrel Alignment System

The overall status of the barrel alignment system is monitored as part of the MDT FSM. In case of its state becoming NOT_READY or its status going to ERROR or FATAL, please call the MDT/MDT DCS expert. Intermittent Warnings of a high bad line fraction of a few percent can be ignored if they disappear after a few minutes by themselves.
If a NOT_READY state is accompagnied by a RasDim Server Alarm, please check with the shift leader if he/she know about any check carried out by an alignment expert from remote, if yes, wait before calling the MDT/MDT DCS expert, normally the alarm will clear itself in this case.

Endcap Alignment System

In rare cases the data acquisition cycle of the end cap alignment system can run into a situation where multiplexers are no longer responding; in such cases it is important to turn off the VME crate housing the multiplexer to ensure devices are not stuck in a constant-ON state which can damage them. The check for multiplexer responsiveness is done by the alignment DCS, which will trigger an automatic power off the all end cap alignment crates if a device is found no longer responding.

An alarm MDT EAL Status. Turn off VMECrates: Hardware At Risk will appear on the DCS alarm screen when devices are no longer responding. In this case, please

  • Check that the endcap alignment crates have indeed been automatically turned off. To do so, navigate in the FSM to MDT --> Infrastructure --> VME Crates --> Alignment.
  • If the automatic switch off of crates has failed, switch them off manually from the FSM (execute the GOTO_SHUTDOWN command on the VME Crates --> Alignment node. Please be extra careful not to give the command accidentally on the full VME crates node or higher up in the tree, since this will hang up any data taking)
  • Once crates have been turned off, either manually or automatically,
    • acknowledge the alarm, it will then disappear
    • re-power the alignment crates after a few minutes, to get the MDT Infrastructure state back to READY in the FSM
    • post an elog entry. The alignment experts will have received an email automatically and take care of recovering the alignment sequence, usually within a few hours.

VME (ROD, Alignment, TTC) Crates

  • DCS.DCSAlarm.pag8.PNG:
    DCS.DCSAlarm.pag8.PNG
Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2011-04-03 - MarcellaCapua
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback