DAQ

Taking or Monitoring a Run

When a combined run is ongoing, you will monitor the ATLAS partition, spying on errors, log messages and the state of the DAQ. In addition, during calibration periods you will run calibrations/tests in standalone mode. Standalone runs currently are separate for each muon sub-detector (CSC, MDT, RPC, TGC).

Monitoring the ATLAS Partition

In order to monitor what is ongoing in the ATLAS partition, click in the menu on TDAQ --> DAQ Panel.

DAQ Panel default settings for tdaq-03-00-01

Parameter Config
Setup script /det/tdaq/scripts/setup_TDAQ.sh
Database File /atlas/oks/tdaq-03-00-01/combined/partitions/ATLAS.data.xml
Partition Name ATLAS
MRS Filter CSC | MDT | RPC | *TGC*
OHP Config Parameters -c /atlas/moncfg/tdaq-03-00-01/muons/ohp/muon.ohp.xml
TRP Config Parameters -c /atlas/moncfg/tdaq-03-00-01/trigger/trp/trp_gui_conf.xml

After opening the DAQ Panel,

  • Enter the setup script and database file. You can also use the Browse button to navigate to the specified files.
  • Press Get Partition.
  • Select the partition from the drop down menu.
  • Press Read Info. Wait a little until information is read.
  • Once the buttons on the right of the DAQ panel become active, click on Monitor Partition and wait until the TDAQ IGUI appears. This may take half a minute or so.
  • Once the ATLAS partition is up, load the TGC Integrated Panel into the the TDAQ IGUI, by selecting it from the Load Panels menu in the TDAQ IGUI.

Starting a Standalone Run - General Procedure

To start a standalone run, open the DAQ panel, then
  • Enter the setup script and database file. You can also use the Browse button to navigate to the specified files. Please refer below for the configurations needed standalone.
  • Press Get Partition.
  • Select the partition from the drop down menu.
  • Press Read Info. Wait a little until information is read.
  • Once the buttons on the right of the DAQ panel become active, click on Start Partition and wait until the TDAQ IGUI appears. This may take half a minute or so.
  • Press Boot, wait for partition state to become BOOTED.
  • Press Initialize, wait for partition state to become INITIALIZED.
  • Press Configure, wait for partition state to become CONNECTED.
  • Check the run settings, in particular run type and recording enabled/disabled. if you do not want to limit the number of events taken, put 0 in the corresponding field. For standalone runs recording should normally be disabled except instructed otherwise .
  • Press Start to start the run and check there are triggers.

CSC calibration (pedestal) runs

Shifters should take a CSC pedestal run once per day.

DAQ Panel CSC pedestal run settings (tdaq-03-00-01)

Parameter Config
Setup script /det/muon/standalone/scripts/setup-03-00-01.sh
Database File /det/muon/standalone/databases/tdaq-03-00-01/muons/partitions/part_CSC_ped.data.xml
Partition Name part_CSC_ped

Pedestal run sequence:

  • Start the CSC pedestal run partition from the DAQ panel as described above.
  • Press Boot, Initialize, Configure. Configuration will take about 2 minutes, subscribe to Information in the MRS window if you want to get status updates for each ROD.
  • The number of events should already be set to 10000. Press Start and make an ELOG entry with the run condition. The run will take 2 minutes.
  • At the end of the run, press Stop, Unconfigure, Terminate and Shutdown. Exit the IGUI and answer 'yes' to the popup window to shut down the partition. It is important to shut down the partition to free its resources for the next combined run.

MDT standalone runs

Currently there are no MDT calibration runs shifters perform on a regular (daily) basis; shifters are however asked to do a standalone run as part of recovering from a power cut, for specific investigations as specified by the experts and when there are interventions in the cavern (replacement of chamber electronics cards etc.).

DAQ Panel default settings for tdaq-03-00-01

Parameter Config
Setup script /det/muon/standalone/setup-03-00-01.sh
Database Files /det/muon/standalone/databases/tdaq-03-00-01/muons/partitions/part_MDT_all.data.xml
/det/muon/standalone/databases/tdaq-03-00-01/muons/partitions/part_MDT_Ba.data.xml
/det/muon/standalone/databases/tdaq-03-00-01/muons/partitions/part_MDT_Ec.data.xml
Partition Names part_MDT_all
part_MDT_Ba
part_MDT_Ec
MRS Filter MDT
OHP Config Parameters -c /atlas/moncfg/tdaq-03-00-01/muons/ohp/mdt/MDT.ohp.xml
TRP Config Parameters -c /atlas/moncfg/tdaq-03-00-01/trigger/trp/trp_gui_conf.xml

MDT standalone partitions:
A MDT standalone run can be done either for the full MDT system or separately for the Endcap and the Barrel. The 3 different partitions to be used are

  • part_MDT_Ec: MDT Endcap
  • part_MDT_Ba: MDT Barrel
  • part_MDT_all: Full MDT system
Endcap and Barrel partitions can run simultaneously when needed.

MDT test pulse runs

ToDo: add instructions here

RPC standalone runs

Currently shifters do not perform RPC standalone runs on a regular (daily) basis.

DAQ Panel RPC standalone settings (tdaq-03-00-01)

Parameter Config
Setup script /det/muon/standalone/scripts/setup_partRPC_TDAQ.sh
Database File /atlas/oks/tdaq-03-00-01/muons/partitions/part-RPC.data.xml
Partition Name part-RPC
MRS Filter RPC
OHP Config Parameters -c /atlas/moncfg/tdaq-03-00-01/muons/ohp/rpc/RPC.ohp.xml
TRP Config Parameters -c /atlas/moncfg/tdaq-03-00-01/trigger/trp/trp_gui_conf.xml

TGC calibration runs

Muon shifters are asked to perform 3 types of standalone runs during calibration periods,
  • random : random trigger run.
  • ASD : analog test pulse run.
  • Track : digital test pulse run. Please inform the shift leader when the calibration is completed.

DAQ Panel TGC calibration run settings (tdaq-03-00-01)

Parameter Config
Setup script /det/muon/TGCFE/installed/bin/setup_partTGC_FillTest.sh
Database File /det/muon/standalone/databases/tdaq-03-00-01/muons/partitions/part_TGC_FillTest.data.xml
Partition Name part_TGC_FillTest

Calibration run sequence:

  • First, prepare DAQ software.
    • DAQ panel for standalone run. For standalone run, open "TDAQ" panel for calibration test named "part_TGC_FillTest". To do that, fill some information on "DAQ Panel".
      • Setup Script : as listed above.
      • Database File : as listed above.
    • Click on "Read Info". The Part name "part TGC FillTest" is filled automatically.
    • Click on "Get Partition".
    • Click on "Start Partition".
    • Now, new TDAQ panel is opened. The name is not "ATLAS" but "part_TGC_FillTest".

  • Next, open "FEtest" panel
    • From menu bar: Muon->DAQ/Config->TGC FE Test . The panel which will appear is shown here.
  • Finally, start data taking.
    • (1) Before start, ensure that TDAQ state says "None". If not, click "SHUTDOWN ".
    • (2) Click "random" on the FEtest panel (or "Track" or "ASD" for the other two calibration runs).
    • (3) TDAQ transition will show a popup window (message for e-Log, use your login and password). Fill information like the screen shot and click OK. Now data taking will be started automatically. TDAQ state changes "None" -> "BOOTED" -> "CONFIGURED" -> ... automatically. If transition failed (some box turned red like this), click "stop test and shutdown DAQ automatically" button on FEtest panel and go back to step (1). At the end of the calibration run, another popup window appears with reason for end of run for e-Log.
    • (4) When TDAQ is "CONNECTED", TDAQ always shows a warning message. Just click OK and wait for a few minutes more because FEtest panel is still configuring electronics modules. After the configuration is done, run automatically starts. If nothing happens after 5 min, something is wrong. stop the test by clicking "SHUTDOWN"and go back to the step (1) above. At the end of the calibration run, another popup window appears for end of run message in e-Log, please press "Ok|" here.
      • If Rod busy occurred, follow the same instruction.
      • Always keep your eye on the Rod Status using the TGC Status panel, which is opened from the TDAQ IGUI by clicking on the "Load Panels" Button and selecting "TGCI Integrated Panel".
      • Data taking will take about 15 mins.
      • TDAQ will automatically go to "NONE" after data taking, which means data taking for "random" finished.
    • (5) Then check run number, *report it to on-call expert immediately*(if it is midnight, you do not. But do not forget to write it down on e-Log.).
    • (6) Data need to be copied to specific directory, which can be done by the FEtest panel, bottom part. Fill in the run number and click "copy to share disk".
    • (7) Go to instruction (1) and do the same thing for "ASD" and "Track" instead of "random".
    • (8) Having finished all three tests, close TDAQ Panel and FEtest Panel. When you close the TDAQ IGUI, you will be asked whether to also shut down the partition infrastructure. Answer "yes" here.
    • (9) Finally, the three calibration run numbers should be written in an e-Log message.
  • If ....
    • Some tests have not been finished yet but shift leader told me that global run will start soon.
      • If you do not have enough time, give up taking data and report it on e-LOG. To finish the testm click "stop test and shutdown DAQ automatically" button on FEtest panel, and after TDAQ goes to "NONE", close TDAQ Panel and FEtest Panel. Then, tell shift leader thet TGC is ready for global run.
    • Waiting for "FEtest" for a long time but it just continues saying "TGCFE-RCD_ECA01_FillTest is still working...".
      • Stop the data taking and try it from scratch. Close FEtest panel and open it again. Click "SHUTDOWN" on TDAQ panel and wait until it goes "NONE" state.
    • Other problem
      • Call on-call expert

Other Monitoring Tools for the Ongoing Run

MRS messages are supplemented by several specific tools and panels the muon shifter is asked to monitor to spot problems in the muon DAQ, data flow and readout.

RPC Busy and Rate Panel

The "RPC Busy and Rate Panel" shows RPC LVL1 DAQ information obtained through DAQ to DCS Communication (DDC) and is started from the muon menu MUON --> RPC --> INFRASTRUCTURE --> RPC LVL1 --> RPC LVL1 DDC. The same panel can be opened from the MUON or RPC secondary panel clicking on the RPC DDC box or as a standalone independent window by clicking on the neighbouring small circle.

  • What is displayed:
    • The individual trigger rates of the ~400 trigger towers and their busy and kill status
    • The total rate and the number of trigger towers with problems (busy, kill etc.)
    • Information from the ongoing run (Run number, lumi block, status running etc)
    • Overall READY state; OK, WARNING AND ERROR messages

In the main table the rates are organized in 64 lines corresponding to the 64 trigger sectors named in the leftmost column. The corresponding detector sectors (16 in total) is indicated in the rightmost column. Individual rates for all trigger towers are shown together with their sum and a bit pattern indicating error conditions (#).

NOTE: Please note that this panel is active and displays valid information only when the RPC is in data taking (combined or standalone). If neither a combined or standalone run is not on-going the information is invalid (grey) and not relevant. In this case a WARNING message is issued which does not need a follow-up.

  • What to monitor:
    • During a run in the ATLAS partition, shifters should periodically monitor the status of all sectors in the main big table.
    • The title should be green "DAQ to DCS Communication Active". The rates displayed in the table should be mostly green with very few exceptions (*). Grey cells indicate disabled towers and should be monitored. A high rate (>1000KHz) in one or more trigger towers or more than 5 errors (#) are considered error conditions shown in the title of the panel and reported also in the ALARM screen. In case of such error call anytime the RPC LVL1 expert.
    • Errors and high rate are also summarized in the bottom line of the panel. In case of occasional high rate and/or a number between 1 and 5 trigger towers killed or busy call the RPC LVL1 expert (during daytime) or add an elog entry addressed to RPC DCS (during night time).

  • Additonal information:
    • The panel allows to visualize history trends of rates, busy and error condition by right/left click on the appropriate table element.
    • The table can be reordered to visualize highers rates, errors, trigger sector logic or detector
    • DB queries and snapshots on archived data are disclosed by clicking on the Online box in the top title.
    • By clicking on the main table border the information displayed flips from Meanrate (last 2 minutes) to instantaneous rate (last measurement)
    • (*) Please note that towers t4 of sectors s21, s22, s25, s26, s53, s54, s57, s58 (corresponding to detector the ATLAS feet sectors 12 and 14) zero rate (blue) is expected.
    • (#) Busy/Killed towers are summarized in red in the "Killed" column of the main table. A bit pattern is associated to each trigger tower. 0 means no errors. 1 Kill, 2 Pad Busy, 4 SL Board Busy, 8 Rx Fifo Busy. More than 5 pads in error (Busy/Kill non zero) are considered an error to be followed up anytime.

TGC FE Monitor

The TGC FE Monitor panel is started from the muon menu MUON --> DAQ/Config --> TGC FE Monitor and shown here.
What to monitor:
During a run in the ATLAS partition, shifters should monitor the status of all sectors in the left hand part of the panel. The state for all sectors should be either OK or CHECKING , where Checking indicates the sector is currently being tested by the tool.

*What to do in case of Errors and Problems: In case of an ERROR reported in the TGC FE Monitor, please call the TGC expert on-call !

TGC Status Panel

The TGC Status panel has been refurbished in March 2011. From the standard RC panel, open the TGC igui by clicking Load Panels and selecting the TGC panels. In the copious tree you are presented, you should open the shifter subtree. All sub-panels there are important and you should understand the reason for any red entry. You can of course browse the other panels; they are harmless except the Configuration section which should not be touched by a non-expert.

  • The top most panel (ROD status) shows a simple summary of the 24 sectors. The most important tag is the ROD busy state.
  • Then come the Recovery/Resync panel. Once you know that Resync means `transparent reconfiguration of the ROD and flushing of SSW pipes' and that Recovery means `procedure to be used if a ROD is stuck', it should be simple. If not, either roll the mouse pointer over the buttons in the panel to get online tips, or follow this link.
  • The third most important Panel in the Occupancy one. It shows, for each of the 7 or 8 star switches in each rod, the averaged wire and strip occupancy. A black box indicates that a star switch has been dropped. If the automatic resync mode is not enabled, please refer to the dedicated page.
  • Then comes the summaries of HV and Thresholds read out monitoring, as written to the condition data base. Disabled indicates a channel which isn't controled by the DCS. The age of newest update is an indicator of the welath to the process writing to the data base: if stuck, all values will increase for ever. Note that you can produce a formatted detailed print-out if these panels from LXPLUS or any other /afs/cern.ch mounted machine. See the detailed instructions.
  • Finally the last panel shows a summary of the DQ flags, as computed by the DCS calculatorAll sub-panels there are important and you should understand the reason any red entry.

Busy Panel

The BUSY panel is started from the DAQ panel pressing on the button Busy, which opens a panel as shown here. The panel is usually also among the ones projected in the control room on the front wall so that you don't need to have it open on the muon desk. In case of a TGC problem, see the new instructions for TGC ROD recovering in the dedicated page..

ISPY Panel

The ISPY Panel is used to check the status of the muon calibration stream as explained here. The panel does not need to be kept open otherwise on the muon desk.

Dealing with Stopless Removal Pop-Up

Recovery Procedures for Dropped/Abandoned/Killed/Removed objects

CSC

  • There is no recovery procedure for CSC objects stoplessly removed during a run.
  • If part of CSCs was stoplessly removed during a run due to a Busy, the CSC segment must be restarted before the next start of run or alternatively CSCs unconfigured/configured.

MDT

Dropped Chambers
You notice chambers being dropped from a run by
  • an alarm in the DCS alarm screen, giving the number of currently dropped chambers per partition
  • a warning message in MRS stating a CSM channel was disabled and the chamber name.
To recover a dropped chamber, while a run is in progress,
  • in the DCS FSM UI navigate on the Muon Systems top panel. Click on "Advanced panels" and select MDT. Then open the "DAQ/Run Status" Panel.
  • Dropped chambers appear in orange. Click on the chamber shape to open an action menu and issue the command SEND_REINCLUDE_REQUEST.
  • If the chamber is dropped immediately again, JTAG REINITIALIZE it, then do another SEND_REINCLUDE_REQUEST.
  • Alternatively to issuing the commands from the DAQ/Run Status panel you can also do so from the JTAG FSM panel of the affected chamber.
Dropped Mezzanines
You notice mezzanines being dropped from a run by
  • an alarm in the DCS alarm screen, giving the number of chambers with dropped mezzanines per partition
  • a warning message in MRS stating one or more mezzanines, given as a mezz mask, of a chamber were disabled
To "recover" dropped mezzanines
  • Note that there is no direct recovery for individual mezzanines cards, recovery is done by forcing the whole chamber to drop and then recovering it. To do so
* in the DCS FSM UI navigate on the Muon Systems top panel. Click on "Advanced panels" and select MDT. Then open the "DAQ/Run Status" Panel.
  • Chambers with dropped mezzanines appear in yellow. Click on the chamber shape to open an action menu and issue the command REINITIALIZE. This will cause the chamber to be dropped.
  • Issue the command SEND_REINCLUDE_REQUEST.
  • Alternatively to issuing the commands from the DAQ/Run Status panel you can also do so from the JTAG FSM panel of the affected chamber.

Important Note: In case there was a stopless removal done for one of the MDT MROD crates (quadrant of a partition, e.g. BA01), all connected chambers appear with all their mezzanines dropped in the DAQ/Run Status panel. In this case there is no possibility of recovery, the described procedure fails. Remember that it is adamant that the MDT segment is restarted/reconfigured before next start of run after any stopless removal!

RPC

TGC

  • TGC recovery for busy RODs is automatic. In case the automatic recovery fails, do a manual recovery as described in the section on the TgcStatusPanel.
  • TGC recovery for abandoned star switches (SSWs) is not yet automatic, do a manual recovery as described in the section on the TgcStatusPanel.

Changing the Run Configuration: In- and Excluding Detector Parts

Major changes to the which parts of muons are in- or excluded in a run are done by the experts. Shifters may however occasionally be asked to in/exclude individual chambers. Changes to the detector configuration should always be discussed with the expert first. Shifters may further find the following useful to check the detector configuration if there are doubts.

MDT

Disabling/Enabling Chambers
To enable/disable a chamber in the DAQ, or to check which chambers are currently included, start the MDTControl Panel via the Muon --> DAQ/Config --> MDT Control Panel menu item. Before the panel opens, you are asked to select the partition to check or modify, e.g. ATLAS, part_MDT_all, part_MDT_Ba, ... .

In the MDT Control Panel chambers (precisely, each MROD channels) are represented by buttons which are displayed in white if a chamber is excluded from and in light grey if a chamber is included in the selected partition.

The Control Panel displays a MROD crate at a given time, navigate between the different crates with the Select Crate drop down menu in the middle bottom. (Note that sectors 1 to 4 belong to crate XX01, e.g. BA01, sectors 5-8 to crate XX02, sectors 9-12 to crate XX03 and sectors 13-16 to crate XX04).

MDT Control Panel

Including/excluding chambers: To change the configuration,

  • You first have to switch from "Safe Mode" to "Include/exclude chambers" mode in the Options field.
  • Next, click on the chamber(s) you want to disable/enable. Chambers edited appear afterwards with a yellow border.
  • You can discard all changes and start over from scratch pressing the "Refresh" button.
  • Once sure changes entered are the correct ones, post an entry to elog specifying the changes you did and update the muon whiteboard accordingly.
  • Last, press the "Commit" button to submit your changes, enter you user name when asked to do so.

If you modified the ATLAS partition, please remember to tell the run control shifter to reload the OKS database before the next start of run for your changes to take affect.
If you modified a standalone partition, and the partition is already up go state INITIAL (via the "Unconfigure" action), the reload the OKS database by clicking on the button "Commit and Reload" in the top left part of the TDAQ IGUI panel.

Disabling/Enabling Mezzanine Cards
Disabling/Enabling a MROD
Disabling/Enabling a MROD crate (experts)

Muon Calibration Stream

The calibration stream is responsible for extracting muon segment data from the event data flow and sending it to the muon calibration centers in Munich, Rome and Michigan. The calibration stream relies on selecting muon segments in the L2 trigger using muFast as algorithm when running in the ATLAS partition.

Checking the Muon calibration Stream is Active:
To check that the calibration stream is active,

  • Open the ISPY panel (shown below) from the DAQ panel
  • Check the run condition: During proton collision physics, the calibration stream is only active when at stable beams !
  • On the ISPY panel, which is shown here, check the 2 entries "MuCalReader" and "MuCalServer"; both should indicate data traffic, ie values different from zero.

Troubleshooting:
If the ISPY panel indicates there is no data flow on the calibration stream while at stable beams(!!),

  • Check the trigger rate of MU10 triggers in the Trigger Presenter. The calibration stream requires muon trigger chains to be active at L2, it is seeded from MU10, so if MU10 is turned off/vetoed there are not event on the calibration stream.
  • Check the TDAQ Run Control tree in the TDAQ IGUI panel. Under TDAQ --> L2 Segment-1 "MuCalServerController" must be running and below it the nodes "MuCalServer", "MDTCalib" and "CSCCalib". The last 2 are responsible for producing monitoring histograms from the calibration stream data.
  • If you find the MuCalServer nodes disabled in the partition, refer to the run control shifter to have understood why and have this cured. Otherwise, call the MDT DAQ expert on call for follow-up and make a separate elog entry stating that the calibration stream appears to be down.

MRS Messages/Errors at Run Start

MRS Messages/Errors During a Run

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r1 - 2011-04-03 - MarcellaCapua
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback