ATLAS TRT Remote Monitoring

A Remote Monitoring framework has been set up by ATLAS, so that users can monitor detector performance and data quality outside of the control room and outside of point1. The framework involves mirroring the online information service available inside the Point 1 network onto machines on the GPN. The role of the Remote Monitor is constrained to be passive; no direct action may be taken by a Remote Monitor that will alter the ATLAS detector or the state of the run through Remote Monitoring Framework though indirect action can and should be taken in the form logging anomalies and errors for others to correct.

Prerequisites

You will need:

  • CERN AFS Account: meed to be able to connect to pc-atlas-rmon-01 or ..-02 or ...-03

  • Point 1 Account optional : a Point 1 account is only necessary if you wish to alter/create configuration files or edit e-logs.

  • NX Client Software (see below) : the NX Server Software has already been set up on the remote machines

  • Terminal Emulator: Windows Users Only will need a terminal emulator like Putty or SSH Secure Shell Client

The Remote Connection Software (NX Client)

  • Obtain: Software is the Free Version of NX Client. Download and Setup instructions for Linux, Windows, and Mac can be found at the general Remote Monitoring page. Sections 2, 3 of "Setup" are relevant.

  • Setup: See Section 4 of "Setup" at Remote Monitoring for Settings and Configurations

Connecting

  • Open a tunnel to the mirror machine:
    • Linux/Mac:
       ssh -L 8000:pc-atlas-rmon-02.cern.ch:22 myusername@lxplus.cern.ch 
      • This is actually a combination of two steps: a tunnel is created between port 8000 on your computer and a port on lxplus. Then a the port on lxplus is connected to port 22 on the mirrored machine.
      • Tip, idea Skip's Tips: It is useful to setup this command as an alias in your .bashrc or equivalent file to avoid having to remember it over and over again
    • Windows/Terminal Emulator: You should have set up a saved session in your terminal emulator that sets up tunnel between your machine and lxplus (See Remote Monitoring Section 7 of "Setup" ). Then tunnel from lxplus to the mirror ssh -L 8001:localhost:22 pc-atlas-rmon-01.cern.ch
  • Start NX Client: enter AFS username and password. Make sure configuration parameter are set correctly: Remote Monitoring Sections 4 of "Setup"

Connection Troubleshooting

  1. Make sure you are able to connect to the mirrored machine pc-atlas-rmon-01 or ..-02 or ...-03 directly by ssh-ing from lxplus. For some reason, only some users are able to connect to pc-atlas-rmon-01 although all that is supposedly required is an AFS account. Hopefully this will be fixed soon.
  2. If you opened the tunnel in two steps, did you make sure the ports match? Are they being used?
  3. Did you disconnect the last session without terminating stale processes (see..). You may need to ssh into the mirror machine and manually kill these.
  4. The machine is unresponsive or being rebooted or the NX server has been shut down. Contact ...at Point 1 to fix this.

Disconnecting ( Read This! )

When you close the NX client window to disconnect, select the Terminate button from the dialog box. This disconnects the session and terminates all processes you have launched. Other wise you may leave processes running on the machine. If these become hung up, you will have to ssh into the remote machine and kill them manually.

Software on the Remote Machine

Desktop

The Desktop is set up such that all available user programs are on the toolbar or in the two drop down menus. No shell terminal is available from this interface. If you need to access a shell command line to, for instance, kill a program, you must ssh into the remote monitoring machine you are on.

* Tip, idea Skip's Tips: Once you start connect with NX Client on a machine with multiple monitors (i.e. at the Penn Remote Monitoring Station) stretch the single window across the multi-monitor viewing space.

RM Desktop.png

DAQ Panel

The DAQ Panel (under the TDAQ menu) is GUI menu which should provide all access the monitoring application needed for Remote Monitoring. Users familiar with the Control Room DAQ Panel will notice that the Remote Monitoring set up is more restrictive, since many control room DAQ Panel applications allow active interaction with the DAQ System. As a TRT Remote Monitor you will find the RC Status, DQMD, OHP, Monitor Partition, and TRTViewer applications most useful. These are discussed individually below.

  • To setup the DAQ Panel enter the necessary setup scripts and options and click Get Partition then Read Info.
  • For more information on the DAQ Panel read the DAQPanel How To (oriented towards control room use)
  • Recommended DAQ Panel Settings are provided below:

DAQ Panel Settings (TRT Remote Monitoring)
Setup Script /data/ATLAS/scripts/setup.sh
Part Name ATLAS
Database File /data/ATLAS/database/ATLAS.data.xml
Setup Opt -newgui
Oks Opt  
MRS Filter TRT
OHP Opt -c /atlas/moncfg/tdaq-02-00-03/trt/ohp/TRTMonitoring-ohp-rmon.nexus.xml
Busy Opt  
OMD Opt  
TriP Opt  

  • Screenshot of the DAQ Panel with available applications and tabs:
    RM DAQPanel.png

RC Status

The Run Control Status GUI shows a quick summary of the run in the ATLAS Partition, including run number, run status, error state. The Detector Mask button can be expanded to show which sub-detectors are involved in the run.

RM RCStatus.png

DQMD

The Data Quality Management Display (DQMD) provides access various histograms published continuously to the mirrored Online Histogramming Service. These histograms are displayed in a pseudo-geometrical way that allows the user to navigate easily with the geometry of the sub-detector of interest. Algorithms are provided for each histogram to test the normality of the data being collected during the run. These algorithms highlight the links to the histogram red (bad), orange (warning), or green (good). The particular algorithm is difficult to deduce from the information provided in DQMD; most make a certain threshold or relative stability constraint. The algorithm parameters are available below the histogram.

Since the DQMD is the most in depth (expert level) access to the online data, other than sifting through the online histograms directly from the server, monitoring detector performance in this way seems to be an ideal job for a remote monitor since the online shifter will be concerned mostly with larger picture issues that require more immediate attention.

Select the detector of interest and navigate through the histograms by either using the detector display to the right or the drop down menu to the left. Whether or not the algorithm is yielding a warning there might be trends in the data that intrigue interest you, some might be normal but some might be important. So take a careful look.

* Tip, idea Skip's Tips: DQMD takes a while to load from the DAQ Panel: Give it a chance!

Some reference info:

RM DQMD TRT 1.png

OHP

The Online Histogram Presenter subscribes to the online histograms published by the mirrored histogramming serivce. The presenter displays various overview histrograms at different levels of depth. Often the histograms have comments and color coding suggesting the health of the data. Most if not all of the OHP data can be viewed using the DQMD but the OHP presents the most crucial histograms in a more convenient way for the shifter.

For a more in depth discussion of OHP:

RM OHP.png

Monitor Partition

The Monitor Partition Application allows you to passively monitor the ATLAS partition. The GUI shows the MRS, a message reporting system, the run control state and the state of all the segments and resources on the partition. You can browse the state/status of various RunControl applications and machines using the central part of the GUI, however you are unable to take any action. In the end this application is not overly useful for the Remote Monitor. It is useful to understand what is running at Point 1 but other people (hopefully) will be making sure that all the run processes are functional.

Tip, idea Skip's Tips: The Monitor Partition application throws a bunch of errors, some severe, when loading. Ignore these as long as the application loads in the end.

RM MonitorPartition.png

Error Logging and Reporting

Viewing Data

TRTViewer

Atlantis

VP1

Web Monitoring

  • The web gate-way to online monitoring is available here
  • LHC Beam Status, General ATLAS detector, run, and sub-detector status are available here as well, including important contact information
  • for the TRT DCS Webpage (shown below) Click on Inner Detector choose the TRT
  • Tip, idea Skip's Tips: You don't need to open this on the remote machine since its just a website

RM TRTDCS.png

Useful Links and Information

  • Calling CERN Phones
    • If you need to call a CERN shifter or expert (browse the point site above under Web Monitoring or the Control Room link directly above for numbers
    • From Penn:
      • Land Line: (9+011+41+22+76 + 5 digits)
      • Cell Phone/GSM: (9+011+41+22+487+4 digits) - cell phone numbers are usually given with 6 digits (you should drop the 2-digit prefix, usually 16)

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng RM_DAQPanel.png r3 r2 r1 manage 137.4 K 2010-01-20 - 18:06 Clester DAQ PANEL Screenshot
PDFpdf RM_DAQPanelHowTo.pdf r1 manage 1220.7 K 2009-11-11 - 19:07 Clester DAQ Panel How To
PNGpng RM_DQMD_TRT.png r1 manage 110.2 K 2010-01-20 - 18:15 Clester DQMD screenshot
PNGpng RM_DQMD_TRT_1.png r1 manage 110.2 K 2010-01-20 - 21:35 Clester DQMD screenshot
PNGpng RM_Desktop.png r1 manage 1065.7 K 2009-11-09 - 19:51 Clester Remote Monitoring Desktop
PNGpng RM_MonitorPartition.png r1 manage 515.4 K 2010-01-21 - 23:21 Clester Monitor Partition screenshot
PNGpng RM_OHP.png r1 manage 145.1 K 2010-01-21 - 23:19 Clester OHP screenshot
PNGpng RM_RCStatus.png r1 manage 4.6 K 2009-11-12 - 20:13 Clester RC Status Screenshot
PNGpng RM_TRTDCS.png r1 manage 705.9 K 2009-11-09 - 21:43 Clester TRT DCS Website Screenshot
Edit | Attach | Watch | Print version | History: r16 < r15 < r14 < r13 < r12 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r16 - 2010-01-22 - Clester
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback