Introduction

The FASER DAQ is built on top of the DAQling framework. For the test beam setup, all of the DAQ processes are running on the "faser-daq-002" PC, mounted in the rack in the control room. These are controlled through the Run Control GUi webserver as explained below and in detail in the slides at DAQ Overview and Control, particularly slide 12-13. Data are recorded to /data/ on the PC and copied every hour to /eos/project-f/faser-commissioning/TestBeamData/.

Operating Instructions

The Run Control can be accessed via Run Control GUI:

For most test beam runs, the configuration file combinedTestBeam.json should be selected, but if the tracker has been power cycled, one will need to use combinedTestBeamFullConfigure.json.

Normal operation sequence:

• Should start in DOWN state. Click "INITIALIZE" to transition to READY
  • Should take less than 10 seconds with the nominal configuration, but up to 30 seconds if the tracker is being configured
• Click START to start a run. A window will pop up in which you need to:
  1. Insert a message describing the details of this run.
  2. Select run type (should normally be "TestBeam") - this can be corrected at run stop if a mistake is made or run should be ignored
• Data taking can be paused during running with the "PAUSE" button and resumed later with "START"
  • Do not click "STOP" while in the PAUSED state - always resume the run first
• Once sufficient data is collected, click STOP:
  1. Insert a message describing the reason for stopping this run.
  2. At this point, you can choose to change the run type.
• The START/STOP messages and run type are logged in the RunList
• Click SHUTDOWN to prepare for next run
  • While technically a new run can be started without doing SHUTDOWN and INITIALIZE, this is not fully working and should NOT be used
  • Some times a process will crash while shutting down and one needs to press SHUTDOWN a second time

During a run log files for each process can be accessed using the "LOG" button and monitoring metrics from the INFO button, though the latter is better visualized in the grafana panels. In case a process detects a possible problem, the INFO button should go yellow or red. In this case please check the log file and grafana pages to try to identify the problem.

Common problems

Problems experienced regularly during running and their work around, should be described below.

Spurt of corrupted events/time outs

This can happen in case of loss of network packets when there is high traffic. Normally this should be a transient error and if the number of corrupted and timeout events do not keep rising, the run can continue.

BCID errors

This will normally happen from the start of the run and will cause the EventBuilder INFO button to turn RED. The BCID errors indicate that the detector data fragments are out of sync and therefore not good data. Restart the run (full STOP/SHUTDOWN/INITIALIZE/START sequence) and see if it recovers. If not, contact the DAQ expert.

Restarting the Run control GUI and monitoring

In case of problem with the run control GUI, one can try to restart the webserver. This can be done even while a run is on-going (it will keep going).

  ssh shifter@faser-daq-002.cern.ch
  tmux a -t rcgui

Press Ctrl-c to stop the process - copy any errors to elog or similar for later debugging

  rcgui

In case this is not a recognized command, try the following (in the tmux session):

  cd /home/shifter/software/faser-daq
  source setup.sh
  rcgui

To detach from the tmux session press Ctrl-b, then d.

To restart the histogram monitoring service a similar procedure is followed:

   ssh shifter@faser-daq-002.cern.ch
  tmux a -t histo
  Ctrl-c
  python3 scripts/Monitoring/Dashboard/run.py
  Ctrl-b, d

To create new tmux session (for example, if previous one accidently killed)

   tmux new -s <name of session>
   cd /home/shifter/software/faser-daq
   source setup.sh
   rcgui

Configuration and software locations

Code repository on faser-daq-002: /home/shifter/software/faser-daq.

Git code repository FASER git

Configuration files: /home/shifter/software/faser-daq/configs and /home/shifter/software/faser-daq/configs/Templates

To compile code

The shifters are not expected to recompile the code, but in case it is needed the instructions are:

  cd /home/shifter/software/faser-daq
  source setup.sh /home/daqling-spack-repo
  cd build
  cmake -DCMAKE_BUILD_TYPE=Release ..
  make -j8

Procedure after DAQ PC powercycle

* for experts * Several steps need to be taken to configure the DAQ PC for data taking in case the DAQ PC was shut down for some reason.

• Configure PC interface IP address for TRBs: sudo ip addr add 10.11.65.8/24 dev em1
• Mount external drive: mount /dev/sdb2 /data
• Restart rcgui and histogram applications in tmux sessions (read section above).
• Restart DCS archiving service: systemctl start dcs_archiver

-- BrianPetersen - 2021-07-24