Run Control

Getting started

Log into np04-srv-024 using the np04daq(!) account.

Move to and setup the DAQ environment for running:

cd DAQ_COLDBOX_AREA
source setup_for_run.sh

You're ready to start!

Using Run Control

• Choose a global configuration corresponding to the required setup. Configurations are located in:

 ls ~np04daq/DAQ_COLDBOX_AREA/runarea/global_configs 

Name	Details	Partition name
np02_coldbox	NP02 coldbox running with real hsi	np02_coldbox

• Start run control (from the runarea) with

nano04rc --timeout 120 global_configs/<config_name>.json <username>

where config_name is from the table above (or printed out when you setup for the run).

To see the commands available use help

shonky rc> help

Documented commands (type help <topic>):
========================================
boot  conf  init  pause  resume  scrap  start  status  stop  terminate  wait

Undocumented commands:
======================
exit  help  quit

• boot will start your applications. In the case of the example, a trigger application to supply triggers, a hardware signal interface (HSI) application, and a readout and dataflow application which receives the triggers.

• You can then send the init, conf and start commands to get things going.

Start takes a parameter run-type which can take TEST if the data is not going to be used or PROD if the data is planned to be suitable for further analysis, usually you want to run start PROD

The commands produce quite verbose output so that you can see what was sent directly to the applications without digging in the logfiles.

Use status to see what's going on.

• When you're done use stop, scrap and exit commands.

• If you want to set a specific trigger rate (and overwrite the one set in the config file), the start and resume command support the option --trigger-interval-ticks
• The parameter you passed are the number of time ticks between generated triggers. As we run with a clock rate of 50MHz at the moment this means that a value of 50000000 results in a trigger rate of 1Hz
• If you want to decrease the trigger rate, you have to increase the number of ticks, for example start --trigger-interval-ticks 500000000 PROD will result in a trigger rate of 0.1Hz
• If you want to change the trigger rate while staying in the same run, you can use the command pause to pause the triggers and then resume (which supports the --trigger-interval-ticks option too)

Using tmux

To be able to share the same terminal session and pass running instances of nanorc between people, we use tmux. The commands that we use are:

• tmux new -s session_name to create a new named session, this name could be for example np02_coldbox to run the nanorc session for the np02 coldbox inside
• To check which sessions are running, use tmux ls
• To attach to an existing tmux session, use tmux attach -t session_name
• To detach from the session, use CTRL + B and then press D

Make sure that you are on the same machine that the session was created on and use the correct user (np04daq) to attach to it!

Align timing

How to align the timing depends on the system that is used, for all the commands you have to be logged in to np04-srv-012 and setup the work area (cd DAQ_COLDBOX_AREA; source setup.sh)

• For the SSPs on top of np02: pdtbutler mst PROD_FANOUT_2 align apply-delay 0 0 0 --force
• For the np02 coldbox: pdtbutler mst PROD_MASTER_VDC align apply-delay 0 0 0 --force
• For the np04 coldbox: pdtbutler mst TERTIARY align apply-delay 0 0 0 --force

Sometimes these commands have to be issued multiple times until the system is correctly aligned

FELIX alignment

For configurations using the FELIX (for example the coldboxes) you have to make sure that the links are correctly aligned. To do so, login to the machine running the readout and check/issue a reset:

• For the np02 coldbox, login to server np04-srv-030
• For the np04 coldbox, login to server np04-srv-028

You can use the following commands (setup the workarea before to load them, cd DAQ_COLDBOX_AREA; source setup.sh)

• flx-info GBT This shows the alignment for every link, for the np02 you should see 8 YES and for the np04 coldbox 10 YES if the links are correctly aligned
• flx-reset GTH Resets the links, check that they are aligned afterwards. If not, there might be a bigger underlying problem (i.e. the WIBs are off/disconnected)

These commands can be executed during a run! They won't affect data taking conditions.

Switching on and off FEMB, and checking their status

source /cvmfs/dunedaq.opensciencegrid.org/products/setup
source /cvmfs/dunedaq-development.opensciencegrid.org/products/setup
setup protodune_wibsoft v351a -q e19:prof
cd /nfs/sw/wib/WIBSoftwareTrunk/scripts
BUTool.exe -X power_status_FEMB_VD_ColdBox.script # will give you the status
BUTool.exe -X power_on_FEMB_VD_ColdBox.script # turn the FEMB on
BUTool.exe -X power_off_FEMB_VD_ColdBox.script # turn the FEMB off

Troubleshooting

• If your run fails on init and you see response code 503, then unset the proxy: exit from nano04rc and run source ~np04daq/bin/web_proxy.sh -u, then restart
• The conf commands failed and you see a message that tells you that some FIFO queues could not be synced: The WIB timing is not correctly aligned, see the section for "Align timing"
• The run was started but there is no data (and you are not running with the SSPs): Make sure that the FELIX is aligned

Viewing run information

• Open the Grafana server at http://np04-srv-009:3000/
• Select your partition id on the top left, usually we use np02_coldbox, np04_coldbox or np02_ssps
• The messages shown in top box can be filtered to show for example just ERROR messages
• Lower panels show key quantities like the trigger rate and the data writer which show that data is flowing through the DAQ system.
• From the overview screen one can select subsystems in the top right

• To get information about the latest runs in the run registry, see this page: http://dunedaq-microservices:5005/runregistry/getRunMetaLast/100

• The DQM display can be found here: https://dune-dqm.app.cern.ch
• If you don't have the correct access rights, contact Yann Donon

-- AlexTapper - 2021-10-19