Run Control

Getting started

Log into np04-srv-024 using the np04daq(!) account.

Please read the 'welcome' message that is printed (if you don't see one, check that you're on the right server and right account). It will give you some instructions like this:

 ---- Welcome to NP04 Run Control Server! ---- 



See https://twiki.cern.ch/twiki/bin/view/CENF/NanorcRunControl for further instructions.


Currently running tmux sessions are:

cmacias: 1 windows (created Tue May  3 12:05:42 2022) [204x47] (attached)
np02_coldbox: 1 windows (created Tue May  3 09:18:00 2022) [102x27]

To attach to a session, do:  'tmux a -t <session_name>'


If you are taking a run, please attach to the appropriate session and run from that environment.

Follow the instructions to attach to the relevant tmux session for running.

If you don't see the right one running, it may need to be restarted. Do

tmux new -s np0X_coldbox
(replace with the appropriate session name). After restarting it, you will need to setup the DAQ environment for running:

NOTA BENE: The tmux prefix key for the np04daq account is set to Ctrl-\

cd ~np04daq/DAQ_COLDBOX_AREA
source setup_for_run.sh

You're ready to start!

Using Run Control

  • Choose a global configuration corresponding to the required setup. Configurations are located in:

 ls ~np04daq/DAQ_COLDBOX_AREA/runarea/global_configs 

NameSorted ascending Details Partition name
np02_coldbox NP02 coldbox running with real hsi np02_coldbox

  • Start run control (from the runarea) with

nano04rc --timeout 120 global_configs/<config_name>.json <username>

where config_name is from the table above (or printed out when you setup for the run).

To see the commands available use help

shonky rc> help

Documented commands (type help <topic>):
========================================
boot  init  conf  pause  resume  scrap  start  status  stop  terminate  wait

Undocumented commands:
======================
exit  help  quit

  • boot will start your applications. In the case of the example, a trigger application to supply triggers, a hardware signal interface (HSI) application, and a readout and dataflow application which receives the triggers.

  • You can then send the init, conf and start commands to get things going.

Start takes a parameter run-type which can take TEST if the data is not going to be used or PROD if the data is planned to be suitable for further analysis, usually you want to run start PROD

The commands produce quite verbose output so that you can see what was sent directly to the applications without digging in the logfiles.

Use status to see what's going on.

  • When you're done use stop, scrap and exit commands.

  • If you want to set a specific trigger rate (and overwrite the one set in the config file), the start and resume command support the option --trigger-interval-ticks
  • The parameter you passed are the number of time ticks between generated triggers. As we run with a clock rate of 50MHz at the moment this means that a value of 50000000 results in a trigger rate of 1Hz
  • If you want to decrease the trigger rate, you have to increase the number of ticks, for example start --trigger-interval-ticks 500000000 PROD will result in a trigger rate of 0.1Hz
  • If you want to change the trigger rate while staying in the same run, you can use the command pause to pause the triggers and then resume (which supports the --trigger-interval-ticks option too)

Using tmux

To be able to share the same terminal session and pass running instances of nanorc between people, we use tmux. The commands that we use are:
  • tmux new -s session_name to create a new named session, this name could be for example np02_coldbox to run the nanorc session for the np02 coldbox inside
  • To check which sessions are running, use tmux ls
  • To attach to an existing tmux session, use tmux attach -t session_name
  • To detach from the session, use CTRL + B and then press D
Make sure that you are on the same machine that the session was created on and use the correct user (np04daq) to attach to it!

Align timing

How to align the timing depends on the system that is used, for all the commands you have to be logged in to np04-srv-012 and setup the work area (cd DAQ_COLDBOX_AREA; source setup.sh)
  • For the SSPs on top of np02: pdtbutler mst PROD_FANOUT_2 align apply-delay 0 0 0 --force
  • For the np02 coldbox: pdtbutler mst PROD_MASTER_VDC align apply-delay 0 0 0 --force
  • For the np04 coldbox: pdtbutler mst TERTIARY align apply-delay 0 0 0 --force
Sometimes these commands have to be issued multiple times until the system is correctly aligned

FELIX alignment

For configurations using the FELIX (for example the coldboxes) a felixcardcontroller daq_application, configured and aligns the links. You can monitor their status via the grafana (frontend dashboard):
  • For the np02 coldbox, the FELIX is on server np04-srv-030
  • For the np04 coldbox, the FELIX is on server np04-srv-028
  • For the VST, the FELIX is on server np04-srv-026
You can use the following commands on the hosts if you suspect that something is not right (setup the workarea before to load them, cd DAQ_COLDBOX_AREA; source setup.sh)
  • flx-info GBT This shows the alignment for every link, for the np02 you should see 8 YES and for the np04 coldbox 10 YES if the links are correctly aligned
  • flx-reset GTH Resets the links, check that they are aligned afterwards. If not, there might be a bigger underlying problem (i.e. the WIBs are off/disconnected)

These commands can be executed during a run! They won't affect data taking conditions.

Switching on and off FEMB, and checking their status

source /cvmfs/dunedaq.opensciencegrid.org/products/setup
source /cvmfs/dunedaq-development.opensciencegrid.org/products/setup
setup protodune_wibsoft v351a -q e19:prof
cd /nfs/sw/wib/WIBSoftwareTrunk/scripts
BUTool.exe -X power_status_FEMB_VD_ColdBox.script # will give you the status
BUTool.exe -X power_on_FEMB_VD_ColdBox.script # turn the FEMB on
BUTool.exe -X power_off_FEMB_VD_ColdBox.script # turn the FEMB off

Troubleshooting

  • If your run fails on init and you see response code 503, then unset the proxy: exit from nano04rc and run source ~np04daq/bin/web_proxy.sh -u, then restart
  • The conf commands failed and you see a message that tells you that some FIFO queues could not be synced: The WIB timing is not correctly aligned, see the section for "Align timing"
  • The run was started but there is no data (and you are not running with the SSPs): Make sure that the FELIX is aligned

Viewing run information

  • Open the Grafana server at http://np04-srv-009:3000/
  • Select your partition id on the top left, usually we use np02_coldbox, np04_coldbox or np02_ssps
  • The messages shown in top box can be filtered to show for example just ERROR messages
  • Lower panels show key quantities like the trigger rate and the data writer which show that data is flowing through the DAQ system.
  • From the overview screen one can select subsystems in the top right

-- AlexTapper - 2021-10-19

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng DAQOverview.png r2 r1 manage 275.9 K 2021-10-19 - 13:19 AlexTapper  
PNGpng EventBuilder.png r1 manage 273.7 K 2021-10-19 - 13:19 AlexTapper  
PNGpng Timing.png r1 manage 318.3 K 2021-10-19 - 13:19 AlexTapper  
PNGpng TriggerHSI.png r1 manage 107.8 K 2021-10-19 - 13:19 AlexTapper  
Edit | Attach | Watch | Print version | History: r28 < r27 < r26 < r25 < r24 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r28 - 2022-06-08 - AlessandroThea
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CENF All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback