NanorcRunControl < CENF

CENF Web>RDProjects>DUNEDAQProto>NanorcRunControl (2022-06-08, AlessandroThea) (raw view)
---+ Run Control

---++ Getting started

 Log into =np04-srv-024= using the np04daq(!) account.

Please read the 'welcome' message that is printed (if you don't see one, check that you're on the right server and right account). It will give you some instructions like this:
<verbatim>
 ---- Welcome to NP04 Run Control Server! ---- 



See https://twiki.cern.ch/twiki/bin/view/CENF/NanorcRunControl for further instructions.


Currently running tmux sessions are:

cmacias: 1 windows (created Tue May  3 12:05:42 2022) [204x47] (attached)
np02_coldbox: 1 windows (created Tue May  3 09:18:00 2022) [102x27]

To attach to a session, do:  'tmux a -t <session_name>'


If you are taking a run, please attach to the appropriate session and run from that environment.
</verbatim>

Follow the instructions to attach to the relevant tmux session for running.

If you don't see the right one running, it may need to be restarted. Do
<verbatim>
tmux new -s np0X_coldbox
</verbatim>
(replace with the appropriate session name). After restarting it, you will need to setup the DAQ environment for running:

*NOTA BENE*: The tmux prefix key for the  =np04daq= account is set to =Ctrl-\=

<verbatim>
cd ~np04daq/DAQ_COLDBOX_AREA
source setup_for_run.sh
</verbatim>

You're ready to start!

---++ Using Run Control

   * Choose a global configuration corresponding to the required setup. Configurations are located in:

<verbatim> ls ~np04daq/DAQ_COLDBOX_AREA/runarea/global_configs </verbatim>

<!--

 | *Name* | *Details* | *Offline metadata "run_type"* |
 | =np02_pds= | NP02 System running with 3 SSP boards (50 MHz) | vd-protodune-pds |
 | =np02_bde_coldbox= | NP02 VD coldbox DAQ will readout 3.5 WIB1 boards (bottom electronics) for the first run | vd-coldbox-bottom |
 | =np04_coldbox= | NP04 HD coldbox DAQ will readout 5 WIB1 boards (full APA) for the first run | hd-coldbox |
 | =np04_vst= | NP04 VST Test crate (in DAQ barrack) with 4 WIB1 cards to validate DAQ integration | test |
 | =np04_vst_16ms= | Same as VST, but 16ms readout window | test |
 | =np04_vst_bigwin_500ms= | Same as VST, but 500ms long readout window | test |


<verbatim> ls /nfs/sw/dunedaq/dunedaq-v2.8.2-dqm/configurations/noise_cfg/ </verbatim>

-->

 | *Name* | *Details* | *Partition name* |
 | =np02_coldbox= | NP02 coldbox running with real hsi | np02_coldbox |
 <!-- | =np04_coldbox= | NP04 coldbox running with fake hsi | np04_coldbox | -->


   * Start run control (from the runarea) with 

<verbatim>
nano04rc --timeout 120 global_configs/<config_name>.json <username>
</verbatim>

where =config_name= is from the table above (or printed out when you setup for the run).

To see the commands available use =help=

<verbatim>
shonky rc> help

Documented commands (type help <topic>):
========================================
boot  init  conf  pause  resume  scrap  start  status  stop  terminate  wait

Undocumented commands:
======================
exit  help  quit
</verbatim>

   * =boot= will start your applications. In the case of the example, a trigger application to supply triggers, a hardware signal interface (HSI) application, and a readout and dataflow application which receives the triggers.

   * You can then send the =init=, =conf= and =start= commands to get things going.

Start takes a parameter =run-type= which can take =TEST= if the data is not going to be used or =PROD= if the data is planned to be suitable for further analysis, usually you want to run =start PROD=

The commands produce quite verbose output so that you can see what was sent directly to the applications without digging in the logfiles.

Use =status= to see what's going on.

   * When you're done use =stop=, =scrap= and =exit= commands. 

   * If you want to set a specific trigger rate (and overwrite the one set in the config file), the start and resume command support the option =--trigger-interval-ticks=
   * The parameter you passed are the number of time ticks between generated triggers. As we run with a clock rate of 50MHz at the moment this means that a value of 50000000 results in a trigger rate of 1Hz
   * If you want to decrease the trigger rate, you have to increase the number of ticks, for example =start --trigger-interval-ticks 500000000 PROD= will result in a trigger rate of 0.1Hz
   * If you want to change the trigger rate while staying in the same run, you can use the command pause to pause the triggers and then resume (which supports the =--trigger-interval-ticks= option too)

<!--

---++ Fake setup for training
If you want to use nanorc without running on real hardware, you can use a fake config. To do so, login to a server that is not used by any running nanorc instances or daq applications. Login as np04daq, setup the workarea and go to the run directory =/nfs/sw/dunedaq/dunedaq-v2.8.2-candidate/runarea/fake_conf=
You can now run a version of nanorc that does not use the run number service (otherwise you would increase the global run number for every test you do). You use this version by running =nanorc= instead of nano04rc=. The fake configuration is located here: =/nfs/sw/dunedaq/dunedaq-v2.8.2-candidate/configurations/fake_conf.json=.
For example do =nanorc /nfs/sw/dunedaq/dunedaq-v2.8.2-candidate/configurations/fake_conf.json= to start nanorc. The only difference to nano04rc in the commands is that start doesn't get PROD or TEST as an option but you directly specify the run number to use, i.e. =start 42=.
You can play around now and change trigger rate if you want and watch how the system behaves on the grafana dashboards, the partition name is =fake_conf=.

-->

---++ Using tmux
To be able to share the same terminal session and pass running instances of nanorc between people, we use tmux. The commands that we use are:
   * =tmux new -s session_name= to create a new named session, this name could be for example =np02_coldbox= to run the nanorc session for the np02 coldbox inside
   * To check which sessions are running, use =tmux ls=
   * To attach to an existing tmux session, use =tmux attach -t session_name=
   * To detach from the session, use CTRL + B and then press D
Make sure that you are on the same machine that the session was created on and use the correct user (np04daq) to attach to it!

---++ Align timing
How to align the timing depends on the system that is used, for all the commands you have to be logged in to np04-srv-012 and setup the work area (=cd DAQ_COLDBOX_AREA; source setup.sh=)
   * For the SSPs on top of np02: =pdtbutler mst PROD_FANOUT_2 align apply-delay 0 0 0 --force=  
   * For the np02 coldbox: =pdtbutler mst PROD_MASTER_VDC align apply-delay 0 0 0 --force=
   * For the np04 coldbox: =pdtbutler mst TERTIARY align apply-delay 0 0 0 --force=
Sometimes these commands have to be issued multiple times until the system is correctly aligned

---++ FELIX alignment
For configurations using the FELIX (for example the coldboxes) a felixcardcontroller daq_application, configured and aligns the links. You can monitor their status via the grafana (frontend dashboard):
   * For the np02 coldbox, the FELIX is on server np04-srv-030 
   * For the np04 coldbox, the FELIX is on server np04-srv-028
   * For the VST, the FELIX is on server np04-srv-026
You can use the following commands on the hosts if you suspect that something is not right (setup the workarea before to load them, =cd DAQ_COLDBOX_AREA; source setup.sh=)
   * =flx-info GBT= This shows the alignment for every link, for the np02 you should see 8 YES and for the np04 coldbox 10 YES if the links are correctly aligned
   * =flx-reset GTH= Resets the links, check that they are aligned afterwards. If not, there might be a bigger underlying problem (i.e. the WIBs are off/disconnected)

These commands can be executed during a run! They won't affect data taking conditions.

---++ Switching on and off FEMB, and checking their status
<verbatim>
source /cvmfs/dunedaq.opensciencegrid.org/products/setup
source /cvmfs/dunedaq-development.opensciencegrid.org/products/setup
setup protodune_wibsoft v351a -q e19:prof
cd /nfs/sw/wib/WIBSoftwareTrunk/scripts
BUTool.exe -X power_status_FEMB_VD_ColdBox.script # will give you the status
BUTool.exe -X power_on_FEMB_VD_ColdBox.script # turn the FEMB on
BUTool.exe -X power_off_FEMB_VD_ColdBox.script # turn the FEMB off
</verbatim>

---++ Troubleshooting
   * If your run fails on init and you see response code 503, then unset the proxy: exit from nano04rc and run =source ~np04daq/bin/web_proxy.sh -u=, then restart
   * The conf commands failed and you see a message that tells you that some FIFO queues could not be synced: The WIB timing is not correctly aligned, see the section for "Align timing"
   * The run was started but there is no data (and you are not running with the SSPs): Make sure that the FELIX is aligned

---++ Viewing run information

   * Open the Grafana server at [[http://np04-srv-009:3000/]]
   * Select your partition id on the top left, usually we use =np02_coldbox=, =np04_coldbox= or =np02_ssps=
   * The messages shown in top box can be filtered to show for example just =ERROR= messages
   * Lower panels show key quantities like the trigger rate and the data writer which show that data is flowing through the DAQ system.
   * From the overview screen one can select subsystems in the top right

   * To get information about the latest runs in the run registry, see this page: http://dunedaq-microservices:5005/runregistry/getRunMetaLast/100

   * The DQM display can be found here: https://dune-dqm.app.cern.ch
 

-- Main.AlexTapper - 2021-10-19
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who
png	DAQOverview.png	r2 r1	manage	275.9 K	2021-10-19 - 13:19	AlexTapper
png	EventBuilder.png	r1	manage	273.7 K	2021-10-19 - 13:19	AlexTapper
png	Timing.png	r1	manage	318.3 K	2021-10-19 - 13:19	AlexTapper
png	TriggerHSI.png	r1	manage	107.8 K	2021-10-19 - 13:19	AlexTapper
Topic revision: r28 - 2022-06-08 - AlessandroThea
CENF
Public webs
- Cern Search
- TWiki Search
- Google Search
CENF All webs
Copyright &© 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback