Visibility from Run Control

Within each partition in Run Control, there is an InhibitMaster button that will bring up a list of the messages from the InhibitMaster process. This can be useful in identifying the source of an inhibit. Of course, if an inhibit is caused by the high voltage (HV) monitoring system or the purity monitor (PrMon) system, there are dedicated indicators on the Run Control screen that will change from green to red.

Setup

Originally, the source code for the InhibitMaster code was in /nfs/sw/work_dirs/dune-artdaq-InhibitMaster. To setup that software area, we would use: source /nfs/sw/work_dirs/dune-artdaq_artdaq_v3_02_01_testing/setupDUNEARTDAQ_forInhibitMaster

The InhibitMaster software is now commonly included and built as part of the wider dune-artdaq software suite. For example, see np04-srv-023:/scratch/biery/dune-artdaq-PrepForFebRel/srcs.

Port/node assignments

The InhibitMaster typically runs on np04-srv-024, and publishes its global status on port numbers following partition number:

• Partition 0: tcp://np04-srv-024:5598
• Partition 1: tcp://np04-srv-024:5599
• Partition 2: tcp://np04-srv-024:5600
• Partition 3: tcp://np04-srv-024:5601
• Partition 4: tcp://np04-srv-024:5602
• Partition 5: tcp://np04-srv-024:5603
• Partition 6: tcp://np04-srv-024:5604

"FrontEnds" are also assigned port numbers on which they publish their own status (which the InhibitMaster listens to). The run control creates a list of these nodes at the boot stage, when it knows what components will be included in a partition. That list is located at np04-srv-024:/tmp/pub_list_0.txt, where "0" is replaced by the partition number.

Useful to know: run control assigns itself a front end port, used for issuing trigger inhibits from run control:

• Partition 0: tcp://np04-srv-024:6345
• Partition 1: tcp://np04-srv-024:6346
• Partition 2: tcp://np04-srv-024:6347
• Partition 3: tcp://np04-srv-024:6348
• Partition 4: tcp://np04-srv-024:6349
• Partition 5: tcp://np04-srv-024:6350
• Partition 6: tcp://np04-srv-024:6351

Description of Python-based scripts

InhibitMaster.py

This is the main definition for the InhibitMaster, based in python. It contains classes for:

• a StatusPUBNode for sending out status messages from "FrontEnd"s;
• a StatusSUBNode for subscribing to status messages;
• an InhibitPUBNode for sending out the global inhibit message;
• an InhibitSUBNode for subscribing to the global inhibit message; and,
• the InhibitMaster itself, which holds the internal logic for the global inhibit value, and when to send it out.

RunInhibitMaster.py

This is a script to run an instance of the InhibitMaster. Run it by doing:

RunInhibitMaster.py InhibitMaster_pub_node status_publisher_node_list.txt

where InhibitMaster_pub_node is the port number on which to publish global inhibit messages, and status_publisher_node_list.txt is a text file with a list of expected status publishers. The run control creates these text file lists on boot and writes them to np04-srv-024:/tmp/pub_list_0.txt, where "0" is replaced by the partition number.

An example is:

TestFrontEnd tcp://localhost:5599

CmndLineRunControl tcp://localhost:5600

where there is one line for each status publisher, and on each line the first argument is a name for the FrontEnd (it doesn't really matter...), and the second argument is the name of the port to subscribe to (note the tcp format).

The script will start an InhibitMaster listening to those ports for status messages and publish the 'XON' or 'XOFF' based on their status (note, it defaults to "XON" if the status publishers aren't running: we explicitly require a bad states to force the inhibit). The InhibitMaster as an update frequency that will force the push of the current inhibit value (1 second by default, but set to 10 seconds in this script).

To stop this script, Ctrl-C.

TestFrontEnd.py

This is an example python-based FrontEnd used for testing. Usage is

TestFrontEnd.py status_pub_node

Where status_pub_node is the port number for the publisher on the local host. The script starts a StatusPUBNode and issues a status update every second. Every ten seconds it will issue a "BAD" status, and then switch back to "OK".

To stop this script, Ctrl-C.

SendCmndLineMsg.py

This is a python-based command-line utility for sending a status message once, and is what the run control uses. Usage is:

SendCmndLineMessage.py status_pub_node status marker

where status_pub_node is the port number for the publisher on the local host, status is the desired status (should be GOOD to keep trigger enabled, and BAD to enable inhibit of the trigger), and marker is a desired marker in the message (no underscores!). The script will establish a one-time pub node and send the desired message. Note there is a 0.5 second sleep to allow the pub node to bind properly before sending the message, otherwise the subscriber on the InhibitMaster end will likely miss the message.

See the above port assignments to know the node number to use given your partition.

MonitorInhibitMaster.py

This script prints the output of the inhibit master to the screen. Usage:

MonitorInhibitMaster.py InhibitMaster_pub_address

where InhibitMaster_pub_address is the address on which the InhibitMaster is publishing (eg tcp://localhost:5601).

Note that this script may currently use up 100% of CPU, and should thus be used judiciously.

MonitorFrontEnds.py

This script prints the output of the status messages received from front-end components (i.e. the input to the InhibitMaster):

MonitorFrontEnds.py status_publisher_node_list.txt

where status_publisher_node_list.txt is a list of status publishers to listen to (see format description above in the RunInhibitMaster.py description).

Note that this script may currently use up 100% of CPU, and should thus be used judiciously.

FakeSpillInhibitor.py

This produces a trigger inhibit during an "inhibit" period, and releases it during an "allow" period in order to fake a spill period. The "status_pub_node" should be the same publisher node number used for run control (so "6345" plus the partition number).

Usage: FakeSpillInhibitor.py status_pub_node allow_time_s inhibit_time_s

Description of C-based code

TriggerManager.C

This is an example code for listening to InhibitMaster messages. It simply prints out the status to the screen (similar to MonitorInhibitMaster.py), but is used as the example for extracting the inhibit information in the timing system code.

MonitorInhibitMaster_DIM.cc

This is an example code for listening to InhibitMaster messages and republishing them on DIM. Its usage is as follows:

MonitorInhibitMaster_DIM InhitbitMaster_HostName InhibitMaster_Port

It sets up a DIM service to simply publish the string of the message with DIM server name InhibitMaster5598 and service names Artdaq/InhibitMaster5598/monitoring/status and Artdaq/InhibitMaster5598/monitoring/status_detail (if port number is 5598).

status is an integer value: 0 if the inhibit is not on ("XON" from InhibitMaster), or 1 if the inhibit is on ("XOFF" from InhibitMaster). status_detail is a string value, and contains the full message from the InhibitMaster, which includes a time stamp and some details on the publish status (no change or due to change)

Improvements will be to...

• make this more configurable
• add in monitoring of status publishers / publish the InhibitMaster internal state
• modify the service output to be ints or bools for easier monitoring.

-- WesleyRobertKetchum - 2018-03-05 GiovannaLehmannMiotto - 2019-02-27