Visibility from Run Control
Within each partition in Run Control, there is an InhibitMaster button that will bring up a list of the messages from the InhibitMaster process. This can be useful in identifying the source of an inhibit. Of course, if an inhibit is caused by the high voltage (HV) monitoring system or the purity monitor (PrMon) system, there are dedicated indicators on the Run Control screen that will change from green to red.
Setup
Originally, the source code for the InhibitMaster code was in
/nfs/sw/work_dirs/dune-artdaq-InhibitMaster
. To setup that software area, we would use:
source /nfs/sw/work_dirs/dune-artdaq_artdaq_v3_02_01_testing/setupDUNEARTDAQ_forInhibitMaster
The InhibitMaster software is now commonly included and built as part of the wider dune-artdaq software suite. For example, see np04-srv-023:/scratch/biery/dune-artdaq-PrepForFebRel/srcs.
Port/node assignments
The InhibitMaster typically runs on
np04-srv-024
, and publishes its global status on port numbers following partition number:
- Partition 0:
tcp://np04-srv-024:5598
- Partition 1:
tcp://np04-srv-024:5599
- Partition 2:
tcp://np04-srv-024:5600
- Partition 3:
tcp://np04-srv-024:5601
- Partition 4:
tcp://np04-srv-024:5602
- Partition 5:
tcp://np04-srv-024:5603
- Partition 6:
tcp://np04-srv-024:5604
"FrontEnds" are also assigned port numbers on which they publish their own status (which the InhibitMaster listens to). The run control creates a list of these nodes at the boot stage, when it knows what components will be included in a partition. That list is located at
np04-srv-024:/tmp/pub_list_0.txt
, where "0" is replaced by the partition number.
Useful to know: run control assigns itself a front end port, used for issuing trigger inhibits from run control:
- Partition 0:
tcp://np04-srv-024:6345
- Partition 1:
tcp://np04-srv-024:6346
- Partition 2:
tcp://np04-srv-024:6347
- Partition 3:
tcp://np04-srv-024:6348
- Partition 4:
tcp://np04-srv-024:6349
- Partition 5:
tcp://np04-srv-024:6350
- Partition 6:
tcp://np04-srv-024:6351
Description of Python-based scripts
InhibitMaster.py
This is the main definition for the InhibitMaster, based in python. It contains classes for:
- a
StatusPUBNode
for sending out status messages from "FrontEnd"s;
- a
StatusSUBNode
for subscribing to status messages;
- an
InhibitPUBNode
for sending out the global inhibit message;
- an
InhibitSUBNode
for subscribing to the global inhibit message; and,
- the
InhibitMaster
itself, which holds the internal logic for the global inhibit value, and when to send it out.
RunInhibitMaster.py
This is a script to run an instance of the InhibitMaster. Run it by doing:
RunInhibitMaster.py InhibitMaster_pub_node status_publisher_node_list.txt
where
InhibitMaster_pub_node
is the port number on which to publish global inhibit messages, and
status_publisher_node_list.txt
is a text file with a list of expected status publishers. The run control creates these text file lists on boot and writes them to
np04-srv-024:/tmp/pub_list_0.txt
, where "0" is replaced by the partition number.
An example is:
TestFrontEnd tcp://localhost:5599
CmndLineRunControl tcp://localhost:5600
where there is one line for each status publisher, and on each line the first argument is a name for the FrontEnd (it doesn't really matter...), and the second argument is the name of the port to subscribe to (note the tcp format).
The script will start an InhibitMaster listening to those ports for status messages and publish the 'XON' or 'XOFF' based on their status (note, it defaults to "XON" if the status publishers aren't running: we explicitly require a bad states to force the inhibit). The InhibitMaster as an update frequency that will force the push of the current inhibit value (1 second by default, but set to 10 seconds in this script).
To stop this script, Ctrl-C.
TestFrontEnd.py
This is an example python-based FrontEnd used for testing. Usage is
TestFrontEnd.py status_pub_node
Where
status_pub_node
is the port number for the publisher on the local host. The script starts a
StatusPUBNode
and issues a status update every second. Every ten seconds it will issue a "BAD" status, and then switch back to "OK".
To stop this script, Ctrl-C.
SendCmndLineMsg.py
This is a python-based command-line utility for sending a status message once, and is what the run control uses. Usage is:
SendCmndLineMessage.py status_pub_node status marker
where
status_pub_node
is the port number for the publisher on the local host,
status
is the desired status (should be
GOOD
to keep trigger enabled, and
BAD
to enable inhibit of the trigger), and
marker
is a desired marker in the message (no underscores!). The script will establish a one-time pub node and send the desired message. Note there is a 0.5 second sleep to allow the pub node to bind properly before sending the message, otherwise the subscriber on the
InhibitMaster end will likely miss the message.
See the above port assignments to know the node number to use given your partition.
MonitorInhibitMaster.py
This script prints the output of the inhibit master to the screen. Usage:
MonitorInhibitMaster.py InhibitMaster_pub_address
where
InhibitMaster_pub_address
is the address on which the InhibitMaster is publishing (eg
tcp://localhost:5601
).
Note that this script may currently use up 100% of CPU, and should thus be used judiciously.
MonitorFrontEnds.py
This script prints the output of the status messages received from front-end components (i.e. the input to the InhibitMaster):
MonitorFrontEnds.py status_publisher_node_list.txt
where
status_publisher_node_list.txt
is a list of status publishers to listen to (see format description above in the RunInhibitMaster.py description).
Note that this script may currently use up 100% of CPU, and should thus be used judiciously.
FakeSpillInhibitor.py
This produces a trigger inhibit during an "inhibit" period, and releases it during an "allow" period in order to fake a spill period. The "status_pub_node" should be the same publisher node number used for run control (so "6345" plus the partition number).
Usage:
FakeSpillInhibitor.py status_pub_node allow_time_s inhibit_time_s
Description of C-based code
TriggerManager.C
This is an example code for listening to InhibitMaster messages. It simply prints out the status to the screen (similar to MonitorInhibitMaster.py), but is used as the example for extracting the inhibit information in the timing system code.
MonitorInhibitMaster_DIM.cc
This is an example code for listening to InhibitMaster messages and republishing them on DIM. Its usage is as follows:
MonitorInhibitMaster_DIM InhitbitMaster_HostName InhibitMaster_Port
It sets up a DIM service to simply publish the string of the message with DIM server name
InhibitMaster5598
and service names
Artdaq/InhibitMaster5598/monitoring/status
and
Artdaq/InhibitMaster5598/monitoring/status_detail
(if port number is 5598).
status
is an integer value: 0 if the inhibit is not on ("XON" from InhibitMaster), or 1 if the inhibit is on ("XOFF" from InhibitMaster).
status_detail
is a string value, and contains the full message from the InhibitMaster, which includes a time stamp and some details on the publish status (no change or due to change)
Improvements will be to...
- make this more configurable
- add in monitoring of status publishers / publish the InhibitMaster internal state
- modify the service output to be ints or bools for easier monitoring.
--
WesleyRobertKetchum - 2018-03-05
GiovannaLehmannMiotto - 2019-02-27