Introduction

Xmon is a trigger rate monitoring system that compares live trigger rates to predicted trigger rate values based on the beam's luminosity.

Offline Predictions

While Xmon simultaneously publishes two predicted rates for each trigger during the run, the offline prediction is almost always preferred for rate comparison. The offline predictions are calculated from fits of trigger cross-section (rate/lumi) vs pileup from a previous run. With a Python script, and soon on the Xmon website, the user inputs a Root file containing trigger information for a specific run and gives the desired triggers as arguments. The script operates in 3 steps: initial fit, outlier rejection, and filtered fit.

The initial fit plots the data (cross-section vs pileup) and calculates a linear fit based on all of the points.

The outlier rejection begins by calculating the residuals and plotting them on a histogram. This histogram is then fitted to a Gaussian. The bounds for the filter are determined by the standard deviation of the Gaussian multiplied by a constant. Any point which is outside of these bounds is ignored by the next stage of the script.

The filtered fit fits only those points which are within the bounds determined by the outlier rejection. It performs both first-order and second-order polynomial fits, as well as a fit based on a custom function. Then, the functions are compared to determine the best fit, which is published to the prediction file.

The script outputs 4 plots for each trigger: the raw fit, the residuals, the color-coded outliers, and the filtered fit. The script also outputs a json file containing the run number, fit function, prediction error, and fit parameters for each trigger. When all the fits are satisfactory, this file is uploaded to P1 for use by the Xmon adapter.

Online Predictions

Online, or on-the-fly (OTF), predictions do not go through such a rigorous process. They are used in scenarios where the offline predictions either aren't available or aren't useful.

When the trigger rate is updated for a trigger that Xmon is monitoring, its cross-section is added to one array, and the current pileup is added to another array. When the arrays have over 2000 values, the oldest value in each array is erased before the new values are added. Every few minutes, linear fit parameters for trigger cross-section vs pileup are calculated using these arrays, allowing the adapter to calculate a predicted trigger rate without any predetermined fit parameters.

Adapter

The Xmon adapter is installed at P1 and runs during beam operation. It requires arguments for its partition and the path to its configuration file. Upon startup, the adapter reads its configuration from an xml file, which provides important information such as the path to the prediction file, the server to read the luminosity from, and names for published results, among other things. The adapter then reads the prediction file and configures the appropriate triggers. Keys, L1 to HLT mapping, and HLT to PS mapping are read in from the trigger menu. In order to obtain trigger rate information, the adapter subscribes to the appropriate IS server for each trigger. It also subscribes to a server for the keys, as well as a server to retrieve information about luminosity and pileup.

When the adapter is subscribed to the IS servers, a callback function will be called whenever an updated value is published. L1 and HLT triggers largely do the same thing in their callback functions. The rate and prescale value are retrieved from the server. If these values are retrieved without error, then the trigger is initialized. If both the trigger and the luminosity are initialized, then prediction values will be calculated. The adapter calculates the parameters for the OTF prediction if necessary, then calculates the OTF predicted rate. It also calculates the predicted rate and error based on the offline predictions. After the calculations are complete, the predicted rate, prediction error, OTF fit parameters, and OTF predicted rate are published to the IS server.

Compiling Xmon

To compile Xmon, you first must setup your environment. Go to a new directory, then create a text filed called CMakeLists.txt which contains the following:

cmake_minimum_required(VERSION 3.14.0)

project(Xmon)

find_package(TDAQ)

include(CTest)

tdaq_work_area()

Then, you must setup TDAQ:

source /cvmfs/atlas.cern.ch/repo/sw/tdaq/tools/cmake_tdaq/bin/cm_setup.sh tdaq-09-00-00 x86_64-centos7-gcc8-opt

To compile, do the following commands:

git clone ssh://git@gitlab.cern.ch:7999/atlas-usa-pitt/Xmon.git

cmake_config x86_64-centos7-gcc8-opt

cd x86_64-centos7-gcc8-opt

make

make install

Running Xmon

After your environment is setup and Xmon is compiled, you can run it. If you want to run with local changes, set the LD_LIBRARY_PATH and PATH environment variables to your installed lib and bin directories, e.g.:

export LD_LIBRARY_PATH=/afs/cern.ch/user/n/nfelice/xmon/installed/x86_64-centos7-gcc8-opt/lib:$LD_LIBRARY_PATH

export PATH=/afs/cern.ch/user/n/nfelice/xmon/installed/x86_64-centos7-gcc8-opt/bin/:$PATH

To check that the executable will run from the right path, you can do which xmonpredadapter_exe and see that it points to your Xmon directory.

PredAdapter Code

The executable is compiled from three source files: PredAdapter.h, PredAdapter.cpp, and PredAdapterMain.cpp. In git, it can be found in the TRP repository and in its own dedicated Xmon repository.

PredAdapter.h

The header file defines the PredAdapter class which contains data structures for storing trigger information. The most important struct is trigger_item. Each instance of this struct corresponds to one trigger that Xmon is monitoring. As such, it contains variables to store information about the trigger, as well as member functions to calculate predicted trigger rate values.

PredAdapterMain.cpp

This is the main file which creates an instance of the PredAdapter class. It reads first reads in the command line arguments, then parses the xml configuration file. It gives the configuration information to the PredAdapter constructer and calls the functions required for setup before calling the run() function to begin the operation of the adapter. When it receives the stop signal, it unsubscribes from the IS servers and terminates the adapter.

PredAdapter.cpp

The bulk of the work is done in PredAdapter.cpp. The first thing it does is configure the adapter. It reads the prediction file, creates trigger_item objects for each trigger, and sets the prediction parameter variables to their appropriate values. Then, it reads the trigger keys, L1 to HLT map, and HLT to PS map from the trigger menu. With the maps, each HLT trigger is matched to its L1 trigger and prescale value. After the triggers are set up, the adapter subscribes to IS servers to retrieve rate information for each trigger. The adapter also subscribes to servers for the keys and for luminosity information. Whenever a value is updated in IS, a callback function is called based on the value that was updated.

The luminosity callback reads in the luminosity and pileup values and stores them in the appropriate variables. When the function is called, it sets LUM_INITIALIZED to true.

The L1 callback first checks to see if the triggers have been formatted. If not, then the setIndices() function is called. This function creates maps from trigger names to their index in the array of trigger_items. If a trigger is successfully formatted, then its status is changed to true. If the trigger's status is still false after this function, the callback does nothing else. If the trigger's status is true, then the adapter reads in the rate, the rate after prescale, and the prescale for the L1 trigger that was updated. If LUM_INITIALIZED is set to true, then the predictions are calculated. Once the values are calculated, they are put into an IS object which is a member of the trigger_item. This IS object is then published to the server.

The HLT callback works largely the same as the L1 callback. The only difference is that the HLT trigger must also retrieve the prescale value of its corresponding L1 trigger. If this value cannot be found, then the HLT trigger is deactivated.

-- NicholasDanteFelice - 2020-06-03