Automation of Data Validation

Introduction

To assist with the validation of the accounting data, a script has been written to download the data from the two sources, calculate their ratios and publish the results on the Site Status Board. The script is set to be executed automatically on a monthly basis. Extra emphasis was given on the visualisation of the data.

This work was done by Dimitrios Christidis as part of the 2016 CERN opelab Summer Student Programme, under the supervision of Julia Andreeva. The full report will be available on ZENODO.

Information for users

At the time of writing, only ATLAS and CMS provide an interface to acquire the accounting data on demand. A limited set of data for ALICE and LHCb has been published manually.

Main view

The Accounting Validation view shows the latest data in a table format. The data is grouped by experiment, then by metric.

view.png

Figure 1: The main view on the Site Status Board.

Graphs

The true usefulness of the Site Status Board comes from the ability to visualise the data over time. Click on one of the Ratio metrics on the table header. This will take you to the Site View for the requested metric. Specify the desired Time Period and Site Selection, then press Refresh.

graph.png

Figure 2: One of the graphs produced by the Site Status Board.

For easy access, below are direct links to the 2016 data:

Colour coding

The colour reflects the distance of a particular ratio from 100%. As the distance grows, the colour changes from green to yellow and finally to red.

Three thresholds are defined to specify which colour to choose. To allow for more flexibility, they are set on a per-experiment basis. The values that are currently used are shown below.

Experiment Green Yellow Red
ALICE 10.0 30.0 50.0
ATLAS 10.0 30.0 50.0
CMS 25.0 50.0 75.0
LHCb 20.0 40.0 60.0

Table 1: Colour-coding thresholds, as of 2016-10-17.

Raw data

Clicking anywhere on the graph will display a JSON file. The value of the Status key is the raw data for that particular site and month, separated by commas. The numbers are: HS06 Experiment, HS06 Site, HS06 Ratio, Raw Experiment, Raw Site and Raw Ratio.

raw.png

Figure 3: Display of raw data in JSON file.

Information for maintainers

Documentation is provided with the source code on CERN GitLab.

-- DimitriosChristidis - 2016-08-23

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng graph.png r2 r1 manage 188.1 K 2016-08-23 - 11:50 DimitriosChristidis  
PNGpng raw.png r2 r1 manage 156.1 K 2016-08-23 - 11:51 DimitriosChristidis  
PNGpng view.png r2 r1 manage 281.9 K 2016-08-23 - 11:50 DimitriosChristidis  
Edit | Attach | Watch | Print version | History: r11 < r10 < r9 < r8 < r7 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r8 - 2016-10-17 - DimitriosChristidis
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback