(r10) AutomationOfDataValidation < LCG

LCG Web>AccountingTaskForce>AutomationOfDataValidation (revision 10) (raw view)~~EditAttachPDF~~

---+ Automation of Data Validation

%TOC%

---++ Introduction

To assist with the validation of the accounting data, a script has been written to download the data from the two sources, calculate their ratios and publish the results on the Site Status Board.
The script is set to be executed automatically on a monthly basis.
Extra emphasis was given on the visualisation of the data.

This work was done by Dimitrios Christidis as part of the CERN opelab Summer Student Programme 2016, under the supervision of Julia Andreeva.
The full report is available on [[https://zenodo.org/record/206942][ZENODO]].

---++ Information for users

---+++ Main view

The [[http://wlcg-mon-dev.cern.ch/dashboard/request.py/siteview#currentView=ACCOUNTING_VALIDATION][Accounting Validation]] view shows the latest data in a table format.
The data is grouped by experiment, then by metric.

%ATTACHURL%/view.png

*Figure 1:* The main view on the Site Status Board.

---+++ Graphs

The true usefulness of the Site Status Board comes from the ability to visualise the data over time.
Click on one of the *Ratio* metrics on the table header.
This will take you to the Site View for the requested metric.
Specify the desired *Time Period* and *Site Selection*, then press *Refresh*.

%ATTACHURL%/graph.png

*Figure 2:* One of the graphs produced by the Site Status Board.

For easy access, below are direct links to the data from January 2016 to June 2017:
   * [[http://wlcg-mon-dev.cern.ch/dashboard/request.py/siteviewhistory?columnid=1528&debug=false#time=custom&start_date=2016-01-01&end_date=2017-06-30&values=false&spline=false&debug=false&resample=false&sites=all&clouds=all][ATLAS HEP-SPEC06 Wall Clock Work]]
   * [[http://wlcg-mon-dev.cern.ch/dashboard/request.py/siteviewhistory?columnid=1525&debug=false#time=custom&start_date=2016-01-01&end_date=2017-06-30&values=false&spline=false&debug=false&resample=false&sites=all&clouds=all][ATLAS Raw Wall Clock Time]]
   * [[http://wlcg-mon-dev.cern.ch/dashboard/request.py/siteviewhistory?columnid=1534&debug=false#time=custom&start_date=2016-01-01&end_date=2017-06-30&values=false&spline=false&debug=false&resample=false&sites=all&clouds=all][CMS HEP-SPEC06 Wall Clock Work]]
   * [[http://wlcg-mon-dev.cern.ch/dashboard/request.py/siteviewhistory?columnid=1531&debug=false#time=custom&start_date=2016-01-01&end_date=2017-06-30&values=false&spline=false&debug=false&resample=false&sites=all&clouds=all][CMS Raw Wall Clock Time]]
   * [[http://wlcg-mon-dev.cern.ch/dashboard/request.py/siteviewhistory?columnid=1552&debug=false#time=custom&start_date=2016-01-01&end_date=2017-06-30&values=false&spline=false&debug=false&resample=false&sites=all&clouds=all][LHCb Wall Clock Work]]
   * [[http://wlcg-mon-dev.cern.ch/dashboard/request.py/siteviewhistory?columnid=1555&debug=false#time=custom&start_date=2016-01-01&end_date=2017-06-30&values=false&spline=false&debug=false&resample=false&sites=all&clouds=all][LHCb Raw Wall Clock Time]]

---++++ Colour coding

The colour reflects the distance of a particular ratio from 100%.
As the distance grows, the colour changes from green to yellow and finally to red.

Three thresholds are defined to specify which colour to choose.
To allow for more flexibility, they are set on a per-experiment basis.
The values that are currently used are shown below.

| *Experiment* | *Green* | *Yellow* | *Red*  |
| ALICE        | 10.0    | 30.0     | 50.0   |
| ATLAS        | 10.0    | 30.0     | 50.0   |
| CMS          | 25.0    | 50.0     | 75.0   |
| LHCb         | 20.0    | 40.0     | 60.0   |

*Table 1:* Colour-coding thresholds, as of 2016-10-17.

---+++ Raw data

Clicking anywhere on the graph will display a JSON file.
The value of the =Status= key is the raw data for that particular site and month, separated by commas.
The numbers are:
_HS06 Experiment_, _HS06 Site_, _HS06 Ratio_, _Raw Experiment_, _Raw Site_ and _Raw Ratio_.

%ATTACHURL%/raw.png

*Figure 3:* Display of raw data in JSON file.

---+++ Offline spreadsheets

The Site Status Board is not suitable for the display of older data in table format.
Thus, the automation script also produces Microsoft Office Excel files.

   * [[%ATTACHURL%/2015.xlsx][2015]]
   * [[%ATTACHURL%/2016.xlsx][2016]]
   * [[%ATTACHURL%/2017.xlsx][2017]] (incomplete)

---++ Information for maintainers

Documentation is provided with the source code on [[https://gitlab.cern.ch/WLCG-Ops-Services/accounting_validation][CERN GitLab]].

-- Main.DimitriosChristidis - 2017-03-02

Attachments

Topic attachments
I	Attachment	History	Action	Size	Date	Who
xlsx	2015.xlsx	r1	manage	192.4 K	2017-03-20 - 10:52	DimitriosChristidis
xlsx	2016.xlsx	r1	manage	185.7 K	2017-03-20 - 10:52	DimitriosChristidis
xlsx	2017.xlsx	r1	manage	35.4 K	2017-03-20 - 10:52	DimitriosChristidis
png	graph.png	r2 r1	manage	188.1 K	2016-08-23 - 11:50	DimitriosChristidis
png	raw.png	r2 r1	manage	156.1 K	2016-08-23 - 11:51	DimitriosChristidis
png	view.png	r2 r1	manage	281.9 K	2016-08-23 - 11:50	DimitriosChristidis

Topic revision: r10 - 2017-03-20 - DimitriosChristidis

LCG Wikis

LCG Service
Coordination

LCG Grid
Deployment

LCG
Apps Area

Public webs

Welcome Guest

- Cern Search
- TWiki Search
- Google Search
LCG All webs

Copyright &© 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback