How to apply corrections to a profile

Introduction

A profile defines how to combine several metrics using an algorithm. The result is a new metric that calculates the availability.

The different operations that can be used to combine metrics are depicted in Figure 1.

Slide1.jpg

The last step before calculating the availability is to overwrite the results obtained so far with a new metric, usually called ' corrections'. This allows to change manually the availability of sites. This document describes how to do that.

How to populate a 'corrections' metric

When a profile gets created, a metric with the profile name appended by ' corrections' is also created automatically. This metric could be fed like any other metric in SSB, for instance giving a source URL where to fetch the data

, or by using the web UI to modify the values. Since the changes in corrections are not that common, the easiest way is to use the http post method, to send the new values whenever there is a need to change the availability. The format of the file that has to be uploaded is a tab separated file with:

<start time> <end time> <metric id> <instance> <status> <color> <url> <options>

See the 'Example' section for more details.

Once the new metric has been populated, SAM3 will recalculate the availability. Note that the system will recalculate everything starting from the time specified in the correction. In other words, if a correction is done for a six months ago, the system will recalculate everything from six month ago until now.

Timeline

This is a rough estimate of the time needed for the system to calculate the new availability:

  • Let's assume that the corrections metric is posted at t0
  • At t1 =~ t0 + 2 minutes, those values would be inserted in the database, and it should be visible in the history of the corrections metric at the history of the corrections of the profile
  • At t2 =~ t1 + 2 minutes, the collector that calculates the availability will take those changes. The time needed by the collector depends on the time period that has to be recomputed. For periods smaller than a month, a couple of minutes should be enough.
  • At t3 =~ t2 +
  • At t4 =~ t3 + 2 minutes, the daily and hourly availabilities should have been calculated, and the new availability should be visible in the SAM3 UI

So, in total, from the moment that the corrections metric gets populated until the changes are visible on the UI, there might be a 10 minute delay

The SAM3 UI provides also a checkbox to display the availabilities before and after the corrections.

Current profiles and correction metrics

This table contains the metric ids for the current profiles and the corrections. It is possible to find these ids also in the metric list

Profile Name VO Profile status Id Corrections id Profile status corrected id
ALICE_CRITICAL ALICE 396 718 736
ALICE_MONALISA ALICE 703 716 734
ATLAS_CRITICAL ATLAS 413 727 751
ATLAS_AnalysisAvailablity ATLAS 790 788 791
CMS_CRITICAL CMS 418 732 749
CMS_CRITICAL_FULL CMS 561 728 745
LHCb_CRITICAL LHCb 400 720 738

Possible values for the availability

The following table describes the values that are understood for availability calculation

Status Color Numerical Value
OK green 0
WARNING yellow 10
DOWNTIME saddlebrown 15
CRITICAL red 20
UNKNOWN grey 30

Example

Let's assume that it is needed to change the availability of CERN to OK for the time period between the 1/11/2014 14:00 UTC to 1/11/2014 16:30 UTC, for the ALICE_MONALISA profile.

The steps to do are the following:

  1. Find out the metric of the corrections. Looking at the table above, it is metric 716.
  2. Create the file with the changes that have to be applied.
    1. echo -e '2014-11-01 14:00:00\t2014-11-01 16:30:00\t716\tCERN\tOK\tgreen\tNone\tnvalue=0 ' > /tmp/my_data
  3. Upload the file. Please, note that the user has to be authorized!
    1. curl -k -X POST -T /tmp/my_data --cert .globus/usercert.pem --key .globus/userkey.pem 'https://wlcg-mon.cern.ch/dashboard/request.py/postMetricValues'

NOTE: curl does not seem able to handle the certificate key being protected with a password... You can create yourself a copy of the key without a passphrase and perhaps put it in a special location that is just used for this purpose. For example:

    (umask 077; openssl rsa -in ~/.globus/userkey.pem -out ~/private/sam-recalc.pem)
Then use the latter key in the curl command.

Next, the system is supposed to calculate the availability, and there are no more actions required from the user. To validate the the change has been applied, the intermediate steps that can be done are:

  1. Verify that the metric history of the corrections has been updated at ALICE_MONALISA corrections history. This might take a couple of minutes
  2. Verify that the metric of the corrected history has been updated at ALICE_MONALISA corrected history. This might take around 5 minutes.
  3. Verify that the availability has been recalculated at SAM3 UI. This might take around 10 minutes

Troubleshooting:

Here we have a list of possible errors and the way to solve them:

  • -bash-4.1$ curl  -k -X POST -T /tmp/my_data --cert .globus/usercert.pem --key .globus/userkey.pem 'https://wlcg-mon.cern.ch/dashboard/request.py/postMetricValues'
    * Unable to load client key -8178.
    
    . The key in the userkey.pem file is not in unencrypted RSA format. Please, check the content of the file, If it is in 'ENCRYPTED CERTIFICATE', then openssl rsa -in userkey.pem -out unencr_key.pem can be used to convert the key to the required format.

-- PabloSaiz - 2014-11-11

Topic attachments
I Attachment History Action Size Date Who Comment
JPEGjpg Slide1.jpg r1 manage 80.6 K 2014-11-11 - 11:47 PabloSaiz  
Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r5 - 2020-04-21 - MaartenLitmaath
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    ArdaGrid All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback