TWiki> HEPTape Web>TapeMetricsJSON (revision 6)EditAttachPDF

WLCG sites Tape Metrics

Introduction

This page contains documentation related to the WLCG sites Tape Metrics that have been agreed and should be periodically reported by all sites.

List of tape related metrics - for each VO

  • Amount of data stored (in Bytes, rounded to decimal GB) = amount of actual data of the given VO stored on tape (including multiple copies).
    • 1 GB = 1000000000 bytes
    • Example of rounding to GB using round(number[, ndigits]) function: round (23125065994338304, -9) = 23125066000000000
  • Amount of space occupied (in Bytes, rounded to decimal GB) = amount of tape space occupied by the VO data (including multiple copies), includes holes after deleted data or partially full tapes.
    • 1 GB = 1000000000 bytes
    • Example of rounding to GB using round(number[, ndigits]) function: round (62715387334098944, -9) = 62715387000000000

  • Average Tape Remounts (count, rounded to 2 decimal points): period = past 24h
    • Formula to use: for a given VO, sum how many times each tape was mounted in the past 24 hours (group by tape), calculate the average
    • Formula in shell script: awk '{print $COLUMN_WITH_TAPE}' PAST_24_HOURS_TAPE.VO.LOG | sort | uniq -c | awk '{ sum += $1; n++ } END { if (n > 0) print sum / n; }'
    • Example: For VOx, in the past 24 hours, tape T1 was mounted once, T2 5 times and T3 twice. Average remount (rounded to 2 decimal points) is: (1 + 5 + 2) / 3 = 8 / 3 = 2.67
    • Ideal value = 1. It means that for a given VO, for the past 24 hours, all tapes were mounted only once as all the (file) requests were properly grouped together.

Reporting mechanism

Data from all sites will be collected by a pull mechanism as part of the WLCG Space Accounting system.

Each site should provide a URL with data stored in the following valid JSON file format:

{
    "storageservice": {
        "name": "SITE-Tape",
        "implementation": "APPLICATION_NAME",
        "implementationversion": "APPLICATION_VERSION",
        "latestupdate": TIMESTAMP_OF_THE_REPORT,
        "storageshares": [
            {
                "name": "VO_NAME",
                "usedsize": BYTES_ROUNDED_TO_GB,
                "occupiedsize": BYTES_ROUNDED_TO_GB,
                "avgtaperemounts": COUNT_2_DECIMAL,
                "timestamp": TIMESTAMP_OF_THE_SUMMARY,
                "vos": ["vo_name"]
            },
             :
             :
             :
            {
                "name": "VO_NAME",
                "usedsize": BYTES_ROUNDED_TO_GB,
                "occupiedsize": BYTES_ROUNDED_TO_GB,
                "avgtaperemounts": COUNT_2_DECIMAL,
                "timestamp": TIMESTAMP_OF_THE_SUMMARY,
                "vos": ["vo_name"]
            }
        ]
    }
}

CERN example

As of 19 January 2018, the JSON file reported by CERN looked like this:

{
    "storageservice": {
        "name": "CERN-PROD-Tape",
        "implementation": "CASTOR",
        "implementationversion": "2.1.17-20",
        "latestupdate": 1516345784,
        "storageshares": [
            {
                "name": "ALICE",
                "usedsize": 29723846000000000,
                "occupiedsize": 28461340000000000,
                "avgtaperemounts": 1.04,
                "timestamp": 1516268694,
                "vos": ["alice"]
            },
            {
                "name": "ATLAS",
                "usedsize": 60749939000000000,
                "occupiedsize": 62753870000000000,
                "avgtaperemounts": 1.00,
                "timestamp": 1516268694,
                "vos": ["atlas"]
            },
            {
                "name": "CMS",
                "usedsize": 48786313000000000,
                "occupiedsize": 58302772000000000,
                "avgtaperemounts": 1.03,
                "timestamp": 1516268694,
                "vos": ["cms"]
            },
            {
                "name": "LHCb",
                "usedsize": 22773166000000000,
                "occupiedsize": 23196534000000000,
                "avgtaperemounts": 1.08,
                "timestamp": 1516268694,
                "vos": ["lhcb"]
            }
        ]
    }
}

-- VladimirBahyl - 2018-01-19

Edit | Attach | Watch | Print version | History: r10 | r8 < r7 < r6 < r5 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r6 - 2018-01-22 - VladimirBahyl
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    HEPTape All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback