ATLAS DDM Dashboard Rucio Migration

This page documents the migration of the ATLAS DDM Dashboard to support Rucio

Overview

The proposal is summarised in the following slides taken from the ATLAS DDM Dashboard presentation on the subject at the ATLAS Software & Computing Workshop in June 2014.

Message Format

The Dashboard and DDM teams have reviewed the current message format.

We have developed the message format for Rucio transfer-done and transfer-failed.

transfer_done = {
  "created_at" : "2014-06-23 08:43:32",
  "event_type" : "transfer-done",
  "payload" : {
    "activity" : "rucio-integration",
    "dst-rse" : "INFN-T1_DATADISK",
    "dst-url" : "srm://storm-fe.cr.cnaf.infn.it:8444/srm/managerv2?SFN=/atlas/atlasdatadisk/rucio/tests/cf/c4/ESD.72b4ecd77b414e6bb2684e1e7aa552af",
    "duration" : 4,
    "file-size" : 1048576,
    "guid" : "9e738f93050c4ba5b6c11722d160ce8b",
    "name" : "ESD.72b4ecd77b414e6bb2684e1e7aa552af",
    "previous-request-id" : null,
    "protocol" : "srm",
    "request-id" : "ed441fee68414c82b8d331b544ec8356",
    "scope" : "tests",
    "src-rse" : "CERN-PROD-RUCIOTEST_DATADISK",
    "src-url" : "srm://srm-eosatlas.cern.ch:8443/srm/v2/server?SFN=/eos/atlas/atlasdatadisk/ruciotest/rucio/tests/cf/c4/ESD.72b4ecd77b414e6bb2684e1e7aa552af",
    "tool-id" : "rucio-conveyor",
    "transfer-endpoint" : "https://fts3-pilot.cern.ch:8446, https://fts3-pilot.cern.ch:8446",
    "transfer-id" : "70f7dbec-fab2-11e3-a569-02163e0078c8"
  }
}

transfer_failed = {
  "created_at" : "2014-06-23 04:07:25",
  "event_type" : "transfer-failed",
  "payload" : {
    "activity" : "rucio-integration",
    "checksum-adler" : "36a03645",
    "checksum-md5" : null,
    "dst-rse" : "NDGF-T1-MWTEST_DATADISK",
    "dst-url" : "srm://bunsen.ndgf.org:8443/srm/managerv2?SFN=/atlas/disk/atlasdatadisk/rucio/tests/c7/56/ESD.e08a880fc7514a91854d6d1594690070",
    "file-size" : 1048576,
    "guid" : "d11bd0aa41f84dc887b487f839892f55",
    "name" : "ESD.e08a880fc7514a91854d6d1594690070",
    "previous-request-id" : "d667ba2e11c744b8b4d8f284ec510e70",
    "protocol" : "srm",
    "reason" : "SOURCE User specified source file size is 1048576 but stat returned 1047856",
    "reason-link" : "https://fts3-pilot.cern.ch:8446, https://fts3-pilot.cern.ch:8446/fts3/ftsmon/#/job/8000bf02-fa83-11e3-b11f-02163e0078c8",
    "request-id" : "d667ba2e11c744b8b4d8f284ec510e70",
    "scope" : "tests",
    "src-rse" : "UNKNOWN",
    "src-url" : "srm://srm-eosatlas.cern.ch:8443/srm/v2/server?SFN=/eos/atlas/atlasdatadisk/ruciotest/rucio/tests/c7/56/ESD.e08a880fc7514a91854d6d1594690070",
    "tool-id" : "rucio-conveyor",
    "transfer-endpoint" : "https://fts3-pilot.cern.ch:8446, https://fts3-pilot.cern.ch:8446",
    "transfer-id" : "8000bf02-fa83-11e3-b11f-02163e0078c8"
  }
}

The following lists DQ2 file events and how they map to Rucio file events.

• File transfer events IMPLEMENTED
  • DQ2 file-bad is covered by Rucio transfer-failed.
  • DQ2 file-copied is covered by Rucio transfer-done.
  • DQ2 file-done is covered by Rucio transfer-done.
  • DQ2 file-transfer-error is covered by Rucio transfer-failed.
• File staging events PROPOSED
  • DQ2 file-staged will be covered by Rucio staging-done (Same as transfer-done but with null values for src-* fields).
  • DQ2 file-staging-error will be covered by Rucio staging-failed (Same as transfer-failed but with null values for src-* fields).
• File deletion events PROPOSED AND EXTENDED to cover file monitoring as in http://bourricot.cern.ch/dq2/deletion/
  • DQ2 file-removed will be covered by Rucio deletion-done (Content TBD)
  • Rucio deletion-planned (Content TBD)
  • Rucio deletion-failed (Content TBD)

The following lists DQ2 dataset events. We will review how to map these to Rucio events when file events are fully implemented.

• We could replace the existing dataset-content event as a Rucio attach/detach event:
  • dataset-content
• We could replace the existing subscription events with Rucio rule events:
  • subscription-broken
  • subscription-canceled
  • subscription-complete
  • subscription-queued

Notes

• 2014-03-31: I've added a "reason-link" field to file-bad and file-transfer-error, on the request of Alessandra Forti, in case it is possible to provide such a thing, see Savannah:89888.
• 2014-03-31: I have a further request to include protocol, such as FTS3, xrootd, ... Would that be possible? Appropriate?
• 2014-04-01: I have replaced "lfn" by scope and name throughout. I have made "activity" a string. I have replaced "checksum" by "checksum-adler", "checksum-md5", I have made "attempt-id" and "previous-attempt-id" UUIDs. I have added "activity", "checksum-adler", "checksum-md5", "file-size", "protocol" and "tool-id" throughout; these should be removed if not available for specific events.
• 2014-04-03: I have made the following changes following Mario's feedback:
  • Remove checksum from -removed, -done, -copied.
  • Remove filesize from -bad, -removed, -transfer-error.
  • Remove protocol from -removed.
  • Format of 'tool-id' will be something like: 'rucio-(dev|prod)-<ve.rs.ion>'.
• 2014-06-23: Update to include current Rucio event formats and recent discussions.

-- DavidTuckett - 14 Mar 2014