ATLAS DDM Dashboard Rucio Migration

This page documents the migration of the ATLAS DDM Dashboard to support Rucio

Overview

The proposal is summarised in the following slides taken from the ATLAS DDM Dashboard presentation on the subject at the ATLAS Software & Computing Workshop in June 2014.

Rucio migration: Overview. Rucio migration: Status Rucio migration: Next steps Deletion monitoring

Message Format

The Dashboard and DDM teams have reviewed the current message format.

We have developed the message format for Rucio transfer-done and transfer-failed.

transfer_done = {
  "created_at" : "2014-06-23 08:43:32",
  "event_type" : "transfer-done",
  "payload" : {
    "activity" : "rucio-integration",
    "dst-rse" : "INFN-T1_DATADISK",
    "dst-url" : "srm://storm-fe.cr.cnaf.infn.it:8444/srm/managerv2?SFN=/atlas/atlasdatadisk/rucio/tests/cf/c4/ESD.72b4ecd77b414e6bb2684e1e7aa552af",
    "duration" : 4,
    "file-size" : 1048576,
    "guid" : "9e738f93050c4ba5b6c11722d160ce8b",
    "name" : "ESD.72b4ecd77b414e6bb2684e1e7aa552af",
    "previous-request-id" : null,
    "protocol" : "srm",
    "request-id" : "ed441fee68414c82b8d331b544ec8356",
    "scope" : "tests",
    "src-rse" : "CERN-PROD-RUCIOTEST_DATADISK",
    "src-url" : "srm://srm-eosatlas.cern.ch:8443/srm/v2/server?SFN=/eos/atlas/atlasdatadisk/ruciotest/rucio/tests/cf/c4/ESD.72b4ecd77b414e6bb2684e1e7aa552af",
    "tool-id" : "rucio-conveyor",
    "transfer-endpoint" : "https://fts3-pilot.cern.ch:8446, https://fts3-pilot.cern.ch:8446",
    "transfer-id" : "70f7dbec-fab2-11e3-a569-02163e0078c8"
  }
}

transfer_failed = {
  "created_at" : "2014-06-23 04:07:25",
  "event_type" : "transfer-failed",
  "payload" : {
    "activity" : "rucio-integration",
    "checksum-adler" : "36a03645",
    "checksum-md5" : null,
    "dst-rse" : "NDGF-T1-MWTEST_DATADISK",
    "dst-url" : "srm://bunsen.ndgf.org:8443/srm/managerv2?SFN=/atlas/disk/atlasdatadisk/rucio/tests/c7/56/ESD.e08a880fc7514a91854d6d1594690070",
    "file-size" : 1048576,
    "guid" : "d11bd0aa41f84dc887b487f839892f55",
    "name" : "ESD.e08a880fc7514a91854d6d1594690070",
    "previous-request-id" : "d667ba2e11c744b8b4d8f284ec510e70",
    "protocol" : "srm",
    "reason" : "SOURCE User specified source file size is 1048576 but stat returned 1047856",
    "reason-link" : "https://fts3-pilot.cern.ch:8446, https://fts3-pilot.cern.ch:8446/fts3/ftsmon/#/job/8000bf02-fa83-11e3-b11f-02163e0078c8",
    "request-id" : "d667ba2e11c744b8b4d8f284ec510e70",
    "scope" : "tests",
    "src-rse" : "UNKNOWN",
    "src-url" : "srm://srm-eosatlas.cern.ch:8443/srm/v2/server?SFN=/eos/atlas/atlasdatadisk/ruciotest/rucio/tests/c7/56/ESD.e08a880fc7514a91854d6d1594690070",
    "tool-id" : "rucio-conveyor",
    "transfer-endpoint" : "https://fts3-pilot.cern.ch:8446, https://fts3-pilot.cern.ch:8446",
    "transfer-id" : "8000bf02-fa83-11e3-b11f-02163e0078c8"
  }
}

The following lists DQ2 file events and how they map to Rucio file events.

  • File transfer events IMPLEMENTED
    • DQ2 file-bad is covered by Rucio transfer-failed.
    • DQ2 file-copied is covered by Rucio transfer-done.
    • DQ2 file-done is covered by Rucio transfer-done.
    • DQ2 file-transfer-error is covered by Rucio transfer-failed.
  • File staging events PROPOSED
    • DQ2 file-staged will be covered by Rucio staging-done (Same as transfer-done but with null values for src-* fields).
    • DQ2 file-staging-error will be covered by Rucio staging-failed (Same as transfer-failed but with null values for src-* fields).
  • File deletion events PROPOSED AND EXTENDED to cover file monitoring as in http://bourricot.cern.ch/dq2/deletion/
    • DQ2 file-removed will be covered by Rucio deletion-done (Content TBD)
    • Rucio deletion-planned (Content TBD)
    • Rucio deletion-failed (Content TBD)

The following lists DQ2 dataset events. We will review how to map these to Rucio events when file events are fully implemented.

  • We could replace the existing dataset-content event as a Rucio attach/detach event:
    • dataset-content
  • We could replace the existing subscription events with Rucio rule events:
    • subscription-broken
    • subscription-canceled
    • subscription-complete
    • subscription-queued

Notes

  • 2014-03-31: I've added a "reason-link" field to file-bad and file-transfer-error, on the request of Alessandra Forti, in case it is possible to provide such a thing, see Savannah:89888.
  • 2014-03-31: I have a further request to include protocol, such as FTS3, xrootd, ... Would that be possible? Appropriate?
  • 2014-04-01: I have replaced "lfn" by scope and name throughout. I have made "activity" a string. I have replaced "checksum" by "checksum-adler", "checksum-md5", I have made "attempt-id" and "previous-attempt-id" UUIDs. I have added "activity", "checksum-adler", "checksum-md5", "file-size", "protocol" and "tool-id" throughout; these should be removed if not available for specific events.
  • 2014-04-03: I have made the following changes following Mario's feedback:
    • Remove checksum from -removed, -done, -copied.
    • Remove filesize from -bad, -removed, -transfer-error.
    • Remove protocol from -removed.
    • Format of 'tool-id' will be something like: 'rucio-(dev|prod)-<ve.rs.ion>'.
  • 2014-06-23: Update to include current Rucio event formats and recent discussions.

-- DavidTuckett - 14 Mar 2014

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng _page3.png r1 manage 76.6 K 2014-06-23 - 14:36 DavidTuckett Rucio migration overview, status, plan. Deletion overview.
PNGpng _page4.png r1 manage 45.3 K 2014-06-23 - 14:36 DavidTuckett Rucio migration overview, status, plan. Deletion overview.
PNGpng _page6.png r1 manage 25.7 K 2014-06-23 - 14:36 DavidTuckett Rucio migration overview, status, plan. Deletion overview.
PNGpng _page7.png r1 manage 121.7 K 2014-06-23 - 14:36 DavidTuckett Rucio migration overview, status, plan. Deletion overview.
Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r8 - 2014-06-23 - DavidTuckett
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    ArdaGrid All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback