Storage Accounting

Definition of a storage accounting record. Due in PM 8:

A storage accounting record is defined (or a standardized schema like the OGF URs might be extended), reflecting practical, financial and legal requirements of storage location, usage and space and data flow.

Information from other communities

  • OSG works with a software project Gratia which implements OGF UR for job accounting
    • Extends OGF UR with two storage accounting records, StorageElement (path, space token name, space type, etc.) and StorageElementRecord (dealing with measurement)
    • OSG is very much interested in having a standard for usage record with a strong preference for extension of (and/or in the same spirit as) the OGF UR (since that is basically what they already do)
  • DEISA
    • currently no direct interest or requirements in storage accounting, only CPU accounting, but DEISA is (passive) commenter in OGF UR-WG
    • John Kennedy (Munich) can provide us with information from iRODS developers about their accounting ideas
  • Amazon S3 accounting record, is written hourly
    • Service (= S3)
    • Operation
      • the operation that was done like listing, reading, writing objects (called buckets on S3)
    • UsageType (see http://aws-portal.amazon.com/gp/aws/developer/common/amz-storage-usage-type-help.html)
      • category of usage data
      • some examples:
        • TimedStorage-ByteHrs: The amount of storage you've used over time. This is the raw data used to calculate your storage charges, and is useful if you are interested in auditing your bill. If you just want to track how much storage you are using on a daily basis, you should look at the AverageStorage-Bytes usage type instead. At least twice a day, we check to see how much storage is used by all your Amazon S3 buckets. The result is multiplied by the amount of time passed since the last checkpoint. For example, if you used 1024 bytes for 12 hours, that is 12,288 byte-hours.
        • DataTransfer-In-Bytes: This contains records of network data transfer into your Amazon S3 account from the outside Internet. This is the same data used to calculate your charges for network use at the end of the billing period. Every time a request is received to put an object or list a bucket, the amount of network traffic involved in transmitting the object data, metadata, or keys is recorded here. You can view the network usage associated with one of these operations by specifying the operation of choice before generating the usage report.
        • Requests-Tier1: This contains records of the number of PUT and LIST requests made to your Amazon S3 account for the given month.
    • StartTime and EndTime
    • UsageValue in the according unit (bytes or counts)
  • iRODS
    • Accounting is a way to document the properties of a collection
      • to track how much is stored
      • assert current status of a collection
      • track changes to the collection
      • validate whether the collection meets assessment criteria
    • Files systems do not provide the context needed to validate assesment criteria - need to think in terms of collection
      • Accounting system for file system is useless to the data grid, digital library and preservation communities
    • Within a collection iRODS need to track
      • desired properties
      • policies that enforce each property
      • procedures that implement the policy
      • system state information that is generated by applying the policies
      • assessment criteria (policies that are run periodically)
      • audit trails on every operation
    • Accounting then corresponds to
      • queries on current state information for current status
      • parsing of audit trails to track compliance with policies over time
      • validation of assessment criteria
    • To iRODS data grid, accounting is the set of procedures composed from micro-services that implement these three functions for each policy

People

Jon, Paul, Riccardo Z., Zsolt, Ralph

Mailing list

emi-jra1-data-sar

Meetings and minutes

EMI JRA1.3

Documents

References

-- JonKerrNilsen - 02-Sep-2010

Topic attachments
I Attachment History Action Size Date Who Comment
PowerPointppt EMI-storage_accounting_egitf2010v2.ppt r1 manage 1205.5 K 2011-06-14 - 10:20 JonKerrNilsen StAR presentation at EGITF2010
PDFpdf EMI_StAR_isgc2011.pdf r1 manage 1007.6 K 2011-06-14 - 10:34 JonKerrNilsen StAR presentation at ISGC2011
Microsoft Word filedoc StAR-EMI-tech-doc-v7.doc r1 manage 317.0 K 2011-02-24 - 12:03 JonKerrNilsen Storage accounting definition technical note (doc)
PDFpdf StAR-EMI-tech-doc-v7.pdf r1 manage 629.0 K 2011-02-24 - 12:02 JonKerrNilsen Storage accounting definition technical note
Unknown file formatpptx emi-star-egiuf2011.pptx r1 manage 1283.8 K 2011-06-14 - 11:04 JonKerrNilsen StAR presentation at EMI TF/EGI UF 2011
Unknown file formatgz starval-0.2.tar.gz r1 manage 12.9 K 2012-11-16 - 16:08 JonKerrNilsen Testsuite for validating StAR records in general and with EGI profile
Edit | Attach | Watch | Print version | History: r22 < r21 < r20 < r19 < r18 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r22 - 2012-11-16 - JonKerrNilsen
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EMI All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback