Storage Accounting

Definition of a storage accounting record. Due in PM 8:

A storage accounting record is defined (or a standardized schema like the OGF URs might be extended), reflecting practical, financial and legal requirements of storage location, usage and space and data flow.

Information from other communities

• OSG works with a software project Gratia which implements OGF UR for job accounting
  • Extends OGF UR with two storage accounting records, StorageElement (path, space token name, space type, etc.) and StorageElementRecord (dealing with measurement)
  • OSG is very much interested in having a standard for usage record – with a strong preference for extension of (and/or in the same spirit as) the OGF UR (since that is basically what they already do)
• DEISA
  • currently no direct interest or requirements in storage accounting, only CPU accounting, but DEISA is (passive) commenter in OGF UR-WG
  • John Kennedy (Munich) can provide us with information from iRODS developers about their accounting ideas
• Amazon S3 accounting record, is written hourly
  • Service (= S3)
  • Operation
    • the operation that was done like listing, reading, writing objects (called buckets on S3)
  • UsageType (see http://aws-portal.amazon.com/gp/aws/developer/common/amz-storage-usage-type-help.html)
    • category of usage data
    • some examples:
      • TimedStorage-ByteHrs: The amount of storage you've used over time. This is the raw data used to calculate your storage charges, and is useful if you are interested in auditing your bill. If you just want to track how much storage you are using on a daily basis, you should look at the AverageStorage-Bytes usage type instead. At least twice a day, we check to see how much storage is used by all your Amazon S3 buckets. The result is multiplied by the amount of time passed since the last checkpoint. For example, if you used 1024 bytes for 12 hours, that is 12,288 byte-hours.
      • DataTransfer-In-Bytes: This contains records of network data transfer into your Amazon S3 account from the outside Internet. This is the same data used to calculate your charges for network use at the end of the billing period. Every time a request is received to put an object or list a bucket, the amount of network traffic involved in transmitting the object data, metadata, or keys is recorded here. You can view the network usage associated with one of these operations by specifying the operation of choice before generating the usage report.
      • Requests-Tier1: This contains records of the number of PUT and LIST requests made to your Amazon S3 account for the given month.
  • StartTime and EndTime
  • UsageValue in the according unit (bytes or counts)
• iRODS
  • Accounting is a way to document the properties of a collection
    • to track how much is stored
    • assert current status of a collection
    • track changes to the collection
    • validate whether the collection meets assessment criteria
  • Files systems do not provide the context needed to validate assesment criteria - need to think in terms of collection
    • Accounting system for file system is useless to the data grid, digital library and preservation communities
  • Within a collection iRODS need to track
    • desired properties
    • policies that enforce each property
    • procedures that implement the policy
    • system state information that is generated by applying the policies
    • assessment criteria (policies that are run periodically)
    • audit trails on every operation
  • Accounting then corresponds to
    • queries on current state information for current status
    • parsing of audit trails to track compliance with policies over time
    • validation of assessment criteria
  • To iRODS data grid, accounting is the set of procedures composed from micro-services that implement these three functions for each policy

People

Jon, Paul, Riccardo Z., Zsolt, Ralph

Mailing list

emi-jra1-data-sar

Meetings and minutes

EMI JRA1.3

Documents

• StAR v1.2
• StAR v1.0
• CommentsOnStar
• CommentsOnStarOgfPublicComments

References

• OGF UR
• OGF storage accounting ideas
• Storage accounting record draft for OGF30
• Summary of SA discussions OGF28
• Gratia (OSG accounting)
• EGI Grid Policy for User-Level Job Accounting Data
• EGEE-1 accounting document
• EGI accounting policy doc
• StAR-EMI-tech-doc-v7.pdf: Storage accounting definition technical note
• StAR-EMI-tech-doc-v7.doc: Storage accounting definition technical note
• starval-0.2.tar.gz: Testsuite for validating StAR records in general and with EGI profile

-- JonKerrNilsen - 02-Sep-2010