Joint Data Management / Storage Management report


  • [ 13 Apr 2012] v.4.0 of Report and Summary. Sent to Ian B. as "final" though recognising that there is further work to be done in some areas.
  • [ 06 Apr 2012] v.2.1 of Report and Summary. Also sent to Ian B. Please comment ahead of final version to be compiled next week.
  • [ 03 Apr 2012] v.1.9 of the report has been updated. Rewordered / compactified recomendations. Please use main report for recomendation text at present. Summary document will be updated when others have checked recomendations tomorrow.
  • [ 02 Apr 2012] v.1.6 of the report has been updated. CMS comments by Daniele on the SRM table are posted below, also.
  • [ 29 Mar 2012] Two DRAFT updated documents attached: report and summary v.1.3
  • [ 28 Mar 2012] Two DRAFT documents attached: report v.1.2 and Summary of recommendations


This area is for your comments. Please edit a section with your name and fill it.


MOVED TO... Transfer Management

Upload / download a complete file (SRM: srmPrepareToPut/Get, srmPut/GetDone)

  • CMS: yes, all Tiers
Manage transfers (SRM: 5.11. srmAbortRequest, 5.15. srmGetRequestSummary)
  • CMS uses it, if FTS does. CMS basically always the clients, via FTS or lcg-cp or srmcp, so it depends on what the client does. BTW, also the T3s could be included in the list, e.g. take T3->T1 uploads, i.e. the Tier level here is not so relevant, in the sense that any Tier configured to use FTS has the same answer (note: CMS does not explicitly does abort of the transfers in PhEDEx, but it's done by hand when we need to clean-up in FTS).
Permit load-balance client requests over multiple transfer servers (SRM: srmPrepareToGet)
  • CMS uses it
Back-pressure (SE tells client to slow down or queues clients) (SRM: no)
  • it actually exists in the SRM specs (maybe nothing usable/efficient/performing was ever implemented?). Also, at the practical level it is actually done by FTS by setting limits on the channels. It would be useful for CMS to have this function as well: PhEDEx actually optimizes the transfers from the destination standpoint, it has a reaction mechanism by which it detects if the outbound rate from the source is too high or saturated and looks for another source, but of course it is not successful if that's the only source. So, having this function could be useful to react better on the status of the source (also by tuning transfer to just better exploit one single source).
Manage third-party copy
  • CMS uses it. SRM must support the 3rd part copies done by FTS, lcg-cp, etc.
Negotiating a transport protocol that client and server support (SRM: srmGetTransferProtocols, srmPrepareToGet/Put)
  • all experiments use gsiftp, no negotiations. Maybe FTS internally does srmGetTransferProtocols and then selected (hardcoded) gsiftp? Anyway, this is not explicitly used by CMS. Maybe LHCb uses this for the local stageout?

MOVED TO... Namespace interaction

Querying information about a file (stat) (SRM: srmLs)

  • CMS uses it, since FTS does. And CMS uses it for the post-transfer checks in case we go through SRM. E.g., once SRM reports "transfer completed" we go and check anyway that the file is the destination place, with the right size and cksum.
Upload of data integrity information (checksums) (SRM: srmSetPermission)
  • CMS does not use it.
Check integrity information (SRM: srmLs?)
  • The cksum is computed by CMS via gridftp during the transfer, amnd/or at a later stage with lcg-get-checksum. The cksum is then read using srmLs. So, CMS uses srmLs, which returns the cksum computed at the time the file was written.
Creating/Deleting data and directories (SRM: srmMkdir, srmRmdir, srmRm, srmMv)
  • CMS uses most of them, and some more details can be found in the following. srmMkdir is used, both explicitly and via lcg-cp or FTS. srmRmdir is generally not used, (but sites can do that when they want/need to clean empty directories). srmRm is used at the sites in which the PhEDEx FileRemove agent is configured to use SRM; it is used also via FTS, in case of failures; additionally, it is used also for the stageout (in case of failure/clean-up). srmMv is not used (in principle, sites could use it internally to migrate files among directories on the storage, but it is risky and it's not a CMS-driven or CMS-recommended action to take).
Changing ownership, permissions and ACLs (SRM: ?)
  • CMS does not use it.
Changing permissions and ACLs (SRM: srmSetPermission, srmCheckPermission, srmGetPermission)
  • CMS does not use it. As PhEDEx agents run locally, the permissions can be set by the local site admin. Probably this is more needed by other experiments.

MOVED TO... Storage Capacity Management

Ability to query how much capacity is being used (like df) (SRM: srmGetSpaceMetaData, srmGetSpaceTokens)

  • CMS does not use it. Read above.
Create or remove reservations; assign characteristics like latency and retention (SRM: srmReserveSpace, srmReleaseSpace, srmUpdateSpace)
  • CMS does not use it. CMS can set the latency/retention policies at an endpoint level; does not need to be at the space reservation level. In a TFC you can put (logically) if custodial than go to SRM A else go to SRM B, and so on.
Targeting uploads to specific reservation (SRM: srmPrepareToPut)
  • if "specific reservations" mean the preconfigured static space tokens, then it is used by FTS, so it's also used by CMS. If it means the possibility to reserve some space on the flight, CMS is currently not using it. It could have implications on the archive vs caches business, e.g. for T1D0<->T0D1 traffic (see also here, WLCG explicitly excluded the T1D0<->T0D1use-case).

MOVED TO... File Locality Management

Non-blocking trigger that files should be staged from archive (SRM: srmBringOnline; srmStatusOfBringOnlineRequest)

  • CMS uses it.
Guarantee that a file will not be garbage collected for a finite period (pinning)
  • CMS would profit of it, if available in all implementations
Specify when a file is eligible for garbage collected (cancel pin)
  • same as above
Server identification Allow simple test of service availability (SRM: srmPing)
  • CMS does not use it in PhEDEx, but also FTS uses it, and sometimes for manual controls it is used.
Allow discovery of arbitrary information about the server (SRM: srmPing and others)
  • Same as above.

Known todo items

done in 4.0
  • Check for formatting problems (e.g. line-spacing); Update other x-refs
  • Update Summary of Recomendations - put in security ones.
  • Federation Recommendation 3: Topical working groups: Do we make this more generic? if so where does it go.
  • Add links to collected information (experiment feedback; questionairres (not done); middleware talks at face-to-face)
  • SRM table - formatting (too wide when printed) and still some question marks
  • SRM table - draw out some key points for the text
  • Make a one liner Summary of Summary of Recommendations for the front with links to page numbers.
  • Michel's comments; xrootd support recomendation reworded; main title change; All other comments / undecided removed; Atlas answers in federation table (with in-discussion caveat.

not done
  • Make nicer pdfs (with x-ref hyperlinks)
  • Remove Appendix A and update references accordingly
  • Integrate or reference better storage security text (currently twiki link)
  • Move "Summary of Recommendations" to front of doc; Shorten some of the recomendations (there) and make it a proper exec. summary.
  • DM recomendations appear embedded in into text while SM ones are in seperate topics - ideally this should be made consistent
  • Federation Recommendation 3: Topical working groups: Do we make this more generic? if so where does it go.
  • Add timescales ; priorities and effort to Summary of Summary

-- DanieleBonacorsi - Mar-2012

Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf DataandStorageTEG-DRAFT-Report-v1.2.pdf r1 manage 1155.2 K 2012-03-28 - 17:57 DanieleBonacorsi DRAFT document (v1.2) - 28 March 2012 - 18:00
PDFpdf DataandStorageTEG-DRAFT-Report-v1.3.pdf r1 manage 1155.7 K 2012-03-29 - 17:39 WahidBhimji  
PDFpdf DataandStorageTEG-DRAFT-Report-v1.6_public.pdf r1 manage 1182.3 K 2012-04-02 - 15:11 DanieleBonacorsi DRAFT document (v1.6) - 2 April 2012 - 15:00
PDFpdf DataandStorageTEG-DRAFT-Report-v1.9.pdf r1 manage 1205.2 K 2012-04-03 - 19:33 WahidBhimji  
PDFpdf DataandStorageTEG-DRAFT-SummaryOfRecomendations-v1.3.pdf r1 manage 90.4 K 2012-03-29 - 17:27 WahidBhimji  
PDFpdf DataandStorageTEG-DRAFT-SummaryofRecommendations.pdf r1 manage 130.6 K 2012-03-28 - 17:58 DanieleBonacorsi DRAFT summary of recommendations - 28 March 2012 - 18:00
PDFpdf DataandStorageTEG-Report-v2.1.pdf r1 manage 1198.3 K 2012-04-06 - 15:51 WahidBhimji  
Unknown file formatdocx DataandStorageTEG-Report-v4.0.docx r1 manage 996.8 K 2012-04-13 - 15:43 WahidBhimji  
PDFpdf DataandStorageTEG-Report-v4.0.pdf r1 manage 1260.4 K 2012-04-13 - 15:44 WahidBhimji  
PDFpdf DataandStorageTEG-SummaryofRecommendationsv2.1.pdf r1 manage 83.1 K 2012-04-06 - 15:52 WahidBhimji  
PDFpdf DataandStorageTEG-SummaryofRecommendationsv4.0.pdf r1 manage 83.5 K 2012-04-13 - 15:45 WahidBhimji  
Edit | Attach | Watch | Print version | History: r16 < r15 < r14 < r13 < r12 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r16 - 2012-04-13 - WahidBhimji
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback