Data model based on AliEn File Catalog

  • presented at the WLCG Data and Storage Management Jamboree (June 2010)
  • presented at the ATLAS Rucio meeting on Dec 9, 2011

Answers to questions

  • some complexity may be removed
    • automatic deletion on expiration: never used
    • simpler schema for LFN to SFN mapping
  • some complexity may get added: new functionality
  • elements:
    • AliEn File Catalog
    • Xrootd
    • xrd3cp
    • MonALISA hooks for transfers + monitoring
  • they are maintainable by just a few experts
  • FTS usage blockers:
    • lack of Xrootd support
    • AliEn security model (envelope) support would be needed
    • automatic catalog updates would also be needed
  • federation with remote fallback: in use since years
    • through ROOT's TAlien (extends TGrid)
    • adequate network performance is necessary
    • SEs should refuse new connections to overloaded servers
    • if an existing connection has bad performance, the client should fail over to another replica
  • cluster file systems: applications should only see Xrootd interface
    • the site could use such a file system behind the scenes
  • cloud file systems: hide them behind Xrootd
    • files read-only, no updates: OK
    • PFN renaming not needed
  • space management:
    • SRM spaces not used
    • no reservations: fail over to other SE as needed
    • occasional cleanup of dark data
  • HTTP access:
    • framework supports it, but applications do not
      • development would be possible
  • tape (custodial storage) only needed for critical data
    • used as backup, data is explicitly staged in again when needed
    • an SE either is fully custodial or just disk
      • custodial SEs can only be written to by privileged workflows
    • disk space recovery: reduce the number of replicas of unpopular files
  • data placement strategy decided by:
    • number of replicas for a particular file
      • 1 copy usually is stored at the site where the job runs
    • availability of SEs
      • also ranked by network performance
  • caching
    • in some respect used by analysis trains that combine and serialize workflows on a given input data set plus intermediate results
    • currently no plans for on-demand placement and local caching of event data
    • conditions data is cached in an area defined by each site
      • typically per WN
  • VOMS: rather limited use, currently OK
  • name space: AliEn File Catalog will continue to hold "everything"
  • access control: AliEn security envelope model
    • no need for user proxies on the WN

-- MaartenLitmaath - 17-Nov-2011

Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf AliEn_FC_2011Dec9.pdf r1 manage 573.0 K 2011-12-09 - 19:12 MaartenLitmaath AliEn File Catalogue presented at ATLAS Rucio meeting Dec 9, 2011
Unknown file formatpptx AliEn_FC_2011Dec9.pptx r1 manage 765.4 K 2011-12-09 - 19:16 MaartenLitmaath AliEn File Catalogue presented at ATLAS Rucio meeting Dec 9, 2011
Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r3 - 2011-12-09 - MaartenLitmaath
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback