Data model based on AliEn File Catalog

• presented at the WLCG Data and Storage Management Jamboree (June 2010)
  • PPT
  • PDF
• presented at the ATLAS Rucio meeting on Dec 9, 2011
  • PPT
  • PDF

Answers to questions

• some complexity may be removed
  • automatic deletion on expiration: never used
  • simpler schema for LFN to SFN mapping
• some complexity may get added: new functionality
• elements:
  • AliEn File Catalog
  • Xrootd
  • xrd3cp
  • MonALISA hooks for transfers + monitoring
• they are maintainable by just a few experts
• FTS usage blockers:
  • lack of Xrootd support
  • AliEn security model (envelope) support would be needed
  • automatic catalog updates would also be needed
• federation with remote fallback: in use since years
  • through ROOT's TAlien (extends TGrid)
  • adequate network performance is necessary
  • SEs should refuse new connections to overloaded servers
  • if an existing connection has bad performance, the client should fail over to another replica
• cluster file systems: applications should only see Xrootd interface
  • the site could use such a file system behind the scenes
• cloud file systems: hide them behind Xrootd
  • files read-only, no updates: OK
  • PFN renaming not needed
• space management:
  • SRM spaces not used
  • no reservations: fail over to other SE as needed
  • occasional cleanup of dark data
• HTTP access:
  • framework supports it, but applications do not
    • development would be possible
• tape (custodial storage) only needed for critical data
  • used as backup, data is explicitly staged in again when needed
  • an SE either is fully custodial or just disk
    • custodial SEs can only be written to by privileged workflows
  • disk space recovery: reduce the number of replicas of unpopular files
• data placement strategy decided by:
  • number of replicas for a particular file
    • 1 copy usually is stored at the site where the job runs
  • availability of SEs
    • also ranked by network performance
• caching
  • in some respect used by analysis trains that combine and serialize workflows on a given input data set plus intermediate results
  • currently no plans for on-demand placement and local caching of event data
  • conditions data is cached in an area defined by each site
    • typically per WN
• VOMS: rather limited use, currently OK
• name space: AliEn File Catalog will continue to hold "everything"
• access control: AliEn security envelope model
  • no need for user proxies on the WN

-- MaartenLitmaath - 17-Nov-2011