StorageSetUp

Introduction

This page is describing the storage organisation for DDM aware sites. It is orientied for site admin or squad support. T3s without DDM components are described in AtlasTier3

Requirements on SE functions

protocols
srmv2 , xrootd , WebDAV
  • ATLAS uses srmv2 protocol to store files in SE
  • xrootd : for local and remote access, read-only to start with, read/write eventually
  • WebDAV : initially for Storage Management operations for the Rucio migration during 2013 (rw) , eventually for local and remote access (rw)

space reservation
ATLAS uses the space tokens described below (see the section Space Tokens)

checksum
ATLAS uses adler32 to verify files

Protocols

srmv2

  • ATLAS uses srmv2 protocol but is being depreciated in 2019:
    • by jobs to get and store files in SE
    • through FTS for 3 rd party transfer
    • to utilize space reservation with space tokens (see below #Space_Tokens and #Space_Reservation
    • to stage files from tape via srmBringOnline

xrootd

  • ATLAS uses the xrootd protocol:
    • For WAN data access (r/o)
    • For LAN data access whenever proved to be more performing than the other available protocols (r/o)
    • As protocol for federating storages through FAX (r/o)

WebDAV

  • From 2013 ATLAS uses HTTP/WebDAV
    • For LAN data access whenever proved to be more performing than the other available protocols (r/o)
    • For testing as Storage Management Protocol (replacing some SRM functionalities) (rw)
    • For testing as protocol for LAN and WAN transfers (e.g. FTS and dq2-get/put) (rw)

Space Tokens or quota tokens

Description

_ATLAS Space Tokens for Data Storage DDM aware sites_
Space Token Storage Type Used For @T0 @T1 @T2 @T3 Comments
ATLASLOCALGROUPTAPE T1D0 Local User Data o o o o  
ATLASDATATAPE T1D0 RAW from T0, AOD from re-processing X X      
ATLASDATADISK T0D1 AOD + data on demand X X X o  
ATLASMCTAPE T1D0 Custodial copy of MC files   X      
ATLASCALIBDISK T0D1 Files for detector calibration a a a a  
ATLASPRODDISK T0D1 buffer for central production       o Only used in T3 sites. DATADISK is used for all other sites and new T3s
ATLASGROUPDISK T0D1 Data managed by groups a a a a Managed by group space managers
ATLASSCRATCHDISK T0D1 Temp. user Data X X X o Mandatory to host Grid analysis outputs
ATLASLOCALGROUPDISK T0D1 Local User Data o o o o  

  • X : Space token is mandatory
  • o : Space token is optional and decided by site
  • a : Space token request is validated by CREM

Each space token is associated to a path on the SE.

SP : Space token in capital letter
sp : Space token in small letter

Space token name : SP
Associated path : se_path/sp/...

Nota Bene 1
The SCRATCHDISK area in US T2s are used as scratch to host temporary datasets transfered from other sites.

Nota Bene 2
The list of sites serving each perf-phys group (ATLASGROUPDISK) is defined by ATLAS. More informations at GroupsOnGrid.

Nota Bene 3
All space tokens contribute to the pledged storage resources except ATLASLOCALGROUPDISK and ATLASLOCALGROUPTAPE

Nota Bene 4
ATLASCALIBDISK is deployed only on demand from detector groups which want to run prompt calibration in these sites.

LOCALGROUPTAPE
DDM Endpoint with TAPE backend, used for local interest, no pledge, not necessarily at T1
  • Set up on request by a site, and accepted as far as DDM-ops does not get an “extra load”
  • Setup should follow the standard ones similar to DATATAPE/MCTAPE + LOCALGROUPDISK
    • Non-standard setup can be discussed, though may not be supported
  • presented and agreed at ICB on 12 September 2013

Space Reservation

Tier-3

  • ATLASSCRATCHDISK (for analysis outputs)
    • for each job slot dedicated to Grid analysis : 50 GB, with a minimum of 1 TB
  • ATLASDATADISK (as production buffer)
    • for each job slot dedicated to MC production : 20 GB, with a minimum of 0.5 TB
  • ATLASLOCALGROUPDISK: to import data. All the rest.
  • ATLASPRODDISK is gradually being phased out and replaced by ATLASDATADISK. It is recommended that new sites use ATLASDATADISK rather than ATLASPRODDISK.

Tier-2

The Tier2s hosts Grid analysis and production jobs. Tier2s can be grouped in federations. The pledge resources are defined per federation. For ATLAS is important that each site provide appropriate disk space, not to have too much fragmented disks.

Required Space Tokens

  • ATLASDATADISK:
  • ATLASSCRATCHDISK:
    • 100TB per 1k analysis job slots (based on calculation of average job output)

Tier-1

Required Space Tokens

  • ATLASDATADISK: size according to MoU
  • ATLASSCRATCHDISK: 100TB per 1k analysis job slots
  • ATLASDATATAPE, ATLASMCTAPE: size according to MoU
  • Tape staging buffers
    • Can be shared or split over the space tokens
    • Recommended size: enough to sustain the required rate of reading/writing, a rough rule of thumb is between 10 and 20TB per PB of data stored

Tier-0

See ATLASStorageAtCERN

ACLs for DDM endpoint and space token/storage

Why different acls between DDM and storage ?

The acls in space tokens are implemented by sites while the DDM acls are managed by ATLAS team. The DDM acls should be more or equaly restrictive than the space token ones. The acls in space tokens are defined to avoid to bypass DDM acls to read/write a file.

DDM acls:

  • atlas/role=production is able to do any action on any DDM enpoint (including ATLASLOCALGROUPDISK)
  • atlas/country/role=production is able to do any action on any LOCALGROUPDISK in the same country
  • atlas/group/role=production is able to do any action only on the associated DDM endpoints.
ATLASLOCALGROUPDISK Yes Yes (for users in the country)
 
Space Token Read Access for user Write Access for user
ATLASDATADISK Yes No
ATLASDATATAPE No No
ATLASMCTAPE No No
ATLASCALIBDISK Yes No
ATLASPRODDISK Yes No
ATLASGROUPDISK Yes No
ATLASSCRATCHDISK Yes Yes

Some US Tier3s have decided to forbid read access to their LOCALGROUPDISK.

Space token acls

Not all the storage implementation at the moment are fully VOMS aware and not all storage implementations at the moment support ACLs. After discussion with developers and experts, those are the recommendations for setting up various ATLAS space tokens concerning groups and users, remembering the following limitation on the currently deployed storage solutions:

N.B. In dCache, one can bound (one or more) directory trees to a space token. Therefore we ask sites to bound the space tokens to the corresponding path in AGIS. Contact the ATLAS contact at the cloud if you need clarifications on what this means.

Recommendation

The following table presents the acls which ATLAS would like to setup. It is coherent for the acls implemented in DDM. The acls should be identical for the space-token and the associated storage path. As stated in the introduction, effective acls will not be exactly the same.

ATLAS Space Tokens
Space Token atlas/Role=production atlas/Role=pilot atlas/Role=NULL atlas/<country>/Role=production atlas/<country>/Role=NULL
ATLASLOCALGROUPTAPE Read/Write Read Read Read/Write Read
ATLASDATATAPE Read/Write *No Access* *No Access* *No Access* *No Access*
ATLASDATADISK Read/Write Read Read Read Read
ATLASMCTAPE Read/Write *No Access* *No Access* *No Access* *No Access*
ATLASPRODDISK Read/Write *No Access* *No Access* *No Access* *No Access*
ATLASGROUPDISK Read/Write Read Read Read Read
ATLASCALIBDISK Read/Write Read Read Read Read
ATLASSCRATCHDISK Read/Write Read/Write Read/Write Read/Write Read/Write
ATLASLOCALGROUPDISK Read/Write Read Read Read/Write Read

  • Write access for atlas/country to ATLASLOCALGROUPDISK/ATLASLOCALGROUPTAPE is not recommended (to avoid the creation of dark data in this area) through rucio upload. Data are to be put into those spaces via R2D2

Space token acl implementation

The ATLASDATADISK/ATLASMCTAPE/ATLASDATATAPE/ATLASPRODDISK/ATLASCALIBDISK

  • In case of CASTOR: this will be handled on a case by case basis
  • In case of DPM: configure the space token to be owned only by atlas/Role=production. Configure the namespace area to be owned by the atlas/Role=production VOMS FQAN.
  • In case of dCache: configure both the space token and the corresponding namespace path to allow rwx to everyone.

The ATLASSCRATCHDISK

Permissions should be rwx for every user both at the level of Space Token and namespace.

The ATLASGROUPDISK

Permissions should be identical to ATLASDATADISK since groups are not allowed to write directly datasets to the DDM endpoint.

The ATLASLOCALGROUPDISK

In principle ATLAS should not discuss any requirement on this, since it is not pledged resources. Anyway, here are some suggestions:

  • In case of CASTOR: this will be handled on a case by case basis
  • In case of DPM: configure the space token to be owned by atlas to allow rwx to everyone (DPM < 1.7.0), or owned by atlas/Role=production and the local group (usually atlas/country). Configure the namespace to grant rwx privileges to the local group (in VOMS) and the atlas/Role=production VOMS FQAN (this is needed to serve those sites via DDM). These privileges should be granted with default ACLs to ensure that permissions propagate correctly.
  • In case of dCache: configure both the space token and the corresponding namespace path to allow rwx to everyone.

Deletion policy per DDM endpoint

All files in space tokens managed by DDM are deleted through central deletion.

ATLASDATADISK

A watermark of free space is set to min(10%, 300TB). When the space goes below this limit, unlocked (secondary) data will be deleted until the limit is reached.

ATLASCRATCHDISK

A watermark of free space is set to min(50%, 30TB). When the space goes below this limit, unlocked (secondary) data will be deleted until the limit is reached.

ATLASPRODDISK

A watermark of free space is set to min(25%, 20TB). When the space goes below this limit, unlocked (secondary) data will be deleted until the limit is reached.

ATLASGROUPDISK/ATLASLOCALGROUPDISK

The deletion is triggered by the responsible persons.


Major updates:
-- KorsBos - 06 Jul 2008 -- DavidCameron - 2015-09-15

Responsible: DavidCameron Last reviewed by: Never reviewed

Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf 20140227_ADCOpsSiteClassification_rev01.pdf r1 manage 512.3 K 2015-11-03 - 11:01 AleDiGGi  
Unix shell scriptsh atlas-group-disk-dpm.sh r3 r2 r1 manage 1.7 K 2008-09-09 - 12:21 UnknownUser Setup script for ATLASGROUPDISK for sites using DPM
Edit | Attach | Watch | Print version | History: r105 < r104 < r103 < r102 < r101 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r105 - 2019-07-29 - StephaneJezequel
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Atlas All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback