StorageSetUp

Introduction

This page is describing the storage organisation for DDM aware sites. It is oriented for site admin or squad support. T3s without DDM components are described in AtlasTier3

Requirements on SE functions

space reservation

ATLAS uses the space tokens described below (see the section Space Tokens)

checksum

ATLAS uses adler32 to verify files

Protocols

GridFTP

• ATLAS uses the GridFTP protocol but work is ongoing to replace it:
  • for jobs to get and store files in SE -> xrootd/~WebDAV is preferred
  • through FTS for 3rd party transfer -> currently testing xrootd/~WebDAV as a replacement

SRM

• SRM is no longer required for disk-only storage. It can be used for:
  • jobs to get and store files in SE
  • through FTS for 3rd party transfer
  • to utilize space reservation with space tokens (see below #Space_Tokens and #Space_Reservation
  • to stage files from tape via srmBringOnline
• Apart from tape staging, all other functionality can be provided through other protocols

xrootd

• ATLAS uses the xrootd protocol:
  • For LAN data access whenever proved to be more performing than the other available protocols (r/o)
  • For occasional WAN data access (r/o)
  • For testing as 3rd party transfer protocol (r/w)

WebDAV

• ATLAS uses HTTP/WebDAV
  • For LAN data access whenever proved to be more performing than the other available protocols (r/o)
  • For testing as 3rd party transfer protocol (r/w)
  • Deletion

• Technical details can be found under WebDAVSetupForATLAS
• The deployment is followed in in WLCG by the HTTP deployment task force

Space Tokens or quota tokens

Description

• X : Space token is mandatory
• o : Space token is optional and decided by site
• a : Space token request is validated by CREM

Each space token is associated to a path on the SE.

SP : Space token in capital letter
sp : Space token in small letter

Space token name : SP
Associated path : se_path/sp/...

Nota Bene 1
The list of sites serving each perf-phys group (ATLASGROUPDISK) is defined by ATLAS. More information at GroupsOnGrid.

Nota Bene 2
All space tokens contribute to the pledged storage resources except ATLASLOCALGROUPDISK and ATLASLOCALGROUPTAPE

Nota Bene 3
ATLASCALIBDISK is deployed only on demand from detector groups which want to run prompt calibration in these sites.

LOCALGROUPTAPE

DDM Endpoint with TAPE backend, used for local interest, no pledge, not necessarily at T1

• Set up on request by a site, and accepted as far as DDM-ops does not get an “extra load”
• Setup should follow the standard ones similar to DATATAPE/MCTAPE + LOCALGROUPDISK
  • Non-standard setup can be discussed, though may not be supported

Space Reservation

Tier-3

• ATLASSCRATCHDISK (for analysis outputs if site runs analysis jobs)
  • for each job slot dedicated to Grid analysis : 50 GB, with a minimum of 1 TB
• ATLASDATADISK (as production buffer)
  • for each job slot dedicated to MC production : 20 GB, with a minimum of 0.5 TB

Tier-2

The Tier2s host Grid analysis and production jobs. Tier2s can be grouped in federations. The pledge resources are defined per federation. For ATLAS it is important that each site provides appropriate disk space, in order that disk space is not too fragmented.

Required Space Tokens

• ATLASDATADISK:
  • 460 TB as bare minimum for each site (i.e. not the whole federation). (taken from Site Jamboree March 2018)
• ATLASSCRATCHDISK:
  • 100TB per 1k analysis job slots (based on calculation of average job output)

Tier-1

Required Space Tokens

• ATLASDATADISK: size according to MoU
• ATLASSCRATCHDISK: 100TB per 1k analysis job slots
• ATLASDATATAPE, ATLASMCTAPE: size according to MoU
• Tape staging buffers
  • Can be shared or split over the space tokens
  • Recommended size: enough to sustain the required rate of reading/writing, a rough rule of thumb is between 10 and 20TB per PB of data stored

Tier-0

See ATLASStorageAtCERN

ACLs for DDM endpoint and space token/storage

Why different acls between DDM and storage ?

The acls in space tokens are implemented by sites while the DDM acls are managed by ATLAS team. The DDM acls should be more or equally restrictive than the space token ones. The acls in space tokens are defined to avoid to bypass DDM acls for reading or writing files.

DDM acls:

• atlas/role=production is able to do any action on any DDM endpoint (including ATLASLOCALGROUPDISK)
• atlas/country/role=production is able to do any action on any LOCALGROUPDISK in the same country
• atlas/group/role=production is able to do any action only on the associated DDM group disk endpoints.

Some US Tier3s have decided to forbid read access to their LOCALGROUPDISK.

Space token acls

Not all the storage implementations at the moment are fully VOMS aware and not all storage implementations at the moment support ACLs. After discussion with developers and experts, those are the recommendations for setting up various ATLAS space tokens concerning groups and users, remembering the following limitation on the currently deployed storage solutions:

N.B. In dCache, one can bound (one or more) directory trees to a space token. Therefore we ask sites to bound the space tokens to the corresponding path in AGIS. Contact the ATLAS contact in your cloud if you need clarifications on what this means.

Recommendation

The following table presents the acls which ATLAS would like to setup. It is coherent with the acls implemented in DDM. The acls should be identical for the space-token and the associated storage path. As stated in the introduction, effective acls will not be exactly the same.

• Write access for atlas/country to ATLASLOCALGROUPDISK/ATLASLOCALGROUPTAPE is not recommended (to avoid the creation of dark data in this area) through rucio upload. Data are to be put into those spaces via R2D2

Space token acl implementation

ATLASDATADISK/ATLASMCTAPE/ATLASDATATAPE/ATLASPRODDISK/ATLASCALIBDISK

• In case of DPM: configure the space token to be owned only by atlas/Role=production. Configure the namespace area to be owned by the atlas/Role=production VOMS FQAN.
• In case of dCache: configure both the space token and the corresponding namespace path to allow rwx to everyone.

ATLASSCRATCHDISK

Permissions should be rwx for every user both at the level of Space Token and namespace.

ATLASGROUPDISK

Permissions should be identical to ATLASDATADISK since groups are not allowed to write directly datasets to the DDM endpoint.

ATLASLOCALGROUPDISK

In principle ATLAS should not discuss any requirement on this, since it is not pledged resources. Anyway, here are some suggestions:

• In case of DPM: configure the space token to be owned by atlas to allow rwx to everyone (DPM < 1.7.0), or owned by atlas/Role=production and the local group (usually atlas/country). Configure the namespace to grant rwx privileges to the local group (in VOMS) and the atlas/Role=production VOMS FQAN (this is needed to serve those sites via DDM). These privileges should be granted with default ACLs to ensure that permissions propagate correctly.
• In case of dCache: configure both the space token and the corresponding namespace path to allow rwx to everyone.

Deletion policy per DDM endpoint

All files in space tokens managed by DDM are deleted through central deletion.

ATLASDATADISK

A watermark of free space is set to min(10%, 300TB). When the space goes below this limit, unlocked (secondary) data will be deleted until the limit is reached.

ATLASCRATCHDISK

A watermark of free space is set to min(25%, 30TB). When the space goes below this limit, unlocked (secondary) data will be deleted until the limit is reached.

ATLASGROUPDISK/ATLASLOCALGROUPDISK

The deletion is triggered by the responsible persons.

Major updates:
-- KorsBos - 06 Jul 2008 -- DavidCameron - 2015-09-15

Responsible: DavidCameron Last reviewed by: Never reviewed

• 20140227_ADCOpsSiteClassification_rev01.pdf: 20140227_ADCOpsSiteClassification_rev01.pdf</verbatim>