TWiki> EGEE Web>InformationSystem>ISproviders>GLUE2Storage (2014-05-13, MariaALANDESPRADILLO) EditAttachPDF

Storage Services in GLUE 2

This twiki tracks open issues that need a common agreement among the different storage service developers. In particular:

  • SURL in GLUE2EndpointURL? This is what experiments are using
  • GLUE2ServiceType: agree on a coherent list of existing storage service types
  • GLUE2EndpointInterfaceName and GLUE2StorageAccessProtocol: agree on a list of protocol names
  • Capabilities: agree which capabilities should be published per protocol type

Storage Resource Managers

The following table shows the existing storage resource managers published in the BDII:

Name Published in the BDII SRM non-SRM GLUE2ServiceType
dCache YES YES ftp, gkftp, gsiftp, xrootd, dcap, gsidcap, file, http, https org.dcache.storage
DPM YES YES xroot, webdav Storage
StoRM YES YES NO SRM

Storage Protocols

The following table shows how the existing protocols used by the different storage services are published in the BDII:

Protocol Published as an endpoint GLUE2EndpointInterfaceName GLUE2EndpointURL protocol Accessible via SRM GLUE2EndpointCapability GLUE2StorageAccessProtocolType
dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM
SRM YES YES YES SRM SRM SRM httpg httpg httpg - - - data.management.storage
data.management.transfer 		Storage Management Storage Management Not published Not published Not published
webdav - YES - - webdav - - https - - NO - - Storage Management - - Not published -
xroot YES YES NO xroot xroot - xroot xroot - YES NO YES data.transfer Storage Management - xrootd Not published root
gsiftp YES NO NO gsiftp - - gsiftp - - YES YES YES data.transfer - - gsiftp gsiftp gsiftp
rfio - NO NO - - - - - - - YES YES - - - - rfio rfio
http YES - NO http - - http - - YES - YES data.transfer - - http - http
https YES - NO https - - https - - YES - YES data.transfer - - https - https
file YES - NO file - - file - - YES - YES data.transfer - - file - file
dcap YES - - dcap - - dcap - - YES - - data.transfer - - dcap - -
gsidcap YES - - gsidcap - - gsidcap - - YES - - data.transfer - - gsidcap - -
ftp YES - - ftp - - ftp - - YES - - data.transfer - - Not published - -
gkftp NA - - NA - - NA - - NA - - NA - - NA - -

  • GLUE2StorageAccessProtocol object describes all the protocols that can be used to store or retrieve data.
  • GLUE2Endpoint may be published for each access protocol that may not require prior negotiation. The InterfaceName must correspond to the access protocol.

Storage Information Providers

This section gives details on how the different storage services publish the storage service related information.

DPM

A pool in DPM is a collection of file systems. File systems can be online, disabled or read-only. Space can be reserved in a pool for a particular VO, which is what it is called space token.

For pools: Note that whereas DPM does not consider the unavailable space due to file systems being disabled or read-only in the total space, the information provider always takes this into account. So total in the information provider means available and unavailable space in the file systems. On the other hand, free in the information provider only refers to available space in the online filesystems.

Command GLUE 1 GLUE 2
/usr/bin/dpm-listspaces --gip --protocols --basedir home --site my-site-name --glue2 GlueSE
GlueSA
GlueVOInfo
GlueSEAccessProtocol
GlueSEControlProtocol 		GLUE2StorageService
GLUE2StorageServiceCapacity
GLUE2StorageEndpoint
GLUE2StorageManager
GLUE2StorageShare
GLUE2StorageShareCapacity
GLUE2StorageAccessProtocol
GLUE2AccessPolicy

A GlueSA/GLUE2StorageShare+GLUE2StorageShareCapacity is published per pool and per space token (reserved space). For the pool, the information that it is published refers to the non reserved space. The following definitions apply for each of them:

Type GlueSATotalOnlineSize
GLUE2StorageShareCapacityTotalSize 		GlueSAUsedOnlineSize
GLUE2StorageShareCapacityUsedSize 		GlueSAFreeOnlineSize
GLUE2StorageShareCapacityFreeSize 		GlueSAReservedOnlineSize
GLUE2StorageShareCapacityReservedSize 		GlueSALocalID
GLUE2StorageShareSharingID
Non reserved pool space total1=total size of the pool - reserved space - unavailable space (due to read-only or disabled file systems) total1 - free free space in the online file systems - free space in the reserved space 0 pool_name:retention_type:latency_type
Space tokens size of the reservation size of the used part of the reservation free part of the reservation size of the reservation pool_name:SR:retention_type:latency_type

The GlueSE/GLUE2StorageServiceCapacity contains the aggregated numbers per pool:

Type GlueSETotalOnlineSize/GlueSESizeTotal
GLUE2StorageServiceCapacityTotalSize 		GlueSEUsedOnlineSize
GLUE2StorageServiceCapacityUsedSize 		GlueSESizeFree
GLUE2StorageServiceCapacityFreeSize
Aggregated pools the total size of all pools defined in DPM Sum per pool of: total - reserved - free total - used

Note that no reserved space is published at service level.

Note that for DPM only online capacity is published. Nearline capacity is always 0.

Note that DPM is rounding the published numbers removing the fractional part. Calculations are done internally in bytes and numbers are transformed to Gigabytes (power of 10) when published in the BDII. Anything less than 1GB is published as 0.

Storage Capacity

Name Service Capacity Share Capacity Comments
Nearline Online Nearline Online
dCache Total, Used Total, Used, Free Total, Used Total, Used, Free Free=Total - Used, numbers always match
StoRM Reserved=0, Total, Used, Free Reserved=0, Total, Used, Free Reserved=0, Total, Used, Free Reserved=0, Total, Used, Free Free=Total - Used, numbers always match. Nearline numbers do not match
DPM - Total, Used, Free - Total, Used, Free, Reserved=Total for space tokens, Reserved=0 for non reserved area

dCache and StoRM do not seem to publish non reserved area (to be confirmed by developers).

GGUS tickets

The following GGUS tickets have been opened to track capacity number inconsistencies. Developers have given more details or explanations on how things are calculated within them.

Name GGUS ticket Description
dCache GGUS:98854 How dCache info provider works, explanation why Free is not published
dCache GGUS:99750 Explanations on how to get storage capacity numbers from dCache
DPM GGUS:98853 Ticket tracking incorrect numbers for unreserved space
DPM GGUS:93097 General ticket tracking glue-validator errors for DPM
DPM GGUS:94264 Request to publish gridftp endpoints
StoRM GGUS:92217 General ticket tracking glue-validator errors for StoRM
StoRM GGUS:94268 Request to publish gridftp endpoints
StoRM GGUS:101086 How to build the gsiftp URL of a storage element?
StoRM GGUS:90865 Discrepancy between BDII and SRM sizes
Edit | Attach | Watch | Print version | History: r17 < r16 < r15 < r14 < r13 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r17 - 2014-05-13 - MariaALANDESPRADILLO
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EGEE All webs login

This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Ask a support question or Send feedback