Difference: GLUE2Storage (1 vs. 17)

Revision 172014-05-13 - MariaALANDESPRADILLO

Line: 1 to 1
 
META TOPICPARENT name="ISproviders"

Storage Services in GLUE 2

Line: 17 to 17
 
Name Published in the BDII SRM non-SRM GLUE2ServiceType
dCache YES YES ftp, gkftp, gsiftp, xrootd, dcap, gsidcap, file, http, https org.dcache.storage
Changed:
<
<
DPM YES YES xroot, webdav SRM
StoRM YES YES NO Storage
>
>
DPM YES YES xroot, webdav Storage
StoRM YES YES NO SRM
 

Storage Protocols

Revision 162014-03-27 - MariaALANDESPRADILLO

Line: 1 to 1
 
META TOPICPARENT name="ISproviders"

Storage Services in GLUE 2

Line: 95 to 95
 
StoRM GGUS:92217 General ticket tracking glue-validator errors for StoRM
StoRM GGUS:94268 Request to publish gridftp endpoints
StoRM GGUS:101086 How to build the gsiftp URL of a storage element?
\ No newline at end of file
Added:
>
>
StoRM GGUS:90865 Discrepancy between BDII and SRM sizes
 \ No newline at end of file

Revision 142014-03-05 - MariaALANDESPRADILLO

Line: 1 to 1
 
META TOPICPARENT name="ISproviders"

Storage Services in GLUE 2

Line: 28 to 28
 
dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM
SRM YES YES YES SRM SRM SRM httpg httpg httpg - - - data.management.storage
data.management.transfer
Storage Management Storage Management Not published Not published Not published
webdav - YES - - webdav - - https - - NO - - Storage Management - - Not published -
Changed:
<
<
xroot YES YES NO xroot xroot - xroot xroot - YES NO YES data.transfer Storage Management - xrootd root root
>
>
xroot YES YES NO xroot xroot - xroot xroot - YES NO YES data.transfer Storage Management - xrootd Not published root
 
gsiftp YES NO NO gsiftp - - gsiftp - - YES YES YES data.transfer - - gsiftp gsiftp gsiftp
rfio - NO NO - - - - - - - YES YES - - - - rfio rfio
http YES - NO http - - http - - YES - YES data.transfer - - http - http

Revision 132014-02-27 - MariaALANDESPRADILLO

Line: 1 to 1
 
META TOPICPARENT name="ISproviders"

Storage Services in GLUE 2

Line: 29 to 29
 
SRM YES YES YES SRM SRM SRM httpg httpg httpg - - - data.management.storage
data.management.transfer
Storage Management Storage Management Not published Not published Not published
webdav - YES - - webdav - - https - - NO - - Storage Management - - Not published -
xroot YES YES NO xroot xroot - xroot xroot - YES NO YES data.transfer Storage Management - xrootd root root
Changed:
<
<
gsiftp YES NO NO gsiftp - - gsiftp - - YES YES YES data.transfer - - gsiftp gridftp gsiftp
>
>
gsiftp YES NO NO gsiftp - - gsiftp - - YES YES YES data.transfer - - gsiftp gsiftp gsiftp
 
rfio - NO NO - - - - - - - YES YES - - - - rfio rfio
http YES - NO http - - http - - YES - YES data.transfer - - http - http
https YES - NO https - - https - - YES - YES data.transfer - - https - https

Revision 122014-02-27 - MariaALANDESPRADILLO

Line: 1 to 1
 
META TOPICPARENT name="ISproviders"

Storage Services in GLUE 2

Line: 18 to 18
 
Name Published in the BDII SRM non-SRM GLUE2ServiceType
dCache YES YES ftp, gkftp, gsiftp, xrootd, dcap, gsidcap, file, http, https org.dcache.storage
DPM YES YES xroot, webdav SRM
Changed:
<
<
StoRM YES YES ? Storage
>
>
StoRM YES YES NO Storage
 

Storage Protocols

Line: 28 to 28
 
dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM
SRM YES YES YES SRM SRM SRM httpg httpg httpg - - - data.management.storage
data.management.transfer
Storage Management Storage Management Not published Not published Not published
webdav - YES - - webdav - - https - - NO - - Storage Management - - Not published -
Changed:
<
<
xroot YES YES NO xroot xroot - xroot xroot - YES NO YES data.transfer Storage Management - xrootd root xroot
>
>
xroot YES YES NO xroot xroot - xroot xroot - YES NO YES data.transfer Storage Management - xrootd root root
 
gsiftp YES NO NO gsiftp - - gsiftp - - YES YES YES data.transfer - - gsiftp gridftp gsiftp
rfio - NO NO - - - - - - - YES YES - - - - rfio rfio
http YES - NO http - - http - - YES - YES data.transfer - - http - http
Line: 39 to 39
 
ftp YES - - ftp - - ftp - - YES - - data.transfer - - Not published - -
gkftp NA - - NA - - NA - - NA - - NA - - NA - -
Added:
>
>
  • GLUE2StorageAccessProtocol object describes all the protocols that can be used to store or retrieve data.
  • GLUE2Endpoint may be published for each access protocol that may not require prior negotiation. The InterfaceName must correspond to the access protocol.
 

Storage Information Providers

This section gives details on how the different storage services publish the storage service related information.

Line: 85 to 88
 
Name GGUS ticket Description
dCache GGUS:98854 How dCache info provider works, explanation why Free is not published
\ No newline at end of file
Added:
>
>
dCache GGUS:99750 Explanations on how to get storage capacity numbers from dCache
DPM GGUS:98853 Ticket tracking incorrect numbers for unreserved space
DPM GGUS:93097 General ticket tracking glue-validator errors for DPM
DPM GGUS:94264 Request to publish gridftp endpoints
StoRM GGUS:92217 General ticket tracking glue-validator errors for StoRM
StoRM GGUS:94268 Request to publish gridftp endpoints
StoRM GGUS:101086 How to build the gsiftp URL of a storage element?

Revision 112014-02-05 - MariaALANDESPRADILLO

Line: 1 to 1
 
META TOPICPARENT name="ISproviders"

Storage Services in GLUE 2

Line: 78 to 78
 
DPM - Total, Used, Free - Total, Used, Free, Reserved=Total for space tokens, Reserved=0 for non reserved area

dCache and StoRM do not seem to publish non reserved area (to be confirmed by developers).

Added:
>
>

GGUS tickets

The following GGUS tickets have been opened to track capacity number inconsistencies. Developers have given more details or explanations on how things are calculated within them.

Name GGUS ticket Description
dCache GGUS:98854 How dCache info provider works, explanation why Free is not published
 \ No newline at end of file

Revision 102013-08-29 - MariaALANDESPRADILLO

Line: 1 to 1
 
META TOPICPARENT name="ISproviders"

Storage Services in GLUE 2

Line: 68 to 68
 Note that for DPM only online capacity is published. Nearline capacity is always 0.

Note that DPM is rounding the published numbers removing the fractional part. Calculations are done internally in bytes and numbers are transformed to Gigabytes (power of 10) when published in the BDII. Anything less than 1GB is published as 0. \ No newline at end of file

Added:
>
>

Storage Capacity

Name Service Capacity Share Capacity Comments
Nearline Online Nearline Online
dCache Total, Used Total, Used, Free Total, Used Total, Used, Free Free=Total - Used, numbers always match
StoRM Reserved=0, Total, Used, Free Reserved=0, Total, Used, Free Reserved=0, Total, Used, Free Reserved=0, Total, Used, Free Free=Total - Used, numbers always match. Nearline numbers do not match
DPM - Total, Used, Free - Total, Used, Free, Reserved=Total for space tokens, Reserved=0 for non reserved area

dCache and StoRM do not seem to publish non reserved area (to be confirmed by developers).

Revision 92013-08-06 - MariaALANDESPRADILLO

Line: 1 to 1
 
META TOPICPARENT name="ISproviders"

Storage Services in GLUE 2

Line: 67 to 67
  Note that for DPM only online capacity is published. Nearline capacity is always 0.
Changed:
<
<
Note that DPM is rounding the published numbers removing the fractional part. Calculations are done internally in bytes and numbers are transformed to Gigabytes (power of 10) when published in the BDII. Anything less than 1GB is published as 0.
>
>
Note that DPM is rounding the published numbers removing the fractional part. Calculations are done internally in bytes and numbers are transformed to Gigabytes (power of 10) when published in the BDII. Anything less than 1GB is published as 0.
 \ No newline at end of file

Revision 82013-07-23 - MariaALANDESPRADILLO

Line: 1 to 1
 
META TOPICPARENT name="ISproviders"

Storage Services in GLUE 2

Line: 55 to 55
 A GlueSA/GLUE2StorageShare+GLUE2StorageShareCapacity is published per pool and per space token (reserved space). For the pool, the information that it is published refers to the non reserved space. The following definitions apply for each of them:

Type GlueSATotalOnlineSize
GLUE2StorageShareCapacityTotalSize
GlueSAUsedOnlineSize
GLUE2StorageShareCapacityUsedSize
GlueSAFreeOnlineSize
GLUE2StorageShareCapacityFreeSize
GlueSAReservedOnlineSize
GLUE2StorageShareCapacityReservedSize
GlueSALocalID
GLUE2StorageShareSharingID
Changed:
<
<
Non reserved pool space total1=total size of the pool - reserved space - unavailable space (due to read-only or disabled file systems) total1 - free free space in the online file systems 0 pool_name:retention_type:latency_type
>
>
Non reserved pool space total1=total size of the pool - reserved space - unavailable space (due to read-only or disabled file systems) total1 - free free space in the online file systems - free space in the reserved space 0 pool_name:retention_type:latency_type
 
Space tokens size of the reservation size of the used part of the reservation free part of the reservation size of the reservation pool_name:SR:retention_type:latency_type

The GlueSE/GLUE2StorageServiceCapacity contains the aggregated numbers per pool:

Revision 72013-07-23 - MariaALANDESPRADILLO

Line: 1 to 1
 
META TOPICPARENT name="ISproviders"

Storage Services in GLUE 2

Line: 67 to 67
  Note that for DPM only online capacity is published. Nearline capacity is always 0.
Changed:
<
<
Note that DPM is rounding the published numbers. Anything less than 1GB is published as 0.
>
>
Note that DPM is rounding the published numbers removing the fractional part. Calculations are done internally in bytes and numbers are transformed to Gigabytes (power of 10) when published in the BDII. Anything less than 1GB is published as 0.
 \ No newline at end of file

Revision 62013-07-22 - MariaALANDESPRADILLO

Line: 1 to 1
 
META TOPICPARENT name="ISproviders"
Changed:
<
<

Storage Services in GLUE 2

>
>

Storage Services in GLUE 2

  This twiki tracks open issues that need a common agreement among the different storage service developers. In particular:
Line: 19 to 20
 
DPM YES YES xroot, webdav SRM
StoRM YES YES ? Storage
Changed:
<
<

Protocols

>
>

Storage Protocols

  The following table shows how the existing protocols used by the different storage services are published in the BDII:
Line: 37 to 38
 
gsidcap YES - - gsidcap - - gsidcap - - YES - - data.transfer - - gsidcap - -
ftp YES - - ftp - - ftp - - YES - - data.transfer - - Not published - -
gkftp NA - - NA - - NA - - NA - - NA - - NA - -
Added:
>
>

Storage Information Providers

This section gives details on how the different storage services publish the storage service related information.

DPM

A pool in DPM is a collection of file systems. File systems can be online, disabled or read-only. Space can be reserved in a pool for a particular VO, which is what it is called space token.

For pools: Note that whereas DPM does not consider the unavailable space due to file systems being disabled or read-only in the total space, the information provider always takes this into account. So total in the information provider means available and unavailable space in the file systems. On the other hand, free in the information provider only refers to available space in the online filesystems.

Command GLUE 1 GLUE 2
/usr/bin/dpm-listspaces --gip --protocols --basedir home --site my-site-name --glue2 GlueSE
GlueSA
GlueVOInfo
GlueSEAccessProtocol
GlueSEControlProtocol
GLUE2StorageService
GLUE2StorageServiceCapacity
GLUE2StorageEndpoint
GLUE2StorageManager
GLUE2StorageShare
GLUE2StorageShareCapacity
GLUE2StorageAccessProtocol
GLUE2AccessPolicy

A GlueSA/GLUE2StorageShare+GLUE2StorageShareCapacity is published per pool and per space token (reserved space). For the pool, the information that it is published refers to the non reserved space. The following definitions apply for each of them:

Type GlueSATotalOnlineSize
GLUE2StorageShareCapacityTotalSize
GlueSAUsedOnlineSize
GLUE2StorageShareCapacityUsedSize
GlueSAFreeOnlineSize
GLUE2StorageShareCapacityFreeSize
GlueSAReservedOnlineSize
GLUE2StorageShareCapacityReservedSize
GlueSALocalID
GLUE2StorageShareSharingID
Non reserved pool space total1=total size of the pool - reserved space - unavailable space (due to read-only or disabled file systems) total1 - free free space in the online file systems 0 pool_name:retention_type:latency_type
Space tokens size of the reservation size of the used part of the reservation free part of the reservation size of the reservation pool_name:SR:retention_type:latency_type

The GlueSE/GLUE2StorageServiceCapacity contains the aggregated numbers per pool:

Type GlueSETotalOnlineSize/GlueSESizeTotal
GLUE2StorageServiceCapacityTotalSize
GlueSEUsedOnlineSize
GLUE2StorageServiceCapacityUsedSize
GlueSESizeFree
GLUE2StorageServiceCapacityFreeSize
Aggregated pools the total size of all pools defined in DPM Sum per pool of: total - reserved - free total - used

Note that no reserved space is published at service level.

Note that for DPM only online capacity is published. Nearline capacity is always 0.

Note that DPM is rounding the published numbers. Anything less than 1GB is published as 0.

Revision 52013-07-12 - MariaALANDESPRADILLO

Line: 1 to 1
 
META TOPICPARENT name="ISproviders"

Storage Services in GLUE 2

Line: 27 to 27
 
dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM
SRM YES YES YES SRM SRM SRM httpg httpg httpg - - - data.management.storage
data.management.transfer
Storage Management Storage Management Not published Not published Not published
webdav - YES - - webdav - - https - - NO - - Storage Management - - Not published -
Changed:
<
<
xroot YES YES NO xroot xroot - xroot xroot - ? NO YES data.transfer Storage Management - xrootd root xroot
>
>
xroot YES YES NO xroot xroot - xroot xroot - YES NO YES data.transfer Storage Management - xrootd root xroot
 
gsiftp YES NO NO gsiftp - - gsiftp - - YES YES YES data.transfer - - gsiftp gridftp gsiftp
rfio - NO NO - - - - - - - YES YES - - - - rfio rfio
http YES - NO http - - http - - YES - YES data.transfer - - http - http
https YES - NO https - - https - - YES - YES data.transfer - - https - https
Changed:
<
<
nfs YES - - nfs - - nfs - - ? - - data.transfer - - nfs - -
file YES - NO file - - file - - - YES - data.transfer - - file - file
>
>
file YES - NO file - - file - - YES - YES data.transfer - - file - file
 
dcap YES - - dcap - - dcap - - YES - - data.transfer - - dcap - -
gsidcap YES - - gsidcap - - gsidcap - - YES - - data.transfer - - gsidcap - -
ftp YES - - ftp - - ftp - - YES - - data.transfer - - Not published - -

Revision 42013-07-11 - MariaALANDESPRADILLO

Line: 1 to 1
 
META TOPICPARENT name="ISproviders"

Storage Services in GLUE 2

Line: 15 to 15
 The following table shows the existing storage resource managers published in the BDII:

Name Published in the BDII SRM non-SRM GLUE2ServiceType
Changed:
<
<
dCache YES YES ? org.dcache.storage
>
>
dCache YES YES ftp, gkftp, gsiftp, xrootd, dcap, gsidcap, file, http, https org.dcache.storage
 
DPM YES YES xroot, webdav SRM
StoRM YES YES ? Storage
Line: 26 to 26
 
Protocol Published as an endpoint GLUE2EndpointInterfaceName GLUE2EndpointURL protocol Accessible via SRM GLUE2EndpointCapability GLUE2StorageAccessProtocolType
dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM
SRM YES YES YES SRM SRM SRM httpg httpg httpg - - - data.management.storage
data.management.transfer
Storage Management Storage Management Not published Not published Not published
Deleted:
<
<
webdav NO YES NO - webdav - - https - - NO - - Storage Management - - Not published -
xroot YES YES NO xroot xroot - xroot xroot - ? NO - data.transfer Storage Management - xrootd - xroot
gsiftp YES NO NO gsiftp - - gsiftp - - ? YES YES data.transfer - - gsiftp gridftp gsiftp
rfio NO NO NO - - - - - - NO YES YES - - - - rfio rfio
http YES NO NO http - - http - - ? - - data.transfer - - http - http
https YES NO NO https - - https - - ? - - data.transfer - - https - https
nfs YES NO NO nfs - - nfs - - ? - - data.transfer - - nfs - -
file YES NO NO file - - file - - ? - - data.transfer - - file - file
dcap YES NO NO dcap - - dcap - - ? - - data.transfer - - dcap - -
gsidcap YES NO NO gsidcap - - gsidcap - - ? - - data.transfer - - gsidcap - -
 \ No newline at end of file
Added:
>
>
webdav - YES - - webdav - - https - - NO - - Storage Management - - Not published -
xroot YES YES NO xroot xroot - xroot xroot - ? NO YES data.transfer Storage Management - xrootd root xroot
gsiftp YES NO NO gsiftp - - gsiftp - - YES YES YES data.transfer - - gsiftp gridftp gsiftp
rfio - NO NO - - - - - - - YES YES - - - - rfio rfio
http YES - NO http - - http - - YES - YES data.transfer - - http - http
https YES - NO https - - https - - YES - YES data.transfer - - https - https
nfs YES - - nfs - - nfs - - ? - - data.transfer - - nfs - -
file YES - NO file - - file - - - YES - data.transfer - - file - file
dcap YES - - dcap - - dcap - - YES - - data.transfer - - dcap - -
gsidcap YES - - gsidcap - - gsidcap - - YES - - data.transfer - - gsidcap - -
ftp YES - - ftp - - ftp - - YES - - data.transfer - - Not published - -
gkftp NA - - NA - - NA - - NA - - NA - - NA - -

Revision 32013-07-10 - MariaALANDESPRADILLO

Line: 1 to 1
 
META TOPICPARENT name="ISproviders"

Storage Services in GLUE 2

Line: 27 to 27
 
dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM
SRM YES YES YES SRM SRM SRM httpg httpg httpg - - - data.management.storage
data.management.transfer
Storage Management Storage Management Not published Not published Not published
webdav NO YES NO - webdav - - https - - NO - - Storage Management - - Not published -
Changed:
<
<
xroot YES YES NO xroot xroot - xroot xroot - ? NO - data.transfer Storage Management - xrootd Not published -
>
>
xroot YES YES NO xroot xroot - xroot xroot - ? NO - data.transfer Storage Management - xrootd - xroot
 
gsiftp YES NO NO gsiftp - - gsiftp - - ? YES YES data.transfer - - gsiftp gridftp gsiftp
rfio NO NO NO - - - - - - NO YES YES - - - - rfio rfio
Changed:
<
<
http YES NO NO http - - http - - ? - - data.transfer - - http - -
https YES NO NO https - - https - - ? - - data.transfer - - https - -
>
>
http YES NO NO http - - http - - ? - - data.transfer - - http - http
https YES NO NO https - - https - - ? - - data.transfer - - https - https
 
nfs YES NO NO nfs - - nfs - - ? - - data.transfer - - nfs - -
Changed:
<
<
file YES NO NO file - - file - - ? - - data.transfer - - file - -
>
>
file YES NO NO file - - file - - ? - - data.transfer - - file - file
 
dcap YES NO NO dcap - - dcap - - ? - - data.transfer - - dcap - -
gsidcap YES NO NO gsidcap - - gsidcap - - ? - - data.transfer - - gsidcap - -

Revision 22013-06-11 - MariaALANDESPRADILLO

Line: 1 to 1
 
META TOPICPARENT name="ISproviders"

Storage Services in GLUE 2

This twiki tracks open issues that need a common agreement among the different storage service developers. In particular:

Changed:
<
<
>
>
  • SURL in GLUE2EndpointURL? This is what experiments are using
  • GLUE2ServiceType: agree on a coherent list of existing storage service types
  • GLUE2EndpointInterfaceName and GLUE2StorageAccessProtocol: agree on a list of protocol names
  • Capabilities: agree which capabilities should be published per protocol type
 
Changed:
<
<

Services

>
>

Storage Resource Managers

 
Changed:
<
<
The following table shows the existing storage services:
>
>
The following table shows the existing storage resource managers published in the BDII:
 
Changed:
<
<
Service Name Published in the BDII SRM non-SRM GLUE2ServiceType
dCache YES YES YES org.dcache.storage
DPM YES YES YES SRM
StoRM YES ? ? ?
Bestman NO ? ? ?
Castor NO ? ? ?
>
>
Name Published in the BDII SRM non-SRM GLUE2ServiceType
dCache YES YES ? org.dcache.storage
DPM YES YES xroot, webdav SRM
StoRM YES YES ? Storage
 

Protocols

Changed:
<
<
The following table shows the existing protocols used by the different storage services:

Protocol Protocol type Published as an endpoint GLUE2EndpointInterfaceName GLUE2EndpointURL Accessible via SRM Capability
SRM Data Access
webdav Data Access
xroot Data Access
gsiftp
rfio
http
https
httpg
>
>
The following table shows how the existing protocols used by the different storage services are published in the BDII:
 
Added:
>
>
Protocol Published as an endpoint GLUE2EndpointInterfaceName GLUE2EndpointURL protocol Accessible via SRM GLUE2EndpointCapability GLUE2StorageAccessProtocolType
dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM dCache DPM StoRM
SRM YES YES YES SRM SRM SRM httpg httpg httpg - - - data.management.storage
data.management.transfer
Storage Management Storage Management Not published Not published Not published
webdav NO YES NO - webdav - - https - - NO - - Storage Management - - Not published -
xroot YES YES NO xroot xroot - xroot xroot - ? NO - data.transfer Storage Management - xrootd Not published -
gsiftp YES NO NO gsiftp - - gsiftp - - ? YES YES data.transfer - - gsiftp gridftp gsiftp
rfio NO NO NO - - - - - - NO YES YES - - - - rfio rfio
http YES NO NO http - - http - - ? - - data.transfer - - http - -
https YES NO NO https - - https - - ? - - data.transfer - - https - -
nfs YES NO NO nfs - - nfs - - ? - - data.transfer - - nfs - -
file YES NO NO file - - file - - ? - - data.transfer - - file - -
dcap YES NO NO dcap - - dcap - - ? - - data.transfer - - dcap - -
gsidcap YES NO NO gsidcap - - gsidcap - - ? - - data.transfer - - gsidcap - -

Revision 12013-06-11 - MariaALANDESPRADILLO

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="ISproviders"

Storage Services in GLUE 2

This twiki tracks open issues that need a common agreement among the different storage service developers. In particular:

Services

The following table shows the existing storage services:

Service Name Published in the BDII SRM non-SRM GLUE2ServiceType
dCache YES YES YES org.dcache.storage
DPM YES YES YES SRM
StoRM YES ? ? ?
Bestman NO ? ? ?
Castor NO ? ? ?

Protocols

The following table shows the existing protocols used by the different storage services:

Protocol Protocol type Published as an endpoint GLUE2EndpointInterfaceName GLUE2EndpointURL Accessible via SRM Capability
SRM Data Access
webdav Data Access
xroot Data Access
gsiftp
rfio
http
https
httpg
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright & by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Ask a support question or Send feedback