HEPTape Web>Survey (2018-10-11, OliverKeeble)

Archive Site Survey

Overview

This initiative aims to investigate how archival systems can be used most optimally by users, and what metrics are available to track the effectiveness of their use.

The objective is to assemble recommendations on best practice, from all WLCG archival sites, which can be used by user communities to optimise their use of these systems. Commonalities will be candidates for incorporation into common services such as FTS.

At this stage it's not necessary for all advice to practically realisable. Deciding whether effort should be invested in implementing particular ideas will happen at a later phase.

Each site is requested to find their name in the list below and fill in the table on the linked page. You will find this easier if you use the WYSIWYG editor.

The Entry for CERN has already been filled in to serve as an example.

Sites

Feel free to edit site name as appropriate.

Results

Do not edit these, they will be regenerated periodically from the submissions above.

Results

Survey conclusions

Conclusions

Template

The following is the template for the survey.

Site

Site and Endpoints
Question	*________________________________Response__________________________________*
What is the site name?
Which endpoint URLs do your archival systems expose?
How is tape storage selected for a write (choice of endpoint, specification of a spacetoken, namespace prefix).
Queue
What limits should clients respect?
---> Max number of outstanding requests in number of files or data volume
---> Max ~~requests submitted at one time~~ submission rate for recalls or queries
---> Min/Max bulk request size (srmBringOnline or equivalent) in files or data volume
Should clients back off under certain circumstances?
---> How is this signalled to client?
---> For which operations?
Is it advantageous to group requests by a particular criterion (e.g. tape family, date)?
---> What criterion?
Prioritisation
Can you handle priority requests?
---> How is this requested?
Protocol support
Are there any unsupported or partially supported operations (e.g. pinning) ?
Timeouts
What timeouts do you recommend?
Do you have hardcoded or default timeouts?
Operations and metrics
Can you provide total sum of data stored by VO in the archive to 100TB accuracy?
Can you provide space occupied on tapes by VO (includes deleted data, but not yet reclaimed space) to 100TB accuracy?
How do you allocate free tape space to VOs?
What is the frequency with which you run repack operations to reclaim space on tapes after data deletion?
Recommendations for clients
Recommendation 1
---> Information required by users to follow advice
Recommendation 2
Buffer Management
Should a client stop submitting recalls if the available buffer space reaches a threshold?
---> How can a client determine the buffer used and free space?
---> What is the threshold (high water mark)?
---> When should the client restart submission (low water mark)?
If the client does not have to back off on a full buffer, and you support pinning, how is the buffer managed?
---> is data moved from buffer to another local disk, either by the HSM or by an external agent?
Additional questions
Should any other questions appear in subsequent iterations of this survey?

Topic revision: r12 - 2018-10-11 - OliverKeeble

HEPTape

Public webs

- Cern Search
- TWiki Search
- Google Search
HEPTape All webs

Copyright &© 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback