Archiving Data in CASTOR at CERN

Introduction

The current user area on CERN CASTOR is being phased out in favour of disk based storage for live data. This leaves a need for archival storage. To accomplish this users will be able to store data onto tape at CERN. Only archival data should be stored there which is not meant to be read. Retrieval of data from this area is expected to be slow.

Limitations

Data to be stored should be put into appropriate sized files (at least 1GB), using tools like tar to aggregate smaller files. Files larger than 200GB begin to cause problems.

CASTOR Path

User data should be stored under:

/castor/cern.ch/cms/archive/user/<u>/<username>

Any data not following this scheme will be deleted.

Group data should be stored under:

/castor/cern.ch/cms/archive/group/<g>/<groupname>

FIXME add more details on group archiving

Usage limits

It is expected users store up to 10TB. If you will use more than this please submit a Savannah ticket outlining the need.

Storing and retrieving data

To store your file from a local filesystem onto tape use:

xrdcopy <file> "root://castorcms.cern.ch//castor/cern.ch/cms/archive/user/<u>/<username>/<file>?svcClass=archive"

To store your file from EOS onto tape use the third-party-copy option, e.g.:

xrdcopy --tpc only "root://eoscms.cern.ch//eos/cms/store/user/<username>/<file>" "root://castorcms.cern.ch//castor/cern.ch/cms/archive/user/<u>/<username>/<file>?svcClass=archive"

If you need to read the file in the future bring it online with;

stager_get -S archive -M /castor/cern.ch/cms/archive/user/<u>/<username>/<file>

Before trying to read it check that it is actually online (will probably take around an hour);

stager_qry -S archive -M /castor/cern.ch/cms/archive/user/<u>/<username>/<file>

If it shows STAGEIN it is still to be read from tape. STAGED shows it is online.

To copy it out to a local filesystem use the command:

xrdcopy "root://castorcms.cern.ch//castor/cern.ch/cms/archive/user/<u>/<username>/<file>?svcClass=archive" <file>

To copy it out to EOS use the third-party copy option, e.g.:

xrdcopy --tpc only "root://castorcms.cern.ch//castor/cern.ch/cms/archive/user/<u>/<username>/<file>?svcClass=archive" "root://eoscms.cern.ch//eos/cms/store/user/<username>/<file>"

The same command, with a valid CMS proxy will work also from outside CERN.

-- NicoloMagini - 14 May 2014

Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r6 - 2014-05-14 - NicoloMagini
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback