Difference: LSDbsTools (1 vs. 13)

Revision 132012-08-21 - unknown

Line: 1 to 1
 
META TOPICPARENT name="LeonardoSala"
Line: 11 to 11
  Here two new tools are provided for accomplish to those purposes. One for transferring and registering data, the other one to invalidate and/or delete data.
Added:
>
>
If you need to transfer data not registered in DBS, you can use LSDataReplica
 

Current release

You can retrieve the updated version from CVS with:

Line: 25 to 27
 

dbs_transferRegister.py

This tool relies on data replica for file transfers and on DBSAPI calls (see e.g. here)

Changed:
<
<
for block registration.
>
>
for block registration. LSDataReplica can be used to transfer files not registered to DBS.
  The usage is straightforward:

Revision 122012-05-30 - unknown

Line: 1 to 1
 
META TOPICPARENT name="LeonardoSala"
Line: 13 to 13
 

Current release

Changed:
<
<
Current release is V00-03. You can retrieve it from CVS with:
>
>
You can retrieve the updated version from CVS with:
 
cvs co -rV00-01 UserCode/leo/Utilities/dbs_utils.py
cvs co -rV00-03 UserCode/leo/Utilities/dbs_InvalidateAndDeleteDataset.py
Changed:
<
<
cvs co -rV00-03 UserCode/leo/Utilities/dbs_transferRegister.py cvs co -rV01-01-05 UserCode/leo/Utilities/data_replica.py
>
>
cvs co -rV00-04 UserCode/leo/Utilities/dbs_transferRegister.py cvs co -rV01-01-06 UserCode/leo/Utilities/data_replica.py
 

dbs_transferRegister.py

Revision 112012-01-11 - unknown

Line: 1 to 1
 
META TOPICPARENT name="LeonardoSala"
Line: 19 to 19
 cvs co -rV00-01 UserCode/leo/Utilities/dbs_utils.py cvs co -rV00-03 UserCode/leo/Utilities/dbs_InvalidateAndDeleteDataset.py cvs co -rV00-03 UserCode/leo/Utilities/dbs_transferRegister.py
Changed:
<
<
cvs co -rV01-01-04 UserCode/leo/Utilities/data_replica.py
>
>
cvs co -rV01-01-05 UserCode/leo/Utilities/data_replica.py
 

dbs_transferRegister.py

Revision 102012-01-10 - unknown

Line: 1 to 1
 
META TOPICPARENT name="LeonardoSala"
Line: 17 to 17
 
cvs co -rV00-01 UserCode/leo/Utilities/dbs_utils.py
Changed:
<
<
cvs co -rV00-02 UserCode/leo/Utilities/dbs_InvalidateAndDeleteDataset.py
>
>
cvs co -rV00-03 UserCode/leo/Utilities/dbs_InvalidateAndDeleteDataset.py
 cvs co -rV00-03 UserCode/leo/Utilities/dbs_transferRegister.py cvs co -rV01-01-04 UserCode/leo/Utilities/data_replica.py
Line: 83 to 83
 Usage: dbs_InvalidateAndDeleteDataset.py [--dbs=ph01|ph02] [--all] [--site=TX_YY_SITE] dataset

If --all is used, the the dataset will be unregistered and deleted from ALL the sites. Otherwise,

Changed:
<
<
it will only be deleted and invalidated from the site specified in --site
>
>
it will only be deleted and invalidated from the site specified in --site.

The invalidation is actually done through the DBSInvalidateDataset.py provided by CRAB [*]. If you just want to invalidate a dataet, but not to delete the data from the SE, try to use this tool instead.

[*] https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideCrabForPublication?redirectedfrom=CMS.SWGuideCrabForPublication#Invalidate_a_dataset_in_DBS

 

Options:

Line: 91 to 97
  --dbs=DBS DBS instance, can be: ph01, ph02 --all Delete the sample from all the sites, and invalidate the dataset
Changed:
<
<
--site=SITE Delete and invalidate the sample from this site. --invalidate Invalidate the dataset --delete Delete the dataset, both physically and from DBS (just the replica information)
>
>
--site=SITE Delete the sample from this site, both physically and from DBS (just the replica information. If the dataset is available at other sites, it is still VALID in DBS). --yes Answer YES to all the questions. USE IT WITH CARE! --debug Verboooose
 

Usecase 1: remove a replica

Line: 103 to 109
 Let's suppose that you want to remove a replica from site T3_AA_FOO. In this case, the proper command is:
Changed:
<
<
python dbs_InvalidateAndDeleteDataset.py --dbs=ph02 --site=T3_AA_FOO --delete
>
>
python dbs_InvalidateAndDeleteDataset.py --dbs=ph02 --site=T3_AA_FOO
 

This will physically delete all files at the given SE and remove the corresponding blocks from DBS. In case of errors in the deletion, the block

Line: 115 to 121
 If you want to completely remove a dataset, then:
Changed:
<
<
python dbs_InvalidateAndDeleteDataset.py --dbs=ph02 --all --delete
>
>
python dbs_InvalidateAndDeleteDataset.py --dbs=ph02 --all
 

This will delete all the replicas at all SEs, remove the SE information from DBS and invalidate the dataset using the DBSInvalidateDataset.py tool

Revision 92011-11-09 - unknown

Line: 1 to 1
 
META TOPICPARENT name="LeonardoSala"
Line: 13 to 13
 

Current release

Changed:
<
<
Current release is V00-02. You can retrieve it from CVS with:
>
>
Current release is V00-03. You can retrieve it from CVS with:
 
cvs co -rV00-01 UserCode/leo/Utilities/dbs_utils.py
cvs co -rV00-02 UserCode/leo/Utilities/dbs_InvalidateAndDeleteDataset.py
Changed:
<
<
cvs co -rV00-02 UserCode/leo/Utilities/dbs_transferRegister.py cvs co -rV01-01-02 UserCode/leo/Utilities/data_replica.py
>
>
cvs co -rV00-03 UserCode/leo/Utilities/dbs_transferRegister.py cvs co -rV01-01-04 UserCode/leo/Utilities/data_replica.py
 

dbs_transferRegister.py

Line: 33 to 33
 python dbs_transferRegister.py --to-site T3_XX_YYYY dataset_name
Added:
>
>
The --help option gives all the needed information:
Usage: /swshare/psit3/bin/dbs_transferRegister.py [--dbs=ph01|ph02] --to-site=TX_YY_SITE dataset

Options:
  -h, --help            show this help message and exit
  --dbs=DBS             DBS instance, can be: ph01, ph02 (default)
  --to-site=TO_SITE     Destination site.
  --whitelist=WHITELIST
                        Sets up a comma-separated White-list (preferred
                        sites). Transfers will start from these sites. Sites
                        not included in the whitelist will be not excluded.
  --blacklist=BLACKLIST
                        Sets up a comma-separated Black-list (excluded sites).
  --retransfer          Do not skip already transferred block.
  --copy-tool=TOOL      Selects the copy tool to be used (lcg-cp or srmcp). By
                        default lcg-cp is used
  --debug               Verbose mode
  --delete              If file exists at destination and its size is
                        _smaller_ than the source one, delete it. WARNING:
                        destination files are checked only for SRM endpoints.
 The tool will find the blocks composing the dataset and transfer/register them separately. In case of transfer failure the block is not registered. Dataset source is retrieved from DBS.

Revision 82011-05-09 - unknown

Line: 1 to 1
 
META TOPICPARENT name="LeonardoSala"
Line: 13 to 13
 

Current release

Changed:
<
<
Current release is V00-01. You can retrieve it from CVS with:
>
>
Current release is V00-02. You can retrieve it from CVS with:
 
cvs co -rV00-01 UserCode/leo/Utilities/dbs_utils.py
Changed:
<
<
cvs co -rV00-01 UserCode/leo/Utilities/dbs_InvalidateAndDeleteDataset.py cvs co -rV00-01 UserCode/leo/Utilities/dbs_transferRegister.py cvs co -rV01-01-01 UserCode/leo/Utilities/data_replica.py
>
>
cvs co -rV00-02 UserCode/leo/Utilities/dbs_InvalidateAndDeleteDataset.py cvs co -rV00-02 UserCode/leo/Utilities/dbs_transferRegister.py cvs co -rV01-01-02 UserCode/leo/Utilities/data_replica.py
 

dbs_transferRegister.py

Revision 72010-12-01 - unknown

Line: 1 to 1
 
META TOPICPARENT name="LeonardoSala"
Line: 19 to 19
 cvs co -rV00-01 UserCode/leo/Utilities/dbs_utils.py cvs co -rV00-01 UserCode/leo/Utilities/dbs_InvalidateAndDeleteDataset.py cvs co -rV00-01 UserCode/leo/Utilities/dbs_transferRegister.py
Changed:
<
<
cvs co -rV01-01-00 UserCode/leo/Utilities/data_replica.py
>
>
cvs co -rV01-01-01 UserCode/leo/Utilities/data_replica.py
 

dbs_transferRegister.py

Revision 62010-06-25 - unknown

Line: 1 to 1
 
META TOPICPARENT name="LeonardoSala"
Line: 11 to 11
  Here two new tools are provided for accomplish to those purposes. One for transferring and registering data, the other one to invalidate and/or delete data.
Changed:
<
<
--++ Current release
>
>

Current release

  Current release is V00-01. You can retrieve it from CVS with:

Revision 52010-06-23 - unknown

Line: 1 to 1
 
META TOPICPARENT name="LeonardoSala"
Line: 6 to 6
 

Aim

Changed:
<
<
A CRAB nice characteristic is the possibility to easily register datasets (edm-compatible) to local BBS. Anyway, migrating these data
>
>
A CRAB nice characteristic is the possibility to easily register datasets (edm-compatible) to local DBS. Anyway, migrating these data
 e.g. to the T3 can be not simple, and also the registration of the new location in DBS.

Here two new tools are provided for accomplish to those purposes. One for transferring and registering data, the other one to invalidate and/or delete data.

Revision 42010-06-22 - unknown

Line: 1 to 1
 
META TOPICPARENT name="LeonardoSala"
Line: 11 to 11
  Here two new tools are provided for accomplish to those purposes. One for transferring and registering data, the other one to invalidate and/or delete data.
Added:
>
>
--++ Current release
 
Added:
>
>
Current release is V00-01. You can retrieve it from CVS with:

cvs co -rV00-01 UserCode/leo/Utilities/dbs_utils.py
cvs co -rV00-01 UserCode/leo/Utilities/dbs_InvalidateAndDeleteDataset.py
cvs co -rV00-01 UserCode/leo/Utilities/dbs_transferRegister.py
cvs co -rV01-01-00 UserCode/leo/Utilities/data_replica.py
 

dbs_transferRegister.py

Revision 32010-06-22 - unknown

Line: 1 to 1
 
META TOPICPARENT name="LeonardoSala"
Line: 78 to 78
 will be removed from DBS anyway and a list of files which failed the deletion will be given as output. PS "No such file" is not considered as an error
Added:
>
>

Usecase 2: remove a dataset

 
Added:
>
>
If you want to completely remove a dataset, then:

python dbs_InvalidateAndDeleteDataset.py --dbs=ph02 --all  --delete <DATASETNAME>

This will delete all the replicas at all SEs, remove the SE information from DBS and invalidate the dataset using the DBSInvalidateDataset.py tool shipped with CRAB

  -- LeonardoSala - 22-Jun-2010

Revision 22010-06-22 - unknown

Line: 1 to 1
 
META TOPICPARENT name="LeonardoSala"
Line: 33 to 33
 
  • if the destination file is smaller than the source file, the former is deleted and the transfer begins
Added:
>
>

dbs_InvalidateAndDeleteDataset.py

Caveat: need some feedback

This tool is intended to easily manage the various private dataset copies at SE, e.g.: you staged out and published at a T2, moved the output to a T3 (and published), then you want to remove the copy located at the T2.

dbs_InvalidateAndDeleteDataset.py provides this for some user cases:

  • Remove data from a SE and unregister it from DBS
  • Remove all data, unregister it and invalidate the dataset
  • Invalidate the dataset

Usage:

Usage: dbs_InvalidateAndDeleteDataset.py [--dbs=ph01|ph02] [--all] [--site=TX_YY_SITE] dataset

If --all is used, the the dataset will be unregistered and deleted from ALL the sites. Otherwise,
it will only be deleted and invalidated from the site specified in --site


Options:
  -h, --help    show this help message and exit
  --dbs=DBS     DBS instance, can be: ph01, ph02
  --all         Delete the sample from all the sites, and invalidate the
                dataset
  --site=SITE   Delete and invalidate the sample from this site.
  --invalidate  Invalidate the dataset
  --delete      Delete the dataset, both physically and from DBS (just the
                replica information)

Usecase 1: remove a replica

Let's suppose that you want to remove a replica from site T3_AA_FOO. In this case, the proper command is:

python dbs_InvalidateAndDeleteDataset.py --dbs=ph02 --site=T3_AA_FOO  --delete <DATASETNAME>

This will physically delete all files at the given SE and remove the corresponding blocks from DBS. In case of errors in the deletion, the block will be removed from DBS anyway and a list of files which failed the deletion will be given as output. PS "No such file" is not considered as an error

 -- LeonardoSala - 22-Jun-2010

Revision 12010-06-22 - unknown

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="LeonardoSala"

DBS replica tools

Aim

A CRAB nice characteristic is the possibility to easily register datasets (edm-compatible) to local BBS. Anyway, migrating these data e.g. to the T3 can be not simple, and also the registration of the new location in DBS.

Here two new tools are provided for accomplish to those purposes. One for transferring and registering data, the other one to invalidate and/or delete data.

dbs_transferRegister.py

This tool relies on data replica for file transfers and on DBSAPI calls (see e.g. here) for block registration.

The usage is straightforward:

python dbs_transferRegister.py --to-site T3_XX_YYYY dataset_name

The tool will find the blocks composing the dataset and transfer/register them separately. In case of transfer failure the block is not registered. Dataset source is retrieved from DBS.

The usage of data_replica ensures the support also for checks on file existing files on destination:

  • if the sizes are equal, no error is reported, just a warning (exit code 0)
  • if the destination file is bigger than the source file, an error is raised (user should check and perform actions)
  • if the destination file is smaller than the source file, the former is deleted and the transfer begins

-- LeonardoSala - 22-Jun-2010

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback