DIRAC Grid Storage Quota

A similar tutorial on working with your grid quota in ganga is given in GridAndDataManagement.

Before executing the commands in this tutorial

If you want to work with LHCbDirac in bash, instead open a sub-shell with

> lb-dirac bash --norc

You will then need to do

> lhcb-proxy-init

Once these things are done, the following commands should work within the bash sub-shell you just opened.

What is my current quota?

To obtain your current DISK quota you can perform the following on the CLI:

> dirac-dms-user-quota
2009-09-29 14:05:49 UTC dirac-dms-user-quota.py  INFO: Current quota found to be 10.0 GB

As you can see the current quota for my account is 10GB.

What is my current usage?

To obtain your current DISK usage you can perform the following on the CLI:

> dirac-dms-storage-usage-summary --Users=username
Storage usage for user dduck (quota: 2.0 TB)
------------------------------------------------
DIRAC SE             Size (TB)            Files               
------------------------------------------------
CERN-USER            0.630                2392                
CNAF-USER            0.128                726                 
GRIDKA-USER          0.158                586                 
IN2P3-USER           0.275                1073                
PIC-USER             0.028                164                 
RAL-USER             0.186                911                 
RRCKI-USER           0.100                409                 
SARA-USER            0.103                642                 
Total (disk)         1.609                6903                

How to find old data?

To remove data one must first obtain the list of LFNs to remove. To find all the files in your directory you can use the following script:

> dirac-dms-user-lfns --help
2011-12-07 15:29:16 UTC Framework NOTICE: Get the list of all the user files.
2011-12-07 15:29:16 UTC Framework NOTICE: Usage:
2011-12-07 15:29:16 UTC Framework NOTICE:   dirac-dms-user-lfns [option|cfgfile] ... 
2011-12-07 15:29:16 UTC Framework NOTICE: General options: 
2011-12-07 15:29:16 UTC Framework NOTICE:   -o:  --option=         : Option=value to add 
2011-12-07 15:29:16 UTC Framework NOTICE:   -s:  --section=        : Set base section for relative parsed options 
2011-12-07 15:29:16 UTC Framework NOTICE:   -c:  --cert=           : Use server certificate to connect to Core Services 
2011-12-07 15:29:16 UTC Framework NOTICE:   -d   --debug           : Set debug mode (-dd is extra debug) 
2011-12-07 15:29:16 UTC Framework NOTICE:   -h   --help            : Shows this help 
2011-12-07 15:29:16 UTC Framework NOTICE:  
2011-12-07 15:29:16 UTC Framework NOTICE: Options: 
2011-12-07 15:29:16 UTC Framework NOTICE:   -D:  --Days=           : Match files older than number of days [0] 
2011-12-07 15:29:16 UTC Framework NOTICE:   -M:  --Months=         : Match files older than number of months [0] 
2011-12-07 15:29:16 UTC Framework NOTICE:   -Y:  --Years=          : Match files older than number of years [0] 
2011-12-07 15:29:16 UTC Framework NOTICE:   -w:  --Wildcard=       : Wildcard for matching filenames [*] 
2011-12-07 15:29:16 UTC Framework NOTICE:   -b:  --BaseDir=        : Base directory to begin search (default /[vo]/user/[initial]/[username]) 
2011-12-07 15:29:16 UTC Framework NOTICE:   -e   --EmptyDirs       : Create a list of empty directories 

By using the --Days, --Months and --Years options it is possible to select files which were produced before a given time. This can be useful when trying to remove old data only. The base directory of the LFC to be search can be supplied (by default this is just your home directory in the LFC) which restricts the search to the supplied namespace. Finally, a wild card can be given to find only files of a given type or name. For example, to find the root files in the /lhcb/user/a/acsmith/B+2DStar-D0_0008/3237 directory which are older than 1 month:

> dirac-dms-user-lfns --BaseDir=/lhcb/user/a/acsmith/B+2DStar-D0_0008/3237 -M 1 --Wildcard='*.root'
2009-09-29 14:26:21 UTC dirac-dms-user-lfns.py  INFO: Will search for files in /lhcb/user/a/acsmith/B+2DStar-D0_0008/3237
2009-09-29 14:26:31 UTC dirac-dms-user-lfns.py  INFO: /lhcb/user/a/acsmith/B+2DStar-D0_0008/3237: 0 files, 205 sub-directories
2009-09-29 14:26:32 UTC dirac-dms-user-lfns.py  INFO: /lhcb/user/a/acsmith/B+2DStar-D0_0008/3237/3237444: 1 files, 0 sub-directories
.
.
.
2009-09-29 14:28:29 UTC dirac-dms-user-lfns.py  INFO: /lhcb/user/a/acsmith/B+2DStar-D0_0008/3237/3237837: 1 files, 0 sub-directories
2009-09-29 14:28:29 UTC dirac-dms-user-lfns.py  INFO: 205 matched files have been put in lhcb-user-a-acsmith-B+2DStar-D0_0008-3237.lfns

As you can see this has placed all the files found in your search in a local file (lhcb-user-a-acsmith-B+2DStar-D0_0008-3237.lfns).

Using the option -e or --EmptyDirs, you can get a list of all your empty directories, that use no space but pollute unduly the namespace. Empty directories can then be deleted by the dirac-dms-clean-directory command (see below).

How much data is that?

Once you have a list of files you can check how much data this is by using the following script (passing LFNs or files that contains LFNs as arguments):

> dirac-dms-data-size --help
2009-09-29 14:56:07 UTC Framework  INFO: Usage:
2009-09-29 14:56:07 UTC Framework  INFO: ./dirac-dms-data-size.py (<options>|<cfgFile>)*
2009-09-29 14:56:07 UTC Framework  INFO: Options:
2009-09-29 14:56:07 UTC Framework  INFO: -o:  --option=  :  Option=value to add
2009-09-29 14:56:07 UTC Framework  INFO: -s:  --section=  :  Set base section for relative parsed options
2009-09-29 14:56:07 UTC Framework  INFO: -c:  --cert=  :  Use server certificate to connect to Core Services
2009-09-29 14:56:07 UTC Framework  INFO: -h  --help  :  Shows this help
2009-09-29 14:56:07 UTC Framework  INFO: -u:  --Unit=  :     Unit to use [GB] (MB,GB,TB,PB)

For example to calculate the size (in MB) of the data found in the previous search plus one other file from another directory I can do the following:

> dirac-dms-data-size -u MB /lhcb/user/a/acsmith/B+2DStar-D0_0005/3130/3130610/Tuple.root lhcb-user-a-acsmith-B+2DStar-D0_0008-3237.lfns
2009-09-29 14:58:35 UTC dirac-dms-data-size.py  INFO: ------------------------------
2009-09-29 14:58:35 UTC dirac-dms-data-size.py  INFO: Files          |      Size (MB)
2009-09-29 14:58:35 UTC dirac-dms-data-size.py  INFO: ------------------------------
2009-09-29 14:58:35 UTC dirac-dms-data-size.py  INFO: 206            |          172.7
2009-09-29 14:58:35 UTC dirac-dms-data-size.py  INFO: ------------------------------

How to remove it?

To remove your old files you can use the following script (passing LFNs or files that contains LFNs as arguments):

> dirac-dms-remove-files lhcb-user-a-acsmith-B+2DStar-D0_0002.lfns /lhcb/user/a/acsmith/B+2DStar-D0_0002/3022/3022769/Tuple.root
2009-09-29 15:21:35 UTC dirac-dms-remove-files.py  INFO: ReplicaManager.__removePhysicalReplica: Successfully issued removal request.
2009-09-29 15:21:43 UTC dirac-dms-remove-files.py  INFO: ReplicaManager.__removeCatalogReplica: Removed 100 replicas
2009-09-29 15:22:04 UTC dirac-dms-remove-files.py  INFO: ReplicaManager.__removePhysicalReplica: Successfully issued removal request.
2009-09-29 15:22:12 UTC dirac-dms-remove-files.py  INFO: ReplicaManager.__removeCatalogReplica: Removed 100 replicas
2009-09-29 15:22:19 UTC dirac-dms-remove-files.py  INFO: ReplicaManager.__removePhysicalReplica: Successfully issued removal request.
2009-09-29 15:22:20 UTC dirac-dms-remove-files.py  INFO: ReplicaManager.__removeCatalogReplica: Removed 5 replicas
2009-09-29 15:22:35 UTC dirac-dms-remove-files.py  INFO: Successfully removed 205 files

Now it probably takes some time to sync the corresponding DB, containing the storage space used. So please wait a bit.

I have a lot of empty directories. What should I do ?

Nothing. These will be cleaned up centrally by the Dirac system periodically for you.

-- AndrewCSmith - 2009-09-29