Difference: GridStorageQuota (1 vs. 17)

Revision 172019-08-28 - GiacomoGraziani

Line: 1 to 1
 
META TOPICPARENT name="Trash.LHCbDIRACUserGuides"

DIRAC Grid Storage Quota

Line: 38 to 38
 To obtain your current DISK usage you can perform the following on the CLI:
Changed:
<
<
> dirac-dms-storage-usage-summary --Dir /lhcb/user///
>
>
> dirac-dms-storage-usage-summary --Users=username
Storage usage for user dduck (quota: 2.0 TB)
 DIRAC SE Size (TB) Files
Changed:
<
<

CERN-USER 1.3 2230 CERN-tape 0.0 215 CNAF-FAILOVER 0.0 1 CNAF-USER 0.2 414 GRIDKA-USER 0.3 867 IN2P3-USER 0.2 424 NIKHEF-USER 0.2 488 PIC-FAILOVER 0.0 1 PIC-USER 0.2 393 RAL-USER 0.6 898
>
>

CERN-USER 0.630 2392 CNAF-USER 0.128 726 GRIDKA-USER 0.158 586 IN2P3-USER 0.275 1073 PIC-USER 0.028 164 RAL-USER 0.186 911 RRCKI-USER 0.100 409 SARA-USER 0.103 642 Total (disk) 1.609 6903
 

Revision 162017-06-27 - ThomasLatham

Line: 1 to 1
 
META TOPICPARENT name="Trash.LHCbDIRACUserGuides"

DIRAC Grid Storage Quota

Line: 14 to 14
  no longer works to set the environment. If you want to work with LHCbDirac in bash, instead open a sub-shell with
Changed:
<
<
> lb-run LHCbDirac/latest bash --norc
>
>
> lb-run LHCbDirac/prod bash --norc
  You will then need to do

Revision 152017-03-30 - ChristopherMarkBurr

Line: 1 to 1
 
META TOPICPARENT name="Trash.LHCbDIRACUserGuides"

DIRAC Grid Storage Quota

Line: 38 to 38
 To obtain your current DISK usage you can perform the following on the CLI:
Changed:
<
<
> dirac-dms-storage-usage-summary --Dir /lhcb/user//
>
>
> dirac-dms-storage-usage-summary --Dir /lhcb/user///
 DIRAC SE Size (TB) Files
CERN-USER 1.3 2230

Revision 142016-08-26 - MichaelWilkinson

Line: 1 to 1
 
META TOPICPARENT name="Trash.LHCbDIRACUserGuides"

DIRAC Grid Storage Quota

Line: 6 to 6
  A similar tutorial on working with your grid quota in ganga is given in GridAndDataManagement.
Added:
>
>

Before executing the commands in this tutorial

Since at least v8r3p2, doing

> SetupProject LHCbDirac
no longer works to set the environment. If you want to work with LHCbDirac in bash, instead open a sub-shell with
> lb-run LHCbDirac/latest bash --norc
You will then need to do
> lhcb-proxy-init
Once these things are done, the following commands should work within the bash sub-shell you just opened.
 

What is my current quota?

To obtain your current DISK quota you can perform the following on the CLI:

Deleted:
<
<
> SetupProject LHCbDirac
> lhcb-proxy-init
 > dirac-dms-user-quota
2009-09-29 14:05:49 UTC dirac-dms-user-quota.py INFO: Current quota found to be 10.0 GB
Line: 24 to 38
 To obtain your current DISK usage you can perform the following on the CLI:
Deleted:
<
<
> SetupProject LHCbDirac
 > dirac-dms-storage-usage-summary --Dir /lhcb/user//
DIRAC SE Size (TB) Files

Revision 132011-12-07 - MarcOlivierBettler

Line: 1 to 1
 
META TOPICPARENT name="Trash.LHCbDIRACUserGuides"

DIRAC Grid Storage Quota

Line: 69 to 69
 By using the --Days, --Months and --Years options it is possible to select files which were produced before a given time. This can be useful when trying to remove old data only. The base directory of the LFC to be search can be supplied (by default this is just your home directory in the LFC) which restricts the search to the supplied namespace. Finally, a wild card can be given to find only files of a given type or name. For example, to find the root files in the /lhcb/user/a/acsmith/B+2DStar-D0_0008/3237 directory which are older than 1 month:
Changed:
<
<
> dirac-dms-user-lfns --BaseDir=/lhcb/user/a/acsmith/B+2DStar-D0_0008/3237 -m 1 --Wildcard='*.root'
>
>
> dirac-dms-user-lfns --BaseDir=/lhcb/user/a/acsmith/B+2DStar-D0_0008/3237 -M 1 --Wildcard='*.root'
 2009-09-29 14:26:21 UTC dirac-dms-user-lfns.py INFO: Will search for files in /lhcb/user/a/acsmith/B+2DStar-D0_0008/3237 2009-09-29 14:26:31 UTC dirac-dms-user-lfns.py INFO: /lhcb/user/a/acsmith/B+2DStar-D0_0008/3237: 0 files, 205 sub-directories 2009-09-29 14:26:32 UTC dirac-dms-user-lfns.py INFO: /lhcb/user/a/acsmith/B+2DStar-D0_0008/3237/3237444: 1 files, 0 sub-directories

Revision 122011-12-07 - PatrickSKoppenburg

Line: 1 to 1
 
META TOPICPARENT name="Trash.LHCbDIRACUserGuides"

DIRAC Grid Storage Quota

Line: 47 to 47
 
> dirac-dms-user-lfns --help
Changed:
<
<
2010-11-24 10:11:26 UTC Framework NOTICE: Usage: 2010-11-24 10:11:26 UTC Framework NOTICE: dirac-dms-user-lfns (|)* 2010-11-24 10:11:26 UTC Framework NOTICE: Options: 2010-11-24 10:11:26 UTC Framework NOTICE: -o: --option= : Option=value to add 2010-11-24 10:11:26 UTC Framework NOTICE: -s: --section= : Set base section for relative parsed options 2010-11-24 10:11:26 UTC Framework NOTICE: -c: --cert= : Use server certificate to connect to Core Services 2010-11-24 10:11:26 UTC Framework NOTICE: -h --help : Shows this help 2010-11-24 10:11:26 UTC Framework NOTICE: -d: --Days= : Match files older than number of days [0] 2010-11-24 10:11:26 UTC Framework NOTICE: -m: --Months= : Match files older than number of months [0] 2010-11-24 10:11:26 UTC Framework NOTICE: -y: --Years= : Match files older than number of years [0] 2010-11-24 10:11:26 UTC Framework NOTICE: -w: --Wildcard= : Wildcard for matching filenames [*] 2010-11-24 10:11:26 UTC Framework NOTICE: -b: --BaseDir= : Base directory to begin search (default /[vo]/user/[initial]/[username]) 2010-11-24 10:11:26 UTC Framework NOTICE: -e --EmptyDirs : Create a list of empty directories
>
>
2011-12-07 15:29:16 UTC Framework NOTICE: Get the list of all the user files. 2011-12-07 15:29:16 UTC Framework NOTICE: Usage: 2011-12-07 15:29:16 UTC Framework NOTICE: dirac-dms-user-lfns [option|cfgfile] ... 2011-12-07 15:29:16 UTC Framework NOTICE: General options: 2011-12-07 15:29:16 UTC Framework NOTICE: -o: --option= : Option=value to add 2011-12-07 15:29:16 UTC Framework NOTICE: -s: --section= : Set base section for relative parsed options 2011-12-07 15:29:16 UTC Framework NOTICE: -c: --cert= : Use server certificate to connect to Core Services 2011-12-07 15:29:16 UTC Framework NOTICE: -d --debug : Set debug mode (-dd is extra debug) 2011-12-07 15:29:16 UTC Framework NOTICE: -h --help : Shows this help 2011-12-07 15:29:16 UTC Framework NOTICE: 2011-12-07 15:29:16 UTC Framework NOTICE: Options: 2011-12-07 15:29:16 UTC Framework NOTICE: -D: --Days= : Match files older than number of days [0] 2011-12-07 15:29:16 UTC Framework NOTICE: -M: --Months= : Match files older than number of months [0] 2011-12-07 15:29:16 UTC Framework NOTICE: -Y: --Years= : Match files older than number of years [0] 2011-12-07 15:29:16 UTC Framework NOTICE: -w: --Wildcard= : Wildcard for matching filenames [*] 2011-12-07 15:29:16 UTC Framework NOTICE: -b: --BaseDir= : Base directory to begin search (default /[vo]/user/[initial]/[username]) 2011-12-07 15:29:16 UTC Framework NOTICE: -e --EmptyDirs : Create a list of empty directories
 

By using the --Days, --Months and --Years options it is possible to select files which were produced before a given time. This can be useful when trying to remove old data only. The base directory of the LFC to be search can be supplied (by default this is just your home directory in the LFC) which restricts the search to the supplied namespace. Finally, a wild card can be given to find only files of a given type or name. For example, to find the root files in the /lhcb/user/a/acsmith/B+2DStar-D0_0008/3237 directory which are older than 1 month:

Revision 112011-03-14 - KazuyoshiAkiba

Line: 1 to 1
 
META TOPICPARENT name="Trash.LHCbDIRACUserGuides"

DIRAC Grid Storage Quota

Line: 12 to 12
 
> SetupProject LHCbDirac
Added:
>
>
> lhcb-proxy-init
 > dirac-dms-user-quota
2009-09-29 14:05:49 UTC dirac-dms-user-quota.py INFO: Current quota found to be 10.0 GB

Revision 102011-03-10 - ChristopherRJones

Line: 1 to 1
 
META TOPICPARENT name="Trash.LHCbDIRACUserGuides"

DIRAC Grid Storage Quota

Line: 122 to 122
  Now it probably takes some time to sync the corresponding DB, containing the storage space used. So please wait a bit.
Changed:
<
<

How to remove empty directories

After removing the files you might want to remove also the empty directories left around. The dirac-dms-clean-directory command will remove all files and directories it finds in the path. i.e.

$ dirac-dms-clean-directory /lhcb/user/u/ukerzel/some/old/dir

If you want to restrict the directories to be deleted to empty ones, use first the command dirac-dms-user-lfns --EmptyDirs as indicated above and submit the resulting file to this command as argument. Example:

[lxplus426] ~ $ dirac-dms-user-lfns --EmptyDirs
2010-11-24 13:41:17 UTC dirac-dms-user-lfns  WARN: Server is not who it's supposed to be Connecting to lhcbprod.pic.es and it's lhcb/lhcbprod.pic.es
2010-11-24 13:41:19 UTC dirac-dms-user-lfns  INFO: Will search for files in /lhcb/user/p/phicharp
2010-11-24 13:41:20 UTC dirac-dms-user-lfns  INFO: /lhcb/user/p/phicharp: 1 files, 2 sub-directories
2010-11-24 13:41:20 UTC dirac-dms-user-lfns  INFO: /lhcb/user/p/phicharp/add: 1 files, 0 sub-directories
2010-11-24 13:41:21 UTC dirac-dms-user-lfns  INFO: /lhcb/user/p/phicharp/new: 0 files, 0 sub-directories
2010-11-24 13:41:21 UTC dirac-dms-user-lfns  INFO: 2 matched files have been put in lhcb-user-p-phicharp.lfns
2010-11-24 13:41:21 UTC dirac-dms-user-lfns  INFO: 1 empty directories have been put in lhcb-user-p-phicharp.emptydirs
[lxplus426] ~ $ more lhcb-user-p-phicharp.emptydirs
/lhcb/user/p/phicharp/new
[lxplus426] ~ $ dirac-dms-clean-directory lhcb-user-p-phicharp.emptydirs
2010-11-24 13:42:20 UTC dirac-dms-clean-directory  WARN: Server is not who it's supposed to be Connecting to lhcbprod.pic.es and it's lhcb/lhcbprod.pic.es
2010-11-24 13:42:22 UTC dirac-dms-clean-directory  INFO: Obtaining the catalog contents for 1 directories:
2010-11-24 13:42:22 UTC dirac-dms-clean-directory  INFO: /lhcb/user/p/phicharp/new
2010-11-24 13:42:23 UTC dirac-dms-clean-directory  INFO: Found 0 files
2010-11-24 13:42:23 UTC dirac-dms-clean-directory  INFO: Removing the contents of /lhcb/user/p/phicharp/new at CERN_MC_M-DST
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: Successfully removed 1 files from /lhcb/user/p/phicharp/new at CERN_MC_M-DST
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: Removing the contents of /lhcb/user/p/phicharp/new at CNAF-RAW
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: The directory /lhcb/user/p/phicharp/new does not exist at CNAF-RAW
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: Removing the contents of /lhcb/user/p/phicharp/new at CNAF_MC-DST
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: The directory /lhcb/user/p/phicharp/new does not exist at CNAF_MC-DST
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: Removing the contents of /lhcb/user/p/phicharp/new at CNAF_MC_M-DST
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: The directory /lhcb/user/p/phicharp/new does not exist at CNAF_MC_M-DST
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: Removing the contents of /lhcb/user/p/phicharp/new at GRIDKA_MC_M-DST
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: The directory /lhcb/user/p/phicharp/new does not exist at GRIDKA_MC_M-DST
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: Removing the contents of /lhcb/user/p/phicharp/new at IN2P3_MC_M-DST
2010-11-24 13:42:26 UTC dirac-dms-clean-directory  INFO: The directory /lhcb/user/p/phicharp/new does not exist at IN2P3_MC_M-DST
2010-11-24 13:42:26 UTC dirac-dms-clean-directory  INFO: Removing the contents of /lhcb/user/p/phicharp/new at NIKHEF_MC_M-DST
2010-11-24 13:42:26 UTC dirac-dms-clean-directory  INFO: The directory /lhcb/user/p/phicharp/new does not exist at NIKHEF_MC_M-DST
2010-11-24 13:42:26 UTC dirac-dms-clean-directory  INFO: Removing the contents of /lhcb/user/p/phicharp/new at PIC_MC_M-DST
2010-11-24 13:42:26 UTC dirac-dms-clean-directory  INFO: The directory /lhcb/user/p/phicharp/new does not exist at PIC_MC_M-DST
2010-11-24 13:42:26 UTC dirac-dms-clean-directory  INFO: Removing the contents of /lhcb/user/p/phicharp/new at RAL_MC_M-DST
2010-11-24 13:42:27 UTC dirac-dms-clean-directory  INFO: The directory /lhcb/user/p/phicharp/new does not exist at RAL_MC_M-DST
2010-11-24 13:42:27 UTC dirac-dms-clean-directory  INFO: Successfully removed directory. /lhcb/user/p/phicharp/new
>
>

I have a lot of empty directories. What should I do ?

Nothing. These will be cleaned up centrally by the Dirac system periodically for you.

  -- AndrewCSmith - 2009-09-29 \ No newline at end of file

Revision 92010-12-22 - RobLambert

Line: 1 to 1
 
META TOPICPARENT name="Trash.LHCbDIRACUserGuides"

DIRAC Grid Storage Quota

Added:
>
>
A similar tutorial on working with your grid quota in ganga is given in GridAndDataManagement.
 

What is my current quota?

To obtain your current DISK quota you can perform the following on the CLI:

Revision 82010-12-21 - RobLambert

Line: 1 to 1
 
META TOPICPARENT name="Trash.LHCbDIRACUserGuides"

DIRAC Grid Storage Quota

Line: 118 to 118
 2009-09-29 15:22:35 UTC dirac-dms-remove-files.py INFO: Successfully removed 205 files
Added:
>
>
Now it probably takes some time to sync the corresponding DB, containing the storage space used. So please wait a bit.
 

How to remove empty directories

After removing the files you might want to remove also the empty directories left around. The dirac-dms-clean-directory command will remove all files and directories it finds in the path. i.e.

Revision 72010-11-25 - RobLambert

Line: 1 to 1
 
META TOPICPARENT name="Trash.LHCbDIRACUserGuides"

DIRAC Grid Storage Quota

Line: 16 to 16
  As you can see the current quota for my account is 10GB.
Added:
>
>

What is my current usage?

To obtain your current DISK usage you can perform the following on the CLI:

> SetupProject LHCbDirac
> dirac-dms-storage-usage-summary --Dir /lhcb/user/<u>/<username>
DIRAC SE             Size (TB)            Files               
--------------------------------------------------
CERN-USER            1.3                  2230                
CERN-tape            0.0                  215                 
CNAF-FAILOVER        0.0                  1                   
CNAF-USER            0.2                  414                 
GRIDKA-USER          0.3                  867                 
IN2P3-USER           0.2                  424                 
NIKHEF-USER          0.2                  488                 
PIC-FAILOVER         0.0                  1                   
PIC-USER             0.2                  393                 
RAL-USER             0.6                  898
 

How to find old data?

To remove data one must first obtain the list of LFNs to remove. To find all the files in your directory you can use the following script:

Revision 62010-11-24 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="Trash.LHCbDIRACUserGuides"

DIRAC Grid Storage Quota

Line: 104 to 104
 $ dirac-dms-clean-directory /lhcb/user/u/ukerzel/some/old/dir
Changed:
<
<
If you want to restrict the directories to be deleted to empty ones, use first the command dirac-dms-user-lfns --EmptyDirs as indicated above and submit the resulting file to this command as argument.
>
>
If you want to restrict the directories to be deleted to empty ones, use first the command dirac-dms-user-lfns --EmptyDirs as indicated above and submit the resulting file to this command as argument. Example:

[lxplus426] ~ $ dirac-dms-user-lfns --EmptyDirs
2010-11-24 13:41:17 UTC dirac-dms-user-lfns  WARN: Server is not who it's supposed to be Connecting to lhcbprod.pic.es and it's lhcb/lhcbprod.pic.es
2010-11-24 13:41:19 UTC dirac-dms-user-lfns  INFO: Will search for files in /lhcb/user/p/phicharp
2010-11-24 13:41:20 UTC dirac-dms-user-lfns  INFO: /lhcb/user/p/phicharp: 1 files, 2 sub-directories
2010-11-24 13:41:20 UTC dirac-dms-user-lfns  INFO: /lhcb/user/p/phicharp/add: 1 files, 0 sub-directories
2010-11-24 13:41:21 UTC dirac-dms-user-lfns  INFO: /lhcb/user/p/phicharp/new: 0 files, 0 sub-directories
2010-11-24 13:41:21 UTC dirac-dms-user-lfns  INFO: 2 matched files have been put in lhcb-user-p-phicharp.lfns
2010-11-24 13:41:21 UTC dirac-dms-user-lfns  INFO: 1 empty directories have been put in lhcb-user-p-phicharp.emptydirs
[lxplus426] ~ $ more lhcb-user-p-phicharp.emptydirs
/lhcb/user/p/phicharp/new
[lxplus426] ~ $ dirac-dms-clean-directory lhcb-user-p-phicharp.emptydirs
2010-11-24 13:42:20 UTC dirac-dms-clean-directory  WARN: Server is not who it's supposed to be Connecting to lhcbprod.pic.es and it's lhcb/lhcbprod.pic.es
2010-11-24 13:42:22 UTC dirac-dms-clean-directory  INFO: Obtaining the catalog contents for 1 directories:
2010-11-24 13:42:22 UTC dirac-dms-clean-directory  INFO: /lhcb/user/p/phicharp/new
2010-11-24 13:42:23 UTC dirac-dms-clean-directory  INFO: Found 0 files
2010-11-24 13:42:23 UTC dirac-dms-clean-directory  INFO: Removing the contents of /lhcb/user/p/phicharp/new at CERN_MC_M-DST
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: Successfully removed 1 files from /lhcb/user/p/phicharp/new at CERN_MC_M-DST
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: Removing the contents of /lhcb/user/p/phicharp/new at CNAF-RAW
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: The directory /lhcb/user/p/phicharp/new does not exist at CNAF-RAW
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: Removing the contents of /lhcb/user/p/phicharp/new at CNAF_MC-DST
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: The directory /lhcb/user/p/phicharp/new does not exist at CNAF_MC-DST
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: Removing the contents of /lhcb/user/p/phicharp/new at CNAF_MC_M-DST
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: The directory /lhcb/user/p/phicharp/new does not exist at CNAF_MC_M-DST
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: Removing the contents of /lhcb/user/p/phicharp/new at GRIDKA_MC_M-DST
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: The directory /lhcb/user/p/phicharp/new does not exist at GRIDKA_MC_M-DST
2010-11-24 13:42:25 UTC dirac-dms-clean-directory  INFO: Removing the contents of /lhcb/user/p/phicharp/new at IN2P3_MC_M-DST
2010-11-24 13:42:26 UTC dirac-dms-clean-directory  INFO: The directory /lhcb/user/p/phicharp/new does not exist at IN2P3_MC_M-DST
2010-11-24 13:42:26 UTC dirac-dms-clean-directory  INFO: Removing the contents of /lhcb/user/p/phicharp/new at NIKHEF_MC_M-DST
2010-11-24 13:42:26 UTC dirac-dms-clean-directory  INFO: The directory /lhcb/user/p/phicharp/new does not exist at NIKHEF_MC_M-DST
2010-11-24 13:42:26 UTC dirac-dms-clean-directory  INFO: Removing the contents of /lhcb/user/p/phicharp/new at PIC_MC_M-DST
2010-11-24 13:42:26 UTC dirac-dms-clean-directory  INFO: The directory /lhcb/user/p/phicharp/new does not exist at PIC_MC_M-DST
2010-11-24 13:42:26 UTC dirac-dms-clean-directory  INFO: Removing the contents of /lhcb/user/p/phicharp/new at RAL_MC_M-DST
2010-11-24 13:42:27 UTC dirac-dms-clean-directory  INFO: The directory /lhcb/user/p/phicharp/new does not exist at RAL_MC_M-DST
2010-11-24 13:42:27 UTC dirac-dms-clean-directory  INFO: Successfully removed directory. /lhcb/user/p/phicharp/new
 -- AndrewCSmith - 2009-09-29 \ No newline at end of file

Revision 52010-11-24 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="Trash.LHCbDIRACUserGuides"

DIRAC Grid Storage Quota

Line: 22 to 22
 
> dirac-dms-user-lfns --help
Changed:
<
<
2009-09-29 14:07:20 UTC Framework INFO: Usage: 2009-09-29 14:07:20 UTC Framework INFO: ./dirac-dms-user-lfns.py (|)* 2009-09-29 14:07:20 UTC Framework INFO: Options: 2009-09-29 14:07:20 UTC Framework INFO: -o: --option= : Option=value to add 2009-09-29 14:07:20 UTC Framework INFO: -s: --section= : Set base section for relative parsed options 2009-09-29 14:07:20 UTC Framework INFO: -c: --cert= : Use server certificate to connect to Core Services 2009-09-29 14:07:20 UTC Framework INFO: -h --help : Shows this help 2009-09-29 14:07:20 UTC Framework INFO: -d: --Days= : Match files older than number of days [0] 2009-09-29 14:07:20 UTC Framework INFO: -m: --Months= : Match files older than number of months [0] 2009-09-29 14:07:20 UTC Framework INFO: -y: --Years= : Match files older than number of years [0] 2009-09-29 14:07:20 UTC Framework INFO: -w: --Wildcard= : Wildcard for matching filenames [*] 2009-09-29 14:07:20 UTC Framework INFO: -b: --BaseDir= : Base directory to begin search [/lhcb/user/initial/username]
>
>
2010-11-24 10:11:26 UTC Framework NOTICE: Usage: 2010-11-24 10:11:26 UTC Framework NOTICE: dirac-dms-user-lfns (|)* 2010-11-24 10:11:26 UTC Framework NOTICE: Options: 2010-11-24 10:11:26 UTC Framework NOTICE: -o: --option= : Option=value to add 2010-11-24 10:11:26 UTC Framework NOTICE: -s: --section= : Set base section for relative parsed options 2010-11-24 10:11:26 UTC Framework NOTICE: -c: --cert= : Use server certificate to connect to Core Services 2010-11-24 10:11:26 UTC Framework NOTICE: -h --help : Shows this help 2010-11-24 10:11:26 UTC Framework NOTICE: -d: --Days= : Match files older than number of days [0] 2010-11-24 10:11:26 UTC Framework NOTICE: -m: --Months= : Match files older than number of months [0] 2010-11-24 10:11:26 UTC Framework NOTICE: -y: --Years= : Match files older than number of years [0] 2010-11-24 10:11:26 UTC Framework NOTICE: -w: --Wildcard= : Wildcard for matching filenames [*] 2010-11-24 10:11:26 UTC Framework NOTICE: -b: --BaseDir= : Base directory to begin search (default /[vo]/user/[initial]/[username]) 2010-11-24 10:11:26 UTC Framework NOTICE: -e --EmptyDirs : Create a list of empty directories
 

By using the --Days, --Months and --Years options it is possible to select files which were produced before a given time. This can be useful when trying to remove old data only. The base directory of the LFC to be search can be supplied (by default this is just your home directory in the LFC) which restricts the search to the supplied namespace. Finally, a wild card can be given to find only files of a given type or name. For example, to find the root files in the /lhcb/user/a/acsmith/B+2DStar-D0_0008/3237 directory which are older than 1 month:

Line: 52 to 53
  As you can see this has placed all the files found in your search in a local file (lhcb-user-a-acsmith-B+2DStar-D0_0008-3237.lfns).
Added:
>
>
Using the option -e or --EmptyDirs, you can get a list of all your empty directories, that use no space but pollute unduly the namespace. Empty directories can then be deleted by the dirac-dms-clean-directory command (see below).
 

How much data is that?

Once you have a list of files you can check how much data this is by using the following script (passing LFNs or files that contains LFNs as arguments):
Line: 101 to 104
 $ dirac-dms-clean-directory /lhcb/user/u/ukerzel/some/old/dir
Added:
>
>
If you want to restrict the directories to be deleted to empty ones, use first the command dirac-dms-user-lfns --EmptyDirs as indicated above and submit the resulting file to this command as argument.
 -- AndrewCSmith - 2009-09-29

Revision 42010-11-11 - unknown

Line: 1 to 1
 
META TOPICPARENT name="Trash.LHCbDIRACUserGuides"

DIRAC Grid Storage Quota

Line: 93 to 93
 2009-09-29 15:22:35 UTC dirac-dms-remove-files.py INFO: Successfully removed 205 files
Added:
>
>

How to remove empty directories

After removing the files you might want to remove also the empty directories left around. The dirac-dms-clean-directory command will remove all files and directories it finds in the path. i.e.

$ dirac-dms-clean-directory /lhcb/user/u/ukerzel/some/old/dir
 -- AndrewCSmith - 2009-09-29 \ No newline at end of file

Revision 32010-02-22 - JoelClosier

Line: 1 to 1
 
META TOPICPARENT name="Trash.LHCbDIRACUserGuides"

DIRAC Grid Storage Quota

Line: 9 to 9
 To obtain your current DISK quota you can perform the following on the CLI:
Changed:
<
<
> SetupProject Dirac
>
>
> SetupProject LHCbDirac
 > dirac-dms-user-quota
2009-09-29 14:05:49 UTC dirac-dms-user-quota.py INFO: Current quota found to be 10.0 GB

Revision 12009-09-29 - unknown

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="DIRACUserGuides"

DIRAC Grid Storage Quota

What is my current quota?

To obtain your current DISK quota you can perform the following on the CLI:

> SetupProject Dirac
> dirac-dms-user-quota
2009-09-29 14:05:49 UTC dirac-dms-user-quota.py  INFO: Current quota found to be 10.0 GB

As you can see the current quota for my account is 10GB.

How to find old data?

To remove data one must first obtain the list of LFNs to remove. To find all the files in your directory you can use the following script:

> dirac-dms-user-lfns --help
2009-09-29 14:07:20 UTC Framework  INFO: Usage:
2009-09-29 14:07:20 UTC Framework  INFO: ./dirac-dms-user-lfns.py (<options>|<cfgFile>)*
2009-09-29 14:07:20 UTC Framework  INFO: Options:
2009-09-29 14:07:20 UTC Framework  INFO: -o:  --option=  :  Option=value to add
2009-09-29 14:07:20 UTC Framework  INFO: -s:  --section=  :  Set base section for relative parsed options
2009-09-29 14:07:20 UTC Framework  INFO: -c:  --cert=  :  Use server certificate to connect to Core Services
2009-09-29 14:07:20 UTC Framework  INFO: -h  --help  :  Shows this help
2009-09-29 14:07:20 UTC Framework  INFO: -d:  --Days=  :       Match files older than number of days [0]
2009-09-29 14:07:20 UTC Framework  INFO: -m:  --Months=  :     Match files older than number of months [0]
2009-09-29 14:07:20 UTC Framework  INFO: -y:  --Years=  :      Match files older than number of years [0]
2009-09-29 14:07:20 UTC Framework  INFO: -w:  --Wildcard=  :   Wildcard for matching filenames [*]
2009-09-29 14:07:20 UTC Framework  INFO: -b:  --BaseDir=  :    Base directory to begin search [/lhcb/user/initial/username]

By using the --Days, --Months and --Years options it is possible to select files which were produced before a given time. This can be useful when trying to remove old data only. The base directory of the LFC to be search can be supplied (by default this is just your home directory in the LFC) which restricts the search to the supplied namespace. Finally, a wild card can be given to find only files of a given type or name. For example, to find the root files in the /lhcb/user/a/acsmith/B+2DStar-D0_0008/3237 directory which are older than 1 month:

> dirac-dms-user-lfns --BaseDir=/lhcb/user/a/acsmith/B+2DStar-D0_0008/3237 -m 1 --Wildcard='*.root'
2009-09-29 14:26:21 UTC dirac-dms-user-lfns.py  INFO: Will search for files in /lhcb/user/a/acsmith/B+2DStar-D0_0008/3237
2009-09-29 14:26:31 UTC dirac-dms-user-lfns.py  INFO: /lhcb/user/a/acsmith/B+2DStar-D0_0008/3237: 0 files, 205 sub-directories
2009-09-29 14:26:32 UTC dirac-dms-user-lfns.py  INFO: /lhcb/user/a/acsmith/B+2DStar-D0_0008/3237/3237444: 1 files, 0 sub-directories
.
.
.
2009-09-29 14:28:29 UTC dirac-dms-user-lfns.py  INFO: /lhcb/user/a/acsmith/B+2DStar-D0_0008/3237/3237837: 1 files, 0 sub-directories
2009-09-29 14:28:29 UTC dirac-dms-user-lfns.py  INFO: 205 matched files have been put in lhcb-user-a-acsmith-B+2DStar-D0_0008-3237.lfns

As you can see this has placed all the files found in your search in a local file (lhcb-user-a-acsmith-B+2DStar-D0_0008-3237.lfns).

How much data is that?

Once you have a list of files you can check how much data this is by using the following script (passing LFNs or files that contains LFNs as arguments):

> dirac-dms-data-size --help
2009-09-29 14:56:07 UTC Framework  INFO: Usage:
2009-09-29 14:56:07 UTC Framework  INFO: ./dirac-dms-data-size.py (<options>|<cfgFile>)*
2009-09-29 14:56:07 UTC Framework  INFO: Options:
2009-09-29 14:56:07 UTC Framework  INFO: -o:  --option=  :  Option=value to add
2009-09-29 14:56:07 UTC Framework  INFO: -s:  --section=  :  Set base section for relative parsed options
2009-09-29 14:56:07 UTC Framework  INFO: -c:  --cert=  :  Use server certificate to connect to Core Services
2009-09-29 14:56:07 UTC Framework  INFO: -h  --help  :  Shows this help
2009-09-29 14:56:07 UTC Framework  INFO: -u:  --Unit=  :     Unit to use [GB] (MB,GB,TB,PB)

For example to calculate the size (in MB) of the data found in the previous search plus one other file from another directory I can do the following:

> dirac-dms-data-size -u MB /lhcb/user/a/acsmith/B+2DStar-D0_0005/3130/3130610/Tuple.root lhcb-user-a-acsmith-B+2DStar-D0_0008-3237.lfns
2009-09-29 14:58:35 UTC dirac-dms-data-size.py  INFO: ------------------------------
2009-09-29 14:58:35 UTC dirac-dms-data-size.py  INFO: Files          |      Size (MB)
2009-09-29 14:58:35 UTC dirac-dms-data-size.py  INFO: ------------------------------
2009-09-29 14:58:35 UTC dirac-dms-data-size.py  INFO: 206            |          172.7
2009-09-29 14:58:35 UTC dirac-dms-data-size.py  INFO: ------------------------------

How to remove it?

To remove your old files you can use the following script (passing LFNs or files that contains LFNs as arguments):

> dirac-dms-remove-files lhcb-user-a-acsmith-B+2DStar-D0_0002.lfns /lhcb/user/a/acsmith/B+2DStar-D0_0002/3022/3022769/Tuple.root
2009-09-29 15:21:35 UTC dirac-dms-remove-files.py  INFO: ReplicaManager.__removePhysicalReplica: Successfully issued removal request.
2009-09-29 15:21:43 UTC dirac-dms-remove-files.py  INFO: ReplicaManager.__removeCatalogReplica: Removed 100 replicas
2009-09-29 15:22:04 UTC dirac-dms-remove-files.py  INFO: ReplicaManager.__removePhysicalReplica: Successfully issued removal request.
2009-09-29 15:22:12 UTC dirac-dms-remove-files.py  INFO: ReplicaManager.__removeCatalogReplica: Removed 100 replicas
2009-09-29 15:22:19 UTC dirac-dms-remove-files.py  INFO: ReplicaManager.__removePhysicalReplica: Successfully issued removal request.
2009-09-29 15:22:20 UTC dirac-dms-remove-files.py  INFO: ReplicaManager.__removeCatalogReplica: Removed 5 replicas
2009-09-29 15:22:35 UTC dirac-dms-remove-files.py  INFO: Successfully removed 205 files

-- AndrewCSmith - 2009-09-29

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback