DIRAC Grid Storage Usage

Web interface

The web interface to the view Grid storage usage is available here. Clicking on this link should take you to a summary page showing the directories, the number of replicas in those directories and their total size (with the largest visible at the top). It should look something like this:

StorageUsage.png

The important features of the webpage are highlighted in the graphic below:

StorageUsage-Highlighted.png

On the left you can see that you are automatically taken to your default home directory on the Grid. This should correspond to the following convention based on the username known to DIRAC: /lhcb/user/initial/username i.e. for acsmith the default path is /lhcb/user/a/acsmith. On the top right you can see the number of replicas and the total size of all the replicas in a given directory. In this example the directory with the largest usage (/lhcb/user/a/acsmith/B+2DStar-D0_0002) contains a single replica with a size of ~170MB. At the bottom right the total number of directories in use (468) can be seen. By refining the search in the directory selection box (seen on the left of the image) the usage of a subset of the displayed directories can be obtained. By clicking on the '+' button highlighted in the bottom right a summary of the usage at each Grid SE can be obtained. For this example the usage is shown in detail below:

StorageUsage-Summary-Highlighted.png

In this example there are 468 replicas with a total size of 444.1MB at the CERN-USER SE. For this example the SE usage is rather simple. By changing the directory being search to /lhcb/MC/MC09 we can see a current snapshot of the SE usage for all MC09 data:

StorageUsage-Summary-Highlighted2.png

When querying for production data it is also possible to obtain the usage for individual productions (e.g. 4838) or file types (e.g. DST, XDST, SIM, RAW etc.). For example, to obtain the usage of the MC09 minimum bias DSTs at CERN the following selection can be made:

StorageUsage-ProductionFile-Type-SE.png

It should be noted that the selections for a logical AND when being performed on the database backend and therefore in this example the directory selection is redundant.

DIRAC CLI

The functionality available through the webpage is also available through the command line using the following script:

> SetupProject LHCbDirac
> dirac-dms-storage-usage-summary --help
2009-08-31 19:05:45 UTC Framework  INFO: Usage:
2009-08-31 19:05:45 UTC Framework  INFO: ./DIRAC/DataManagementSystem/scripts/dirac-dms-storage-usage-summary.py (<options>|<cfgFile>)*
2009-08-31 19:05:45 UTC Framework  INFO: Options:
2009-08-31 19:05:45 UTC Framework  INFO: -o:  --option=  :  Option=value to add
2009-08-31 19:05:45 UTC Framework  INFO: -s:  --section=  :  Set base section for relative parsed options
2009-08-31 19:05:45 UTC Framework  INFO: -c:  --cert=  :  Use server certificate to connect to Core Services
2009-08-31 19:05:45 UTC Framework  INFO: -h  --help  :  Shows this help
2009-08-31 19:05:45 UTC Framework  INFO: -u:  --Unit=  :     Unit to use [TB] (MB,GB,TB,PB)
2009-08-31 19:05:45 UTC Framework  INFO: -d:  --Dir=  :     Dir to search [ALL]
2009-08-31 19:05:45 UTC Framework  INFO: -t:  --Type=  :     File type to search [ALL]
2009-08-31 19:05:45 UTC Framework  INFO: -p:  --Prod=  :     Production ID to search [ALL]
2009-08-31 19:05:45 UTC Framework  INFO: -g:  --Sites=  :    Sites to consider [ALL] (space or comma seperated list)
2009-08-31 19:05:45 UTC Framework  INFO: -c:  --SEs=  :    SEs to consider [ALL] (space or comma seperated list)
2009-08-31 19:05:45 UTC Framework  INFO: -l  --LCG  :    Group results by tape and disk

For example to obtain the usage of the my user directory (denominated in MB) I do the following:

> dirac-dms-storage-usage-summary --Unit=MB --Dir=/lhcb/user/a/acsmith
DIRAC SE             Size (MB)            Files               
--------------------------------------------------
CERN-USER            465.7                468                 

To get the usage of all the MC09 data one can do the following:

> dirac-dms-storage-usage-summary --Dir=/lhcb/MC/MC09
DIRAC SE             Size (TB)            Files               
--------------------------------------------------
CERN-FAILOVER        0.0                  4                   
CERN_MC_M-DST        102.0                54780               
CNAF-DST             0.0                  1                   
CNAF-FAILOVER        0.0                  13                  
CNAF_MC-DST          17.2                 12924               
CNAF_MC_M-DST        16.2                 7180                
GRIDKA-FAILOVER      0.0                  14                  
GRIDKA_MC-DST        15.8                 7868                
GRIDKA_MC_M-DST      19.2                 7541                
IN2P3-FAILOVER       0.0                  14                  
IN2P3_MC-DST         15.8                 8172                
IN2P3_MC_M-DST       18.9                 8137                
NIKHEF-FAILOVER      0.0                  15                  
NIKHEF_MC-DST        17.9                 10165               
NIKHEF_MC_M-DST      12.1                 6588                
PIC-FAILOVER         0.0                  8                   
PIC_MC-DST           16.2                 8345                
PIC_MC_M-DST         20.5                 11704               
RAL-FAILOVER         0.0                  17                  
RAL_MC-DST           18.3                 10155               
RAL_MC_M-DST         12.3                 7627                

To get the usage of the DSTs from production 4838 at one can do the following

> dirac-dms-storage-usage-summary --Prod=4838 --Type=DST
DIRAC SE             Size (TB)            Files               
--------------------------------------------------
CERN_MC_M-DST        28.4                 25130               
CNAF_MC-DST          4.7                  4146                
CNAF_MC_M-DST        4.5                  4026                
GRIDKA_MC-DST        4.7                  4128                
GRIDKA_MC_M-DST      5.3                  4703                
IN2P3_MC-DST         4.4                  3887                
IN2P3_MC_M-DST       6.2                  5480                
NIKHEF_MC-DST        4.8                  4248                
NIKHEF_MC_M-DST      3.7                  3249                
PIC_MC-DST           5.1                  4552                
PIC_MC_M-DST         3.6                  3152                
RAL_MC-DST           4.7                  4173                
RAL_MC_M-DST         5.1                  4520                

To obtain the usage at a given list of SEs the --SEs option can be used:

> dirac-dms-storage-usage-summary --Prod=4838 --Type=DST --SEs="GRIDKA_MC-DST,GRIDKA_MC_M-DST"
DIRAC SE             Size (TB)            Files               
--------------------------------------------------
GRIDKA_MC-DST        4.7                  4128                
GRIDKA_MC_M-DST      5.3                  4703                

Since many SEs can be associated to a DIRAC site the --Sites option can be used to obtain the usage for all the associated SEs:

> dirac-dms-storage-usage-summary --Sites="LCG.IN2P3.fr"
DIRAC SE             Size (TB)            Files               
--------------------------------------------------
IN2P3-FAILOVER       0.1                  249                 
IN2P3-RAW            5.0                  2907                
IN2P3-RDST           0.0                  54                  
IN2P3-USER           1.5                  53556               
IN2P3-disk           68.7                 359203              
IN2P3-tape           7.2                  43638               
IN2P3_M-DST          0.0                  27                  
IN2P3_MC-DST         15.8                 8189                
IN2P3_MC_M-DST       18.9                 8137                

The --LCG option sums all the files present on T1D* and T*D1 storages to give a usage of disk and tape resources. For example the previous selection with the --LCG option gives:

> dirac-dms-storage-usage-summary.py --Sites="LCG.IN2P3.fr" --LCG
Storage Type         Size (TB)            Files               
--------------------------------------------------
T1D*                 31.1                 54763               
T*D1                 104.9                429361              

-- AndrewCSmith - 2009-08-31

Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf StorageUsage-Highlighted.pdf r1 manage 184.9 K 2009-08-31 - 20:01 UnknownUser User storage usage summary web page with important features highlighted
PNGpng StorageUsage-Highlighted.png r1 manage 331.1 K 2009-08-31 - 20:05 UnknownUser User storage usage summary web page with important features highlighted
PNGpng StorageUsage-ProductionFile-Type-SE.png r1 manage 163.4 K 2009-08-31 - 20:50 UnknownUser Example storage usage selection for MC09 MB sample using productionID, file type and SE.
PNGpng StorageUsage-Summary-Highlighted.png r1 manage 324.2 K 2009-08-31 - 20:19 UnknownUser SE usage for a example user.
PNGpng StorageUsage-Summary-Highlighted2.png r1 manage 407.6 K 2009-08-31 - 20:29 UnknownUser SE usage for current MC09 sample
PNGpng StorageUsage.png r1 manage 261.3 K 2009-08-31 - 18:48 UnknownUser User storage usage summary web page
Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r5 - 2016-05-24 - JasonAndrews
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback