A review of space token deployment and space usage at sites

(ALL DIFFERENCES ARE ALWAYS LFC-SRM)

CERN

Related GGUS tickets Some LHCb files outside the space tokens

Update at July 2011

LHCb_USER :
Space usage (TB): SRM: 112.4 and LFC: 98.5 => diff= -13.8 
LHCb-Disk :
Space usage (TB): SRM: 912.1 and LFC: 840.5 => diff= -71.6 

Comments:

• For LHCb_USER: big discrepancy. Probably due to the well known issue, i.e. the SRM interface for Castor reports the space of only the disk servers currently on-line
• For LHCb-Disk big discrepancy, more space used according to SRM than LFC. For Castor at CERN these data are not really meaningful (see previous point).

CNAF

Summary of the situation: everything fine: the only existing space tokens are the new ones (LHCB-Disk, LHCb-USER, LHCb-Tape) and all the data have been migrated there. OK!

Update at March 2012

Space token LHCb-Tape to be checked

From LFC: Files: 203203, Size: 625.15 TB
From storage dumps: Files: 199644, Size: 612.71 TB - last update 2012-03-08 07:13:44

Space token LHCb-Disk to be checked

From LFC: Files: 222865, Size: 466.82 TB
From SRM: Total Assigned Space: 540.00, Used Space: 475.45, Free Space: 64.55 TB
From storage dumps: Files: 210974, Size: 448.56 TB - last update 2012-03-08 07:13:55

Space token LHCb_USER OK

From LFC: Files: 512080, Size: 59.27 TB
From SRM: Total Assigned Space: 61.02, Used Space: 59.25, Free Space: 1.77 TB
From storage dumps: Files: 499275, Size: 60.59 TB - last update 2012-03-08 07:13:32

Update at October 2011

--------- USER space ---------------------------------------------------
LFC: 34.3 SRM: 35.0 => diff -0.7
------------- LCG.CNAF.it - LHCb-Disk :
LCG.CNAF.it-LHCb-Disk Space usage (TB): SRM: 399.9 and LFC: 400.2 => diff= 0.3 
--------- Disk tokens ---------------------------------------------------
LFC: 389.8 summing the Dirac SEs: ['CNAF-DST', 'CNAF_M-DST', 'CNAF-FAILOVER', 'CNAF_MC_M-DST', 'CNAF_MC-DST'] 
Total disk usage from Storage dumps: 389.0 TB, from space tokens ['LHCb-Disk'] 
--------- Tape tokens ---------------------------------------------------
LFC: 475.5 summing the Dirac SEs: ['CNAF-RAW', 'CNAF-RDST', 'CNAF-ARCHIVE'] 
Total tape usage: 476.2 TB, from space tokens ['LHCb-Tape'] 

Everything OK.

Update at July 2011

LHCb_USER :
Space usage (TB): SRM: 29.6 and LFC: 28.8 => diff= -0.8 
LHCb-Disk :
Space usage (TB): SRM: 352.3 and LFC: 352.2 => diff= -0.1 
Remaining used space in 2010 space tokens:
ST: LHCb_RAW SRM used space: 155.0 
ST: LHCb_RDST SRM used space: 155.0 
ST: LHCb_M-DST SRM used space: 352.3 
ST: LHCb_DST SRM used space: 352.3 
ST: LHCb_MC_M-DST SRM used space: 352.3 
ST: LHCb_MC_DST SRM used space: 352.3 
Info not available for ST LHCb_HISTOS
ST: LHCb_FAILOVER SRM used space: 352.3 
Total remaning space: 2071.8 

comments:

• almost no discrepancy for LHCb-Disk and LHCb_USER. And there is always a bit more data in the LFC. OK.

GRIDKA

Summary of the situation: there is a clear separation between the pools assigned to the T1D* space tokens, and the pools assigned to the T0D1 spaces. So, if you have to re-assign some free space from T1D* pools to the T0D1 pools, it can be a not so easy operation. In our particular case, after removal campaigns, some space is freed up in the old T0D1 tokens ( LHCb-DST and LHCb-MC-DST) and this can be easily re-assigned to LHCb-Disk. And on the other hand, some space is freed in the old LHCb-M-DST and LHCb-M-MC-DST (which were T1D1 spaces), and this free space cannot be easily assigned to LHCb-Disk.

Related GGUS tickets:

• space tokens situation at GRIDKA 2011-08-22: it has been agreed that, when needed, the experiment will notify the site though GGUS or mail to do the space re-allocation
• request for storage dumps 2011-08-31 . Space tokens dump provided since 2011-10-06 .
• available disk space at GRIDKA 2011-10-04 About re-allocation of space and difficulties to move it from T1D* to T0D0.
• Completing space token migration 2012-02-17 Ongoing...

Update at March 2012

Space token LHCb-Tape probably some data outside space tokens

From LFC: Files: 208308, Size: 608.61 TB
From storage dumps: Files: 175702, Size: 562.39 TB - last update 2012-03-16 15:51:54

Space token LHCb-Disk ok

From LFC: Files: 220852, Size: 406.15 TB
From SRM: Total Assigned Space: 495.00, Used Space: 408.87, Free Space: 86.13 TB
From storage dumps: Files: 218084, Size: 404.14 TB - last update 2012-03-16 15:52:09

Space token LHCb_USER ok

From LFC: Files: 451771, Size: 45.28 TB
From SRM: Total Assigned Space: 60.00, Used Space: 45.12, Free Space: 14.88 TB
From storage dumps: Files: 451909, Size: 45.11 TB - last update 2012-03-16 15:52:40

Update at Oct 2011

--------- USER space ---------------------------------------------------
LFC: 31.7 SRM: 31.2 => diff 0.5
--------- Disk tokens ---------------------------------------------------
LFC: 382.5 summing the Dirac SEs: ['GRIDKA-DST', 'GRIDKA_M-DST', 'GRIDKA-FAILOVER', 'GRIDKA_MC_M-DST', 'GRIDKA_MC-DST'] 
Total disk usage from Storage dumps: 371.3 TB, from space tokens ['LHCb-Disk', 'LHCb_DST', 'LHCb_FAILOVER', 'LHCb_M-DST', 'LHCb_MC_DST', 'LHCb_MC_M-DST'] 
--------- Tape tokens ---------------------------------------------------
LFC: 510.5 summing the Dirac SEs: ['GRIDKA-RAW', 'GRIDKA-RDST', 'GRIDKA-ARCHIVE'] 
Total tape usage: 417.2 TB, from space tokens ['LHCb-Tape', 'LHCb_RDST'] 

• LHCb_USER ok.
• LHCb-Disk: more data in the LFC than on storage: probably due to some data at site outside space tokens
• LHCb-Tape: the storage dump for LHCb_RAW is not provided, so the checks doesn't make sense.

Update at July 2011

Summary of space usage:

---->   SRM Space tokens:
LHCb_USER :
Space usage (TB): SRM: 26.1 and LFC: 26.1 => diff= 0.0 
LHCb-Disk :
Space usage (TB): SRM: 135.6 and LFC: 331.6 => diff= 196.1 
Remaining used space in 2010 space tokens:
Info not available for ST LHCb_RAW
Info not available for ST LHCb_RDST
Info not available for ST LHCb_M-DST
ST: LHCb_DST SRM used space: 36.7 
Info not available for ST LHCb_MC_M-DST
ST: LHCb_MC_DST SRM used space: 73.2 
Info not available for ST LHCb_HISTOS
ST: LHCb_FAILOVER SRM used space: 0.1 
Total remaning space: 109.9 

Comments:

• LHCb_USER: perfect agreement between LFC and SRM OK
• LHCb-Disk: much more used space in LFC. Even taking into account the space still on the 'old' space tokens of 2010 (136 TB), still 60 TB difference. To be kept in mind the issue with data outside any space token with this site TO BE INVESTIGATED

IN2P3

Summary of the situation:

Related GGUS tickets:

• request for storage dumps
• Some questions related to storage dumps provided for LHCB data (Oct 2011) where they explain how data outside space token can be re-allocated to the right space token. Very interesting possibility!
• Reallocation of disk space to LHCb-Disk space token: here also like at Gridka, there is a difference between the T1D* STS and the T0D1, and the procedure to allocate the space is different, however they didn't mention the difficulties mentioned by Gridka.

Update at February 2012

Migration of data from the old space tokens to the new ones completed. The consequence of this migration is that:

• what you see displayed by SLS sensors for LHCb-Disk (https://sls.cern.ch/sls/history.php?id=IN2P3_LHCb-Disk&more=ALL&period=24h) is the total amount of data stored on disk, so it makes sense to compare it with the pledge to compute how much free space is left (before this was not the case, as some data was still hosted in old space tokens, and thus not accounted for LHCb disk)
• the used space reported by SLS sensors for LHCb can be directly compared with the space usage computed on the basis of the LFC and (in principle) it should match.

In order to do the comparison SRM vs LFC I have developed a script which summarizes the space usage, from different sources ( https://savannah.cern.ch/task/?26552 )

$ dirac-dms-spaceTokens-usage.py -S LCG.IN2P3.fr
Storage usage summary for site LCG.IN2P3.fr - Wed Feb 15 15:59:06 2012
Space token LHCb-Tape
From LFC: Files: 233031, Size: 663.13 TB
From storage dumps: Files: 181425, Size: 573.39 TB - last update 2012-02-13 13:27:24
Space token LHCb-Disk
From LFC: Files: 216764, Size: 448.42 TB
From SRM: Total Assigned Space: 793.85, Used Space: 449.00, Free Space: 344.85 TB
From storage dumps: Files: 217593, Size: 448.32 TB - last update 2012-02-13 13:27:37
Space token LHCb_USER
From LFC: Files:  458470, Size: 44.65 TB
From SRM: Total Assigned Space: 197.91, Used Space: 46.16, Free Space: 151.75 TB
From storage dumps: Files: 467925, Size: 45.28 TB - last update 2012-02-13 13:27:12

quite a good agreement for LHCb-Disk and LHCb-User (we should keep in mind that the information from SRM is up to date, whereas the information from LFC can have a latency of 12h, and the info from storage dumps a delay of up to one week). For LHCb-Tape: the space usage reported by the storage dump is less than what reported by LFC, but it's not due to missing data, it's rather data that are on chimera fs but are not associated to any space token, so they are not reported into the storage dumps. This is a known issue.

Update at July 2011

LHCb_USER :
Space usage (TB): SRM: 29.5 and LFC: 29.4 => diff= -0.1 
LHCb-Disk :
Space usage (TB): SRM: 123.7 and LFC: 411.8 => diff= 288.1 
Remaining used space in 2010 space tokens:
Info not available for ST LHCb_RAW
Info not available for ST LHCb_RDST
ST: LHCb_M-DST SRM used space: 27.8 
ST: LHCb_DST SRM used space: 18.9 
ST: LHCb_MC_M-DST SRM used space: 77.7 
ST: LHCb_MC_DST SRM used space: 96.0 
Info not available for ST LHCb_HISTOS
ST: LHCb_FAILOVER SRM used space: 0.1 
Total remaning space: 220.5 

Comments:

• LHCb_USER : OK
• LHCb-Disk: much more space reported by LFC, probably because old space tokens still have data, but the remaining space in the old space tokens (220 TB) is not enough to explain the difference.. TO BE INVESTIGATED

PIC

Related GGUS tickets:

• request to complete the migration to new space tokens pending
• request for storage dumps pending

Update at March 2012

Migration to new space tokens is still ongoing, see ticket above 79305. Neither storage dumps are provided.

Space token LHCb-Tape
From LFC: Files: 89118, Size: 277.21 TB
From storage dumps: Information not available

Space token LHCb-Disk
From LFC: Files: 151090, Size: 303.44 TB
From SRM: Total Assigned Space: 400.00, Used Space: 294.61, Free Space: 105.39 TB
From storage dumps: Information not available

Space token LHCb_USER OK
From LFC: Files: 389523, Size: 39.24 TB
From SRM: Total Assigned Space: 100.00, Used Space: 40.52, Free Space: 59.48 TB
From storage dumps: Information not available

Update at July 2011

---->   SRM Space tokens:
LHCb_USER :
Space usage (TB): SRM: 21.9 and LFC: 21.7 => diff= -0.2 
LHCb-Disk :
Space usage (TB): SRM: 155.2 and LFC: 220.2 => diff= 65.0 
Remaining used space in 2010 space tokens:

Comments:

• LHCb_USER: OK
• LHCb-Disk: considerable difference from what reported in the LFC, as a sum of the Dirac SEs ('PIC-DST', 'PIC_M-DST', 'PIC-FAILOVER', 'PIC_MC_M-DST', 'PIC_MC-DST') and what reported by SRM. But the old space tokens are not exposed any more through SRM (they have been released), so it is impossible to check.

RAL

Update at March 2012

WARNING! apply a 0.94 factor to total space returned by SRM for RAL

Space token LHCb-Tape OK
From LFC: Files: 216711, Size: 639.35 TB
From storage dumps: Files: 216284, Size: 637.73 TB - last update 2012-03-19

Space token LHCb-Disk good agreement LFC vs SRM - value from storage dump slightly lower
From LFC: Files: 249888, Size: 494.41 TB
From SRM: Total Assigned Space: 1077.28, Used Space: 497.77, Free Space: 579.52 TB
From storage dumps: Files: 240882, Size: 487.01 TB - last update 2012-03-19

Space token LHCb_USER OK
From LFC: Files: 594744, Size: 59.41 TB
From SRM: Total Assigned Space: 93.99, Used Space: 60.54, Free Space: 33.45 TB
From storage dumps: Files: 606649, Size: 59.34 TB - last update 2012-03-19

Update at Feb 2012

WARNING! apply a 0.94 factor to total space returned by SRM for RAL

Space token LHCb-Tape OK
From LFC: Files: 209290, Size: 613.31 TB
From storage dumps: Files: 209564, Size: 614.48 TB - last update 2012-02-21 15:55:33

Space token LHCb-Disk LITTLE EXCESS IN USED SPACE FOR SRM. TO BE UNDERSTOOD
From LFC: Files: 210592, Size: 442.09 TB
From SRM: Total Assigned Space: 1041.61, Used Space: 449.16, Free Space: 592.46 TB
From storage dumps: Files: 217179, Size: 442.26 TB - last update 2012-02-21 15:55:21

Space token LHCb_USER OK
From LFC: Files: 557746, Size: 58.19 TB
From SRM: Total Assigned Space: 93.99, Used Space: 58.95, Free Space: 35.04 TB
From storage dumps: Files: 571054, Size: 58.33 TB - last update 2012-02-21 15:56:10

Update at July 2011

LHCb_USER :
Space usage (TB): SRM: 33.6 and LFC: 32.5 => diff= -1.1 
LHCb-Disk :
Space usage (TB): SRM: 388.6 and LFC: 376.7 => diff= -11.9 
Remaining used space in 2010 space tokens:
Info not available for ST LHCb_RAW, LHCb_RDST, LHCb_M-DST, LHCb_DST, LHCb_MC_M-DST, LHCb_MC_DST, LHCb_HISTOS
ST: LHCb_FAILOVER SRM used space: 388.6 
Total remaning space: 388.6 

Comments:

• LHCb_USER: OK
• LHCb-Disk: the SRM returns exactly the same data for LHCb-Disk and the old space token LHCb_FAILOVER, as they are synonyms:

>>> lcg_util.lcg_stmd( 'LHCb-Disk', ep , True, 0)
(0, [{'guaranteedsize': 606705404235776L, 'totalsize': 606705404235776L, 'lifetimeassigned': 0, 'spacetoken': 'lhcb:LHCb-Disk', 'lifetimeleft': -1, 'unusedsize': 181135608743680L, 'retentionpolicy': 'output', 'owner': None, 'accesslatency': 'unknown'}], '')
>>> lcg_util.lcg_stmd( 'LHCb_FAILOVER', ep , True, 0)
(0, [{'guaranteedsize': 606705404235776L, 'totalsize': 606705404235776L, 'lifetimeassigned': 0, 'spacetoken': 'lhcb:LHCb_FAILOVER', 'lifetimeleft': -1, 'unusedsize': 181135064098560L, 'retentionpolicy': 'output', 'owner': None, 'accesslatency': 'unknown'}], '')

Raja confirmed that this has been done upon request of LHCb. See here the ELOG with the plans for the migration Space token deployment fine. About consistency: a 3% excess of data in SRM wrt LFC (not really a problem, but to keep under control)

SARA

Summary of the situation: differently from Gridka and In2p3, in this site there is no need to reallocate space after data removal from old inactive space tokens! See ticket below

Related GGUS tickets:

• request for storage dumps
• Reallocation of disk space to LHCb-Disk space token - SARA: where they explain that after data removal from inactive space tokens (like LHCb-M-DST and LHCb-MC-M-DST) there is no need for space re-allocation, the space is automatically available..

though not clear where. ticket still open...

Update at March 2012

SARA completed the migration to new space tokens.

Storage usage summary for site LCG.SARA.nl - Wed Mar 21 11:55:46 2012

Space token LHCb-Tape probably some files outside space token

From LFC: Files: 237844, Size: 656.77 TB
From storage dumps: Files: 236179, Size: 650.1 TB - last update 2012-03-18

Space token LHCb-Disk good agreement LFC vs SRM - probably storage dump obsolete

From LFC: Files: 265043, Size: 481.40 TB
From SRM: Total Assigned Space: 650.00, Used Space: 480.03, Free Space: 169.97 TB
From storage dumps: Files: 273590, Size: 466.50 TB - last update 2012-03-18

Space token LHCb_USER OK

From LFC: Files: 427717, Size: 43.84 TB From SRM: Total Assigned Space: 60.00, Used Space: 43.43, Free Space: 16.57 TB
From storage dumps: Files: - Size: 43.5 TB - last update 2012-02-16 08:37:26 (two users space tokens!)

Update at 22nd Feb 2012

Space token LHCb-Disk
From LFC: Files: 230503, Size: 426.68 TB slightly less data in storage, but for sure it is due to the fact that some data are outside space token
From SRM: Total Assigned Space: 372.12, Used Space: 317.92, Free Space: 54.20 TB
From storage dumps: 317.08 | LHCb-Disk.47971090.ACTIVE + 19.2 | LHCb_DST.4012189.ACTIVE + 0.05 LHCb_FAILOVER.4077241.ACTIVE + 12.5 LHCb_M-DST.4013116.INACTIVE + 49.09 LHCb_MC_DST.4012051.ACTIVE + 25.7 LHCb_MC_M-DST.4013119.INACTIVE = total 423.7 TB

Space token LHCb_USER
From LFC: Files: 400641, Size: 43.49 TB
From SRM: Total Assigned Space: 50.00, Used Space: 42.80, Free Space: 7.20 TB
From storage dumps: 40 TB (last update 16th Feb)

Update at 10 Oct 2011

--------- Disk tokens ---------------------------------------------------
LFC: 419.5 summing the Dirac SEs: ['SARA-DST', 'SARA_M-DST', 'SARA_MC_M-DST', 'SARA_MC-DST', 'SARA-FAILOVER'] 
Total disk usage from Storage dumps: 412.1 TB, from space tokens ['LHCb-Disk.47971090.ACTIVE', 'LHCb_DST.4012189.ACTIVE', 'LHCb_FAILOVER.4077241.ACTIVE', 'LHCb_M-DST.31483.INACTIVE', 'LHCb_M-DST.4013116.INACTIVE', 'LHCb_MC_DST.4012051.ACTIVE', 'LHCb_MC_M-DST.4013119.INACTIVE'] 
--------- Tape tokens ---------------------------------------------------
LFC: 508.1 summing the Dirac SEs: ['SARA-RAW', 'SARA-RDST', 'SARA-ARCHIVE'] 
Total tape usage: 501.7 TB, from space tokens ['LHCb-Tape.47971042.ACTIVE', 'LHCb_RAW.4012396.ACTIVE', 'LHCb_RDST.31470.INACTIVE', 'LHCb_RDST.4012397.ACTIVE'] 

LHCb_USER: OK!

LHCb-Disk and LHCb-Tape: More data in LFC than in Storage. Possible reasons: 1-the check has been run some hours after the storage dump creation 2-there is the data outside space tokens. e.g. the file:

 > lcg-ls -l srm://srm.grid.sara.nl/pnfs/grid.sara.nl/data/lhcb/data/2009/RAW/FULL/FEST/FEST/59784/059784_0000000023.raw
-rw-r--r--   1     2     2 1887442620             NEARLINE /pnfs/grid.sara.nl/data/lhcb/data/2009/RAW/FULL/FEST/FEST/59784/059784_0000000023.raw
        * Checksum: 53f717b9 (adler32)

and it is not listed in the list of files outside space tokens provided with the storage dumps (files-outside-space-tokens.txt).

Update at July 2011

LHCb_USER :
Space usage (TB): SRM: 19.3 and LFC: 18.5 => diff= -0.8 
LHCb-Disk :
Space usage (TB): SRM: 135.4 and LFC: 376.5 => diff= 241.0 
Remaining used space in 2010 space tokens:
ST: LHCb_RAW SRM used space: 0.0 
ST: LHCb_RDST SRM used space: 0.0 
Info not available for ST LHCb_M-DST
ST: LHCb_DST SRM used space: 47.8 
Info not available for ST LHCb_MC_M-DST
ST: LHCb_MC_DST SRM used space: 113.7 
Info not available for ST LHCb_HISTOS
ST: LHCb_FAILOVER SRM used space: 0.1 
Total remaning space: 161.6 

Comments:

• LHCb_USER: OK
• LHCb-Disk: same than IN2P3

More details here

Summary

Situation at July 2011

SITE	ST LHCb_USER	ST space token LHCb-Disk	comments
CERN	need further investigation	need further investigation
CNAF	Ok	Ok	Deployment fine, and good agreement SRM VS LFC
GRIDKA	Ok	need further investigation
IN2P3	Ok	need further investigation
PIC	Ok	not possible to check
RAL	Ok	OK	Deployment OK. 3% excess in SRM wrt LFC
SARA	Ok	need further investigation

-- ElisaLanciotti - 07-Oct-2011