A review of space token deployment and space usage at sites
(
ALL DIFFERENCES ARE ALWAYS LFC-SRM)
CERN
Related GGUS tickets
Some LHCb files outside the space tokens
Update at July 2011
LHCb_USER :
Space usage (TB): SRM: 112.4 and LFC: 98.5 => diff= -13.8
LHCb-Disk :
Space usage (TB): SRM: 912.1 and LFC: 840.5 => diff= -71.6
Comments:
- For LHCb_USER: big discrepancy. Probably due to the well known issue, i.e. the SRM interface for Castor reports the space of only the disk servers currently on-line
- For LHCb-Disk big discrepancy, more space used according to SRM than LFC. For Castor at CERN these data are not really meaningful (see previous point).
CNAF
Summary of the situation:
everything fine: the only existing space tokens are the new ones (LHCB-Disk, LHCb-USER, LHCb-Tape) and all the data have been migrated there. OK!
Update at March 2012
Space token LHCb-Tape
to be checked
From LFC: Files: 203203, Size: 625.15 TB
From storage dumps: Files: 199644, Size: 612.71 TB - last update 2012-03-08 07:13:44
Space token LHCb-Disk
to be checked
From LFC: Files: 222865, Size: 466.82 TB
From SRM: Total Assigned Space: 540.00, Used Space: 475.45, Free Space: 64.55 TB
From storage dumps: Files: 210974, Size: 448.56 TB - last update 2012-03-08 07:13:55
Space token LHCb_USER
OK
From LFC: Files: 512080, Size: 59.27 TB
From SRM: Total Assigned Space: 61.02, Used Space: 59.25, Free Space: 1.77 TB
From storage dumps: Files: 499275, Size: 60.59 TB - last update 2012-03-08 07:13:32
Update at October 2011
--------- USER space ---------------------------------------------------
LFC: 34.3 SRM: 35.0 => diff -0.7
------------- LCG.CNAF.it - LHCb-Disk :
LCG.CNAF.it-LHCb-Disk Space usage (TB): SRM: 399.9 and LFC: 400.2 => diff= 0.3
--------- Disk tokens ---------------------------------------------------
LFC: 389.8 summing the Dirac SEs: ['CNAF-DST', 'CNAF_M-DST', 'CNAF-FAILOVER', 'CNAF_MC_M-DST', 'CNAF_MC-DST']
Total disk usage from Storage dumps: 389.0 TB, from space tokens ['LHCb-Disk']
--------- Tape tokens ---------------------------------------------------
LFC: 475.5 summing the Dirac SEs: ['CNAF-RAW', 'CNAF-RDST', 'CNAF-ARCHIVE']
Total tape usage: 476.2 TB, from space tokens ['LHCb-Tape']
Everything OK.
Update at July 2011
LHCb_USER :
Space usage (TB): SRM: 29.6 and LFC: 28.8 => diff= -0.8
LHCb-Disk :
Space usage (TB): SRM: 352.3 and LFC: 352.2 => diff= -0.1
Remaining used space in 2010 space tokens:
ST: LHCb_RAW SRM used space: 155.0
ST: LHCb_RDST SRM used space: 155.0
ST: LHCb_M-DST SRM used space: 352.3
ST: LHCb_DST SRM used space: 352.3
ST: LHCb_MC_M-DST SRM used space: 352.3
ST: LHCb_MC_DST SRM used space: 352.3
Info not available for ST LHCb_HISTOS
ST: LHCb_FAILOVER SRM used space: 352.3
Total remaning space: 2071.8
comments:
- almost no discrepancy for LHCb-Disk and LHCb_USER. And there is always a bit more data in the LFC. OK.
GRIDKA
Summary of the situation:
there is a clear separation between the pools assigned to the T1D* space tokens, and the pools assigned to the T0D1 spaces. So, if you have to re-assign some free space from T1D* pools to the T0D1 pools, it can be a not so easy operation. In our particular case, after removal campaigns, some space is freed up in the old T0D1 tokens ( LHCb-DST and LHCb-MC-DST) and this can be easily re-assigned to LHCb-Disk. And on the other hand, some space is freed in the old LHCb-M-DST and LHCb-M-MC-DST (which were T1D1 spaces), and this free space cannot be easily assigned to LHCb-Disk.
Related GGUS tickets:
Update at March 2012
Space token LHCb-Tape
probably some data outside space tokens
From LFC: Files: 208308, Size: 608.61 TB
From storage dumps: Files: 175702, Size: 562.39 TB - last update 2012-03-16 15:51:54
Space token LHCb-Disk
ok
From LFC: Files: 220852, Size: 406.15 TB
From SRM: Total Assigned Space: 495.00, Used Space: 408.87, Free Space: 86.13 TB
From storage dumps: Files: 218084, Size: 404.14 TB - last update 2012-03-16 15:52:09
Space token LHCb_USER
ok
From LFC: Files: 451771, Size: 45.28 TB
From SRM: Total Assigned Space: 60.00, Used Space: 45.12, Free Space: 14.88 TB
From storage dumps: Files: 451909, Size: 45.11 TB - last update 2012-03-16 15:52:40
Update at Oct 2011
--------- USER space ---------------------------------------------------
LFC: 31.7 SRM: 31.2 => diff 0.5
--------- Disk tokens ---------------------------------------------------
LFC: 382.5 summing the Dirac SEs: ['GRIDKA-DST', 'GRIDKA_M-DST', 'GRIDKA-FAILOVER', 'GRIDKA_MC_M-DST', 'GRIDKA_MC-DST']
Total disk usage from Storage dumps: 371.3 TB, from space tokens ['LHCb-Disk', 'LHCb_DST', 'LHCb_FAILOVER', 'LHCb_M-DST', 'LHCb_MC_DST', 'LHCb_MC_M-DST']
--------- Tape tokens ---------------------------------------------------
LFC: 510.5 summing the Dirac SEs: ['GRIDKA-RAW', 'GRIDKA-RDST', 'GRIDKA-ARCHIVE']
Total tape usage: 417.2 TB, from space tokens ['LHCb-Tape', 'LHCb_RDST']
- LHCb_USER ok.
- LHCb-Disk: more data in the LFC than on storage: probably due to some data at site outside space tokens
- LHCb-Tape: the storage dump for LHCb_RAW is not provided, so the checks doesn't make sense.
Update at July 2011
Summary of space usage:
----> SRM Space tokens:
LHCb_USER :
Space usage (TB): SRM: 26.1 and LFC: 26.1 => diff= 0.0
LHCb-Disk :
Space usage (TB): SRM: 135.6 and LFC: 331.6 => diff= 196.1
Remaining used space in 2010 space tokens:
Info not available for ST LHCb_RAW
Info not available for ST LHCb_RDST
Info not available for ST LHCb_M-DST
ST: LHCb_DST SRM used space: 36.7
Info not available for ST LHCb_MC_M-DST
ST: LHCb_MC_DST SRM used space: 73.2
Info not available for ST LHCb_HISTOS
ST: LHCb_FAILOVER SRM used space: 0.1
Total remaning space: 109.9
Comments:
- LHCb_USER: perfect agreement between LFC and SRM OK
- LHCb-Disk: much more used space in LFC. Even taking into account the space still on the 'old' space tokens of 2010 (136 TB), still 60 TB difference. To be kept in mind the issue with data outside any space token with this site TO BE INVESTIGATED
IN2P3
Summary of the situation:
Related GGUS tickets:
Update at February 2012
Migration of data from the old space tokens to the new ones completed.
The consequence of this migration is that:
- what you see displayed by SLS sensors for LHCb-Disk (https://sls.cern.ch/sls/history.php?id=IN2P3_LHCb-Disk&more=ALL&period=24h
) is the total amount of data stored on disk, so it makes sense to compare it with the pledge to compute how much free space is left (before this was not the case, as some data was still hosted in old space tokens, and thus not accounted for LHCb disk)
- the used space reported by SLS sensors for LHCb can be directly compared with the space usage computed on the basis of the LFC and (in principle) it should match.
In order to do the comparison SRM vs LFC I have developed a script which summarizes the space usage, from different sources (
https://savannah.cern.ch/task/?26552
)
$ dirac-dms-spaceTokens-usage.py -S LCG.IN2P3.fr
Storage usage summary for site LCG.IN2P3.fr - Wed Feb 15 15:59:06 2012
Space token LHCb-Tape
From LFC: Files: 233031, Size: 663.13 TB
From storage dumps: Files: 181425, Size: 573.39 TB - last update 2012-02-13 13:27:24
Space token LHCb-Disk
From LFC: Files: 216764, Size: 448.42 TB
From SRM: Total Assigned Space: 793.85, Used Space: 449.00, Free Space: 344.85 TB
From storage dumps: Files: 217593, Size: 448.32 TB - last update 2012-02-13 13:27:37
Space token LHCb_USER
From LFC: Files: 458470, Size: 44.65 TB
From SRM: Total Assigned Space: 197.91, Used Space: 46.16, Free Space: 151.75 TB
From storage dumps: Files: 467925, Size: 45.28 TB - last update 2012-02-13 13:27:12
quite a good agreement for LHCb-Disk and LHCb-User (we should keep in mind that the information from SRM is up to date, whereas the information from LFC can have a latency of 12h, and the info from storage dumps a delay of up to one week).
For LHCb-Tape: the space usage reported by the storage dump is less than what reported by LFC, but it's not due to missing data, it's rather data that are on chimera fs but are not associated to any space token, so they are not reported into the storage dumps. This is a known issue.
Update at July 2011
LHCb_USER :
Space usage (TB): SRM: 29.5 and LFC: 29.4 => diff= -0.1
LHCb-Disk :
Space usage (TB): SRM: 123.7 and LFC: 411.8 => diff= 288.1
Remaining used space in 2010 space tokens:
Info not available for ST LHCb_RAW
Info not available for ST LHCb_RDST
ST: LHCb_M-DST SRM used space: 27.8
ST: LHCb_DST SRM used space: 18.9
ST: LHCb_MC_M-DST SRM used space: 77.7
ST: LHCb_MC_DST SRM used space: 96.0
Info not available for ST LHCb_HISTOS
ST: LHCb_FAILOVER SRM used space: 0.1
Total remaning space: 220.5
Comments:
- LHCb_USER : OK
- LHCb-Disk: much more space reported by LFC, probably because old space tokens still have data, but the remaining space in the old space tokens (220 TB) is not enough to explain the difference.. TO BE INVESTIGATED
PIC
Related GGUS tickets:
Update at March 2012
Migration to new space tokens is still ongoing, see ticket above 79305.
Neither storage dumps are provided.
Space token LHCb-Tape
From LFC: Files: 89118, Size: 277.21 TB
From storage dumps: Information not available
Space token LHCb-Disk
From LFC: Files: 151090, Size: 303.44 TB
From SRM: Total Assigned Space: 400.00, Used Space: 294.61, Free Space: 105.39 TB
From storage dumps: Information not available
Space token LHCb_USER
OK
From LFC: Files: 389523, Size: 39.24 TB
From SRM: Total Assigned Space: 100.00, Used Space: 40.52, Free Space: 59.48 TB
From storage dumps: Information not available
Update at July 2011
----> SRM Space tokens:
LHCb_USER :
Space usage (TB): SRM: 21.9 and LFC: 21.7 => diff= -0.2
LHCb-Disk :
Space usage (TB): SRM: 155.2 and LFC: 220.2 => diff= 65.0
Remaining used space in 2010 space tokens:
Comments:
- LHCb_USER: OK
- LHCb-Disk: considerable difference from what reported in the LFC, as a sum of the Dirac SEs ('PIC-DST', 'PIC_M-DST', 'PIC-FAILOVER', 'PIC_MC_M-DST', 'PIC_MC-DST') and what reported by SRM. But the old space tokens are not exposed any more through SRM (they have been released), so it is impossible to check.
RAL
Update at March 2012
WARNING! apply a 0.94 factor to total space returned by SRM for RAL
Space token LHCb-Tape
OK
From LFC: Files: 216711, Size: 639.35 TB
From storage dumps: Files: 216284, Size: 637.73 TB - last update 2012-03-19
Space token LHCb-Disk
good agreement LFC vs SRM -
value from storage dump slightly lower
From LFC: Files: 249888, Size: 494.41 TB
From SRM: Total Assigned Space: 1077.28, Used Space: 497.77, Free Space: 579.52 TB
From storage dumps: Files: 240882, Size: 487.01 TB - last update 2012-03-19
Space token LHCb_USER
OK
From LFC: Files: 594744, Size: 59.41 TB
From SRM: Total Assigned Space: 93.99, Used Space: 60.54, Free Space: 33.45 TB
From storage dumps: Files: 606649, Size: 59.34 TB - last update 2012-03-19
Update at Feb 2012
WARNING! apply a 0.94 factor to total space returned by SRM for RAL
Space token LHCb-Tape
OK
From LFC: Files: 209290, Size: 613.31 TB
From storage dumps: Files: 209564, Size: 614.48 TB - last update 2012-02-21 15:55:33
Space token LHCb-Disk
LITTLE EXCESS IN USED SPACE FOR SRM. TO BE UNDERSTOOD
From LFC: Files: 210592, Size: 442.09 TB
From SRM: Total Assigned Space: 1041.61, Used Space: 449.16, Free Space: 592.46 TB
From storage dumps: Files: 217179, Size: 442.26 TB - last update 2012-02-21 15:55:21
Space token LHCb_USER
OK
From LFC: Files: 557746, Size: 58.19 TB
From SRM: Total Assigned Space: 93.99, Used Space: 58.95, Free Space: 35.04 TB
From storage dumps: Files: 571054, Size: 58.33 TB - last update 2012-02-21 15:56:10
Update at July 2011
LHCb_USER :
Space usage (TB): SRM: 33.6 and LFC: 32.5 => diff= -1.1
LHCb-Disk :
Space usage (TB): SRM: 388.6 and LFC: 376.7 => diff= -11.9
Remaining used space in 2010 space tokens:
Info not available for ST LHCb_RAW, LHCb_RDST, LHCb_M-DST, LHCb_DST, LHCb_MC_M-DST, LHCb_MC_DST, LHCb_HISTOS
ST: LHCb_FAILOVER SRM used space: 388.6
Total remaning space: 388.6
Comments:
- LHCb_USER: OK
- LHCb-Disk: the SRM returns exactly the same data for LHCb-Disk and the old space token LHCb_FAILOVER, as they are synonyms:
>>> lcg_util.lcg_stmd( 'LHCb-Disk', ep , True, 0)
(0, [{'guaranteedsize': 606705404235776L, 'totalsize': 606705404235776L, 'lifetimeassigned': 0, 'spacetoken': 'lhcb:LHCb-Disk', 'lifetimeleft': -1, 'unusedsize': 181135608743680L, 'retentionpolicy': 'output', 'owner': None, 'accesslatency': 'unknown'}], '')
>>> lcg_util.lcg_stmd( 'LHCb_FAILOVER', ep , True, 0)
(0, [{'guaranteedsize': 606705404235776L, 'totalsize': 606705404235776L, 'lifetimeassigned': 0, 'spacetoken': 'lhcb:LHCb_FAILOVER', 'lifetimeleft': -1, 'unusedsize': 181135064098560L, 'retentionpolicy': 'output', 'owner': None, 'accesslatency': 'unknown'}], '')
Raja confirmed that this has been done upon request of LHCb. See here the
ELOG
with the plans for the migration
Space token deployment fine. About consistency: a 3% excess of data in SRM wrt LFC (not really a problem, but to keep under control)
SARA
Summary of the situation:
differently from Gridka and In2p3, in this site there is no need to reallocate space after data removal from old inactive space tokens! See ticket below
Related GGUS tickets:
though not clear where. ticket still open...
Update at March 2012
SARA completed the migration to new space tokens.
Storage usage summary for site LCG.SARA.nl - Wed Mar 21 11:55:46 2012
Space token LHCb-Tape
probably some files outside space token
From LFC: Files: 237844, Size: 656.77 TB
From storage dumps: Files: 236179, Size: 650.1 TB - last update 2012-03-18
Space token LHCb-Disk
good agreement LFC vs SRM -
probably storage dump obsolete
From LFC: Files: 265043, Size: 481.40 TB
From SRM: Total Assigned Space: 650.00, Used Space: 480.03, Free Space: 169.97 TB
From storage dumps: Files: 273590, Size: 466.50 TB - last update 2012-03-18
Space token LHCb_USER
OK
From LFC: Files: 427717, Size: 43.84 TB
From SRM: Total Assigned Space: 60.00, Used Space: 43.43, Free Space: 16.57 TB
From storage dumps: Files: - Size: 43.5 TB - last update 2012-02-16 08:37:26 (two users space tokens!)
Update at 22nd Feb 2012
Space token LHCb-Disk
From LFC: Files: 230503, Size: 426.68 TB
slightly less data in storage, but for sure it is due to the fact that some data are outside space token
From SRM: Total Assigned Space: 372.12, Used Space: 317.92, Free Space: 54.20 TB
From storage dumps: 317.08 | LHCb-Disk.47971090.ACTIVE + 19.2 | LHCb_DST.4012189.ACTIVE + 0.05 LHCb_FAILOVER.4077241.ACTIVE + 12.5 LHCb_M-DST.4013116.INACTIVE + 49.09 LHCb_MC_DST.4012051.ACTIVE + 25.7 LHCb_MC_M-DST.4013119.INACTIVE = total 423.7 TB
Space token LHCb_USER
From LFC: Files: 400641, Size: 43.49 TB
From SRM: Total Assigned Space: 50.00, Used Space: 42.80, Free Space: 7.20 TB
From storage dumps: 40 TB (last update 16th Feb)
Update at 10 Oct 2011
--------- Disk tokens ---------------------------------------------------
LFC: 419.5 summing the Dirac SEs: ['SARA-DST', 'SARA_M-DST', 'SARA_MC_M-DST', 'SARA_MC-DST', 'SARA-FAILOVER']
Total disk usage from Storage dumps: 412.1 TB, from space tokens ['LHCb-Disk.47971090.ACTIVE', 'LHCb_DST.4012189.ACTIVE', 'LHCb_FAILOVER.4077241.ACTIVE', 'LHCb_M-DST.31483.INACTIVE', 'LHCb_M-DST.4013116.INACTIVE', 'LHCb_MC_DST.4012051.ACTIVE', 'LHCb_MC_M-DST.4013119.INACTIVE']
--------- Tape tokens ---------------------------------------------------
LFC: 508.1 summing the Dirac SEs: ['SARA-RAW', 'SARA-RDST', 'SARA-ARCHIVE']
Total tape usage: 501.7 TB, from space tokens ['LHCb-Tape.47971042.ACTIVE', 'LHCb_RAW.4012396.ACTIVE', 'LHCb_RDST.31470.INACTIVE', 'LHCb_RDST.4012397.ACTIVE']
LHCb_USER: OK!
LHCb-Disk and LHCb-Tape: More data in LFC than in Storage.
Possible reasons:
1-the check has been run some hours after the storage dump creation
2-there is the data outside space tokens.
e.g. the file:
> lcg-ls -l srm://srm.grid.sara.nl/pnfs/grid.sara.nl/data/lhcb/data/2009/RAW/FULL/FEST/FEST/59784/059784_0000000023.raw
-rw-r--r-- 1 2 2 1887442620 NEARLINE /pnfs/grid.sara.nl/data/lhcb/data/2009/RAW/FULL/FEST/FEST/59784/059784_0000000023.raw
* Checksum: 53f717b9 (adler32)
and it is not listed in the list of files outside space tokens provided with the storage dumps (files-outside-space-tokens.txt).
Update at July 2011
LHCb_USER :
Space usage (TB): SRM: 19.3 and LFC: 18.5 => diff= -0.8
LHCb-Disk :
Space usage (TB): SRM: 135.4 and LFC: 376.5 => diff= 241.0
Remaining used space in 2010 space tokens:
ST: LHCb_RAW SRM used space: 0.0
ST: LHCb_RDST SRM used space: 0.0
Info not available for ST LHCb_M-DST
ST: LHCb_DST SRM used space: 47.8
Info not available for ST LHCb_MC_M-DST
ST: LHCb_MC_DST SRM used space: 113.7
Info not available for ST LHCb_HISTOS
ST: LHCb_FAILOVER SRM used space: 0.1
Total remaning space: 161.6
Comments:
- LHCb_USER: OK
- LHCb-Disk: same than IN2P3
More details
here
Summary
Situation at July 2011
SITE |
ST LHCb_USER |
ST space token LHCb-Disk |
comments |
CERN |
need further investigation |
need further investigation |
|
CNAF |
Ok |
Ok |
Deployment fine, and good agreement SRM VS LFC |
GRIDKA |
Ok |
need further investigation |
|
IN2P3 |
Ok |
need further investigation |
|
PIC |
Ok |
not possible to check |
|
RAL |
Ok |
OK |
Deployment OK. 3% excess in SRM wrt LFC |
SARA |
Ok |
need further investigation |
|
--
ElisaLanciotti - 07-Oct-2011