Title: Consistency between Grid Storage Elements and File Catalogs for the LHCb experiment's data

In the distributed computing model of WLCG Grid Storage Elements (SE) are by construction completely decoupled from the File Catalogs (FC) where the experiment's files are registered. On the basis of the experience of managing large volumes of data in such environment, inconsistencies have often happened either causing a waste of disk space, in case the data were deleted from the FC, but still physically on the SE, or serious operational problems in the opposite case, when some data registered in the FC was not found on the SE. Therefore, the LHCbDirac data management system has been equipped with a new dedicated system to ensure the consistency of the data stored on the SEs with the information reported in the FCs implementing systematic checks. Objective of the checks is to spot any inconsistency above a certain threshold, that cannot only be due to the expected latency between data upload and registration, and in such case try and identify the problematic data. The system relies on information provided by the sites who should make available to the experiment a full dump of their SEs on weekly or monthly basis. In this talk we shall present the definition of a common format and procedure to produce the storage dumps that has been coordinated with the other LHC experiments in order to provide a solution as generic as possible that can suit all LHC experiments and will reduce the effort for the sites who are asked to provide such data. We will also present the LHCb specific implementation for checking the consistency between SEs and FC and discuss the results.

  • Track: "Distributed Processing and Analysis on Grids and Clouds"

  • Presentation type: Talk

  • Authors: Vincent Bernardoff, David Bouvet, Marco Cattaneo, Philippe Charpentier, Pete Clarke, Joel Closier, Ricardo Graciani, Elisa Lanciotti, Victor Mendez, Raja Nandakumar, Daniela Remenska, Stefan Roiser, Vladimir Romanovski, Roberto Santinelli, Federico Stagni, Andrei Tsaregorodtsev, Mario Ubeda Garcia, Alexey Zhelezov, Natalia Ratnikova, Nicolo' Magini, Cedric Serfon

-- ElisaLanciotti - 06-Oct-2011

This topic: LCG > WebHome > WLCGCommonComputingReadinessChallenges > WLCGOperationsWeb > ItEsHome > ConsistencyChecksSEsDumps > ConsistencyChecksSEsDumpsCHEP2012Abstract
Topic revision: r4 - 2011-10-27 - unknown
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback