DDMDarkDataAndLostFilesSANDBOX

Introduction

This page is a merge and update of DDMDarkDataCleaning and DDMLostFiles, it is still on DEVELOPMENT and SHOULD NOT BE USED for real checks yet.

Two possible types of inconsistencies between the Rucio database and the Storage elements are:

• Dark data: Files present on the Storage Element that don't exists on the Rucio Database.
• Lost files: Files registered on Rucio that don't exists on the Storage Element.

These inconsistencies can be detected comparing the occupied space reported by the SE via SRM protocol with the sum of the size of the files accounted by rucio for a given DDM endpoint. If the difference of the sizes is considerable it would be advisable to check if there are Dark Data or Lost files following the procedures described below.

Depending on the size of the storage “big” may be in the order of several TiB or > 1 − 2%.

The specific files involved can be found comparing dumps of the contents of the Storage Element with dumps of the Rucio Replicas, the objective of this wiki page is to describe the process to run this checks.

Determine the amount of dark data and lost files

As stated above the amount of dark data for a given DDM endpoint can be calculated as the difference between:

• occupied space reported by SE via SRM protocol
• sum of files for given DDM endpoint as accounted by Rucio

This information can be gathered using the "SRM View" button in the DDM Accounting Dashboard. Or in the command line with:

show example hide

$ lcg-stmd -e golias100.farm.particle.cz -s ATLASLOCALGROUPDISK | awk '/Total Size:/ {total=$NF} /Unused Size:/ {free=$NF} END {print "Occupied bytes: ", total-free}'
Occupied bytes:  288861963543746

$ rucio list-rse-usage PRAGUELCG2_LOCALGROUPDISK
Usage:
======
  used: 287203280562763
...

Note: For given DDM endpoint (e.g. PRAGUELCG2_LOCALGROUPDISK) the storage element hostname (option -e for lcg-stmd) and spacetoken (option -s for lcg-stmd) can be both obtained from AGIS.

Downloading/Creating dumps

Dates of the dumps

To use this procedure, in order to check dark and lost files, the SE dump should be at least one day old (as the checking algorithm needs to download a rucio replica dump newer than the SE dump and the rucio replica dumps are generated on a daily basis).

As for the rucio replica dump, the difference in days is not too important as far as one of the dumps is older and the other newer than the SE dump. But is recomended to take rucio replica dumps with at least 3 days of difference with the SE dump (this can be set with the --delta argument of the script rucio_dump) where the newest rucio replica dump is as new as possible to avoid false positives by transient files.

Storage Element Dumps

The procedure to generate the storage dumps depend on the implementation of the Storage element used. The output needed is a text file with the following properties:

• List of files (with relative path) from the directory tree belonging to particular DDM endpoint
  • No slash at the beginning of the file path (as this is a relative path)
  • Generally the mapping between DDM endpoint and storage directory is SITE_TOKEN -> /pathtoatlasdiskspace/atlasTOKEN/
    • For PRAGUELCG2_LOCALGROUPDISK it means everything under /dpm/farm.particle.cz/home/atlas/atlaslocalgroupdisk/rucio/ but listed without this prefix (see AGIS for details)
    • An exception is group space, where many different physics group tokens can be located under atlasgroupdisk/<groupname>/rucio
  • There should be no other directories under atlasTOKEN except SAM. If there are others they can be deleted.

In the past we used the syncat format for the dumps of Storage elements. Our experience showed that this format is difficult to use and comparing dumps of huge Storage elements takes too long. This is why we switched to simple text format.

DPM

The DPM dump can be obtained with script dpm_dump.py.

show example hide

$ python dpm_dump.py -t dump_20150730 -p /dpm/farm.particle.cz/home/atlas/atlaslocalgroupdisk/rucio -a 30
$ head dump_20150730
user/mzeman/33/7a/user.mzeman.012434._2063124537.log.tgz
user/mzeman/33/7a/user.mzeman.011680._1951810566.log.tgz
...

dCache

The dCache (chimera DB) dump script by Gerd Behrmann can be obtained here: chimera_find.sh

show example hide

$ sh chimera_find.sh -D "3 days ago" dump_20150727 /pnfs/ndgf.org/data/atlas/disk/atlasdatadisk/rucio /

Posix Storage Elements (NFS, PGFS, Lustre, etc...)

The Posix dump script based on the checks of Horst Severini can be obtained here: posix_dump.sh.

show example hide

$ sh posix_dump.sh --prefix /storage/data --spacetokens atlasdatadisk,atlaslocalgroupdisk > dump_20150727

Other implementations of SE

If the site runs another implementation of Storage element, the site administrator should find a specific way to dump the content in a file with the structure described above.

Rucio Replica Dumps

The rucio repica dumps are downloaded automatically by the rucio_dump script.

If you want to download the dumps manually for whatever reason check at the end of this wiki page.

Compare the dumps

The script to download and compare the dumps is called "rucio_dump" and can be downloaded rucio_dump (a bundled version of the script with no dependencies can be downloaded from rucio_dump (binary x86_64)).

Example:

show example hide

$ ./rucio_dump.py consistency --delta 3 FZK-LCG2_SCRATCHDISK dump_20150721
DARK,atlas/atlasscratchdisk/user10.KaiLeffhalm/test.2gb.01
DARK,SAM/testfile--GET-ATLASSCRATCHDISK.txt
DARK,SAM/testfile-DEL-ATLASSCRATCHDISK-1417766663-22ab37a0dcbd.txt
DARK,SAM/testfile-DEL-ATLASSCRATCHDISK-1417853202-5c6b32af5f7f-sgl.txt
...
LOST,data10_7TeV/00/41/data10_7TeV.00152221.physics_MinBias.recon.ESD.f572._lb0145._0003.1
LOST,data10_7TeV/00/42/data10_7TeV.00152221.physics_MinBias.recon.ESD.f572._lb0101._0001.1
LOST,data10_7TeV/00/4e/data10_7TeV.00152221.physics_MinBias.recon.ESD.f572._lb0063._0010.1
....

This script does the following:

• Downloads rucio replica dumps 3 days older and 3 days newer than the SE dump (taking the date of the SE dump from its name).
• SE dump lines are parsed removing the prefix of the path (if there is one) based on the prefix reported by AGIS. This is done in order to match the paths in the SE dumps with the ones in the Rucio Dumps.
• Sorts the rucio replica dumps and the SE dump by path.
• Traverse the 3 sorted dumps printing:
  • Files present in both rucio dumps but not in the SE dump (lost files)
  • Files present in the SE dump but not in any of the rucio dumps (dark files)

Important note: Sorted and downloaded files are saved under a "cache" subdirectory. If you are working with different SE dumps for the same spacetoken and with the same date, delete the "_sorted" file corresponding to the SE dump in the "cache" subdirectory in each run.

It is possible to run the check with custom dates or with manually downloaded rucio dumps, check the README.md file for further details.

The results of this script can be checked using Rucio to verify darks and GFAL to verify lost files.

cat output | grep '^LOST,' | ./verify-gfal.py

The output of verify-gfal.py has the format:

STATUS<CODE>,URL

Where

<CODE>

The next example output only the URLs of lost files verified with GFAL:

cat output | grep ^LOST, | ./verify-gfal.py | grep ^STATUS2, | cut -d, -f2

Dark Data

Clean the dark data

Identifying and cleaning dark data should be done only if there is a significant amount of dark data on the Storage element (approx. 5%). If the instructions below are not clear please contact DDM operations.

The cleaning itself is done in four steps:

1. Get the list of files on your storage element that belong to given DDM endpoint.
2. Download the list of files in Rucio catalog for the DDM endpoint.
3. Compare both lists.
4. Remove the files identified as dark.

Only apply this procedure against disk endpoints. There are as yet no cleaning procedures for tapes.

The steps 2 and 3 can be automated running the rucio_dump script.

Remove the files identified as dark

The files can be removed from the SE using "lcg-del -l" command or using a local command suitable for SE (e.g. rfrm for DPM).

show example hide

$ rfrm /dpm/farm.particle.cz/home/atlas/atlasproddisk/rucio/valid2/eb/6b/log.01418511._000730.job.log.tgz.10
$ lcg-del -l srm://golias100.farm.particle.cz/dpm/farm.particle.cz/home/atlas/atlasproddisk/rucio/valid2/fd/f5/TAG.01427540._004458.pool.root.1
$

The amount of dark data should be a small fraction of the total. If it is a significant fraction of the total then probably there is a mistake somewhere in the procedure. If the procedure and the data looks incorrect or you feel uncomfortable deleting the identified files please contact DDM ops for advice.

Detection of lost files (UNDER VALIDATION)

The principle is to compare the Storage Element and the Rucio dumps. Because of shift in time of the dumps, a final check is done on file by file bases (check that the file is not on the SE now and that the file is expected to be located at the RSE now).

Recovery of lost or corrupted files

• If a file is lost or confirmed to be corrupted at a site it should be declared bad.
• If the total file loss is large (>50k files) then please report it in the DDM ops Jira so the recovery can be coordinated with central ADC operations.
• The bad files can be declared using the rucio-admin CLI. Cloud admins can declare bad files in their own cloud. Anyone inside the egroup atlas-adc-cloud-xx is given cloud admin privileges for cloud xx.
  • To check if you have cloud admin privileges run rucio-admin account list-attributes --account youraccount and look for "cloud-xx: admin" in the output.

rucio-admin replicas declare-bad -h
usage: rucio-admin replicas declare-bad [-h] --reason REASON
                                        [--inputfile [INPUTFILE]]
                                        [listbadfiles [listbadfiles ...]]

positional arguments:
  listbadfiles          The list of bad files

optional arguments:
  -h, --help            show this help message and exit
  --reason REASON       Reason
  --inputfile [INPUTFILE]

You can either pass a list of bad files directly or give a file (using the --inputfile option) where the list of bad files is located. IMPORTANT the files must all have the srm protocol. No other protocol is currently supported for declaring bad files. You also have to specify a reason : it can be a URL pointing to a GGUS/JIRA ticket or whatever text describing the reason of the loss or corruption :

rucio-admin replicas declare-bad --reason "https://its.cern.ch/jira/browse/ATLDDMOPS-xyz" --inputfile mytestfile.txt

If some files are not known to Rucio, they will be printed out like this :

UKI-LT2-QMUL_DATADISK : PFN srm://se03.esc.qmul.ac.uk/atlas/atlasdatadisk/rucio/tests/38/50/blahblah cannot be declared.
UKI-NORTHGRID-LIV-HEP_DATADISK : PFN srm://hepgrid11.ph.liv.ac.uk/dpm/ph.liv.ac.uk/home/atlas/atlasdatadisk/rucio/tests/38/50/blahblah cannot be declared.

These files must be removed manually by site admins using whatever method is easiest.

For the other files, Rucio will try to recover them from another place if possible or delete the files from the site otherwise. The progress of the system dealing with bad files can be seen in 2 places :

• In the DDM dashboard by selecting the recovery activity : DDM dashboard with activity Recovery .
• In the Rucio UI, where you have access to the whole history : Rucio UI Recovery. You can use different filter to check the status of the recovery at a given site or on a given period.

Lost files on tape

Files on tape are treated differently from those on disk. On disk the same filename is always used since the path is deterministic, however the convention is different for tapes and transfer retries and recovery always use a different filename (a timestamp is appended) for safety reasons. In addition deletion of data on tape is only run manually and occasionally during large cleanup campaigns. As a result bad files on tape are never removed from the storage namespace whether they are recovered or not. Therefore after declaring bad files on tape, the site admins should themselves remove the bad files from the storage namespace.

Manual checks

If instead of the rucio_dump script you want to make the checks manually you can find the procedure below.

Description of the detection of dark data and lost files using dumps

The following is a visualization of how to verify the state of the files given we have a dump of the DDM endpoint and two Rucio replica dumps (one previous to the endpoint dump and another newer), this is an idealized model based on optimal conditions (ie: the dumps are complete, and the date of the dumps accurate):

The subsets marked with an asterisk are supposed to be "Deleted Files" or "New Files" respectively but it cannot be confirmed only with this 3 dumps. The rest of the subsets are more likely to be ok, but anyways is a good idea to verify the results by other means before taking action.

In conclusion:

• The files in the storage dump but not in the rucio dumps are very likely Dark Data.
• The files in both rucio dumps but not in the storage dump are very likely Lost Files.
• Adjusting D can change the results, larger numbers in D produce more reliable information about Lost Files but at the same time create more false positives for Dark Data (if files are created between T-D and T and then deleted between T and T+D).

Downloading Rucio dumps manually

The dump of files registered in Rucio for a given DDM endpoint (rucio replica dumps) can be obtained from https://rucio-ui.cern.ch/dumps, these are tab-separated files compressed with bzip2.

show example hide

$ wget -q --no-check-certificate 'https://rucio-hadoop.cern.ch/replica_dumps?rse=PRAGUELCG2_LOCALGROUPDISK&date=18-08-2014' -O PRAGUELCG2_LOCALGROUPDISK.ruciodump.18-08-2014.bz2
$ bunzip2 PRAGUELCG2_LOCALGROUPDISK.ruciodump.18-08-2014.bz2

Note: Is necessary to be careful parsing the dumps with shellscripts as fields may contain spaces (i.e: "bzcat PRAGUELCG2_LOCALGROUPDISK.ruciodump.18-08-2014.bz2 | awk '{print $7}'" may not print the 7th column). One way to overcome this is: "bzcat PRAGUELCG2_LOCALGROUPDISK.ruciodump.18-08-2014.bz2 | tr '\t' ',' | cut -d, -f7".

Hints:

• In the past the command "comm" was used to compare the dumps
• The command "sort" can sort big files in a efficient way
• Setting the environment variable LC_ALL=C makes many GNU commands related to text processing to work faster ("sort" in particular)

General algorithm

• Download two rucio replica dumps (one older and one newer than the SE dump)
• Parse the files and generate a file for each replica dump with only the paths of the files
• Parse the SE dump and remove the path prefixes, the result should have paths consistent with the rucio replica dumps (relative parts without the "rucio" component).
• In the following steps below, every mention of a "dump" refers to the parsed and sorted version generated here.

Dark files

• Generate a sorted file with the content of both rucio replica dumps.
• Compare the file generated above with the SE dump. Every path present only in the SE dump is dark data.

Lost files

• Generate a sorted file with the intersection of both rucio replica dumps.
• Compare the file generated above with the SE dump. Every path present only in the intersection is a lost file.
• The current status of the files can be obtained from the SE with "gfal-ls" in order to verify the results.

%RESPONSIBLE% Main.unknown
%REVIEW% Never reviewed

-- FernandoLopez1 - 2015-08-13