CALICE data is being transferred into the ilcdirac catalogue.

we'll update the status on this page.

a little explanation.

• SE = Storage Element (where the physical file is stored)
• LFN=Logical File Name (the name in the catalogue, which points to the physical file(s)/replica(s) )
• PFN=Physical File Name (name of replica on a SE)

a "good" case:

• LFN: /grid/calice/tb-cern/native/cernAhcal2012/Runs_04112012.tgz
• PFN: srm://dcache-se-desy.desy.de/pnfs/desy.de/calice/tb-cern/native/cernAhcal2012/Runs_04112012.tgz

the PFN is just the LFN (with some SE-dependent prefix).

in this case, we can just create a catalogue entry in dirac.

a "bad" case:

• LFN: /grid/calice/tb-desy-siw-2012/data_skiroc_HVok/sk_beam_sc31_120_3003_19h_noW_3GeV.raw
• PFN: srm://polgrid4.in2p3.fr/dpm/in2p3.fr/home/calice/generated/2012-04-12/file9ab4a8cf-e837-40b9-b84e-d61d28c1cedc

PFN bears no relation to the LFN: this type of file must physically re-copied using dirac tools.

progress

Plan to migrate the following directories:

/grid/calice/tb-cern

this directory considered complete

• 157k files
• 136k good, at DESY
• 11k bad filenames

LFN with no replicas found:

• /calice/tb-cern/raw/conv_v0402/Run350346.001.slcio
• /calice/tb-cern/raw/conv_v0402/log/theconv_write.steer.330004.tar.gz
• /calice/tb-cern/raw/conv_v0402/log_dbup_oct07/theconv_write.steer.330062.tar.gz
• /calice/tb-cern/raw/conv_v0402/log_dbup_oct07/theconv_write.steer.300719.tar.gz
• /calice/tb-cern/rec/rec_v0406/Run331536_rec.000.slcio
• /calice/tb-cern/rec/rec_v0407pre/Run330794_rec.000.slcio
• /calice/tb-cern/rec/rec_v0407track/Run330294_rec.000.slcio
• /calice/tb-cern/data_quality/prod_005/reco/Reco350316.000.slcio
• /calice/tb-cern/data_quality/prod_005/reco/Reco350357.000.slcio
• /calice/tb-cern/data_quality/prod_005/reco/Reco350364.000.slcio
• /calice/tb-cern/data_quality/prod_001/Reco350145_log.tar.gz

126 files with zero file size not imported:

• /calice/tb-cern/rec/rec_v0407/Run330232_rec.000.slcio
• and the remainder with filenames like /calice/tb-cern/data_quality/prod_005/reco/Reco3*.000.slcio

/grid/calice/gain

Eldwan

• 67k files
• 44k at DESY
• 18k at in2p3
• 4k at *.ac.uk
• remainder at kek, ciemat

these are all BAD. Are they needed?

/grid/calice/ic_ahcal

Eldwan

155 files, all bad

/calice/ic_ahcal directory does not exist on the DESY SE, and is troublesome to create. Can we copy these somewhere else?

/grid/calice/reco_job

This directory is considered finished

29 files (all bad),

copied to IN2P3 SE, except the following, which have some problems (files not found on SE...)

• /calice/reco_job/lfn_root40402libs.tgz_junk
• /calice/reco_job/lfn_root40402libs.tgz

/grid/calice/SDHCAL

Gerald is working on it. running out of space.

• 137k files
• 129k at lyon
• all BAD

/grid/calice/software

This directory is considered finished

all "bad"

all copied (to SE IN2P3-USER)

/grid/calice/status

This directory is considered finished

copied to IN2P3 SE

/grid/calice/statusFnal08

This directory is considered finished

copied to IN2P3 SE

/grid/calice/statusFnal08_SCECAL

This directory is considered finished

copied to IN2P3 SE

/grid/calice/statusFnal09

This directory is considered finished

copied to IN2P3 SE

/grid/calice/tb-desy

This directory is considered finished

almost all files transferred

exceptions:

entries in LFC without any replicas on a SE:

• /calice/tb-desy/maps/native/490000/Run490003.004.bin
• /calice/tb-desy/maps/native/490000/Run490067.001.bin
• /calice/tb-desy/maps/native/470000/Run470011.003.bin
• /calice/tb-desy/maps/native/450000/Run450011.000.bin
• /calice/tb-desy/maps/native/450000/Run450033.000.bin
• /calice/tb-desy/rec/rec_v0403-pre4/log/therec.steer.230097.tar.gz
• /calice/tb-desy/native/desyAhcal2016/December2016/raw/run000007.raw
• /calice/tb-desy/native/dat/Run240211.000.bin
• /calice/tb-desy/native/dat/Run200070.000.bin
• /calice/tb-desy/native/dat/Run200071.001.bin
• /calice/tb-desy/native/dat/Run240210.000.bin
• /calice/tb-desy/native/dat/Run200071.000.bin
• /calice/tb-desy/native/dat/Run240046.000.bin
• /calice/tb-desy/native/dat/Run240215.000.bin
• /calice/tb-desy/native/dat/Run240136.000.bin

91 files under: * /calice/tb-desy/maps/native/450000 * /calice/tb-desy/maps/native/470000 which were stored at srm:/dcache01.tier2.hep.manchester.ac.uk/ (this SE seems to have been retired)

the following 28 files no longer reachable under srm:/grid05.lal.in2p3.fr/dpm/lal.in2p3.fr/home/calice/

• /calice/tb-desy/native/desySiEcal-07-2013/run0026/run0026_sc_2.txt
• /calice/tb-desy/native/desySiEcal-07-2013/run0027/run0027_sc_5.txt
• /calice/tb-desy/native/desySiEcal-07-2013/run0027/run0027_by_dif2.raw
• /calice/tb-desy/native/desySiEcal-07-2013/run0008/run0008_sc_2.txt
• /calice/tb-desy/native/desySiEcal-07-2013/run0008/run0008_by_dif1.raw.root
• /calice/tb-desy/native/desySiEcal-07-2013/run0021/run0021_by_dif5.raw.root
• /calice/tb-desy/native/desySiEcal-07-2013/run0021/run0021_by_dif1.raw
• /calice/tb-desy/native/desySiEcal-07-2013/run0021/run0021_sc_0.txt
• /calice/tb-desy/native/desySiEcal-07-2013/run0004/run0004_by_dif6.raw.root
• /calice/tb-desy/native/desySiEcal-07-2013/run0005/run0005_by_dif6.raw.root
• /calice/tb-desy/native/desySiEcal-07-2013/run0007/run0007_sc_1.txt
• /calice/tb-desy/native/desySiEcal-07-2013/run0001/first_run5_by_dif6.raw.root
• /calice/tb-desy/native/desySiEcal-07-2013/run0002/first_run6_by_dif6.raw
• /calice/tb-desy/native/desySiEcal-07-2013/run0003/conf_runCC.xml
• /calice/tb-desy/native/desySiEcal-07-2013/run0022/run0022_by_dif5.raw.root
• /calice/tb-desy/native/desySiEcal-07-2013/run0022/run0022_by_dif2.raw.root
• /calice/tb-desy/native/desySiEcal-07-2013/run0023/run0023_sc_0.txt
• /calice/tb-desy/native/desySiEcal-07-2013/run0019/run0019_by_dif4.raw.root
• /calice/tb-desy/native/desySiEcal-07-2013/run0032/run0032_by_dif3.raw
• /calice/tb-desy/native/desySiEcal-07-2013/run0013/run0013_by_dif5.raw
• /calice/tb-desy/native/desySiEcal-07-2013/run0012/run0012_by_dif6.raw.root
• /calice/tb-desy/native/desySiEcal-07-2013/run0011/run0011_by_dif5.raw.root
• /calice/tb-desy/native/desySiEcal-07-2013/run0011/run0011_sc_3.txt
• /calice/tb-desy/native/desySiEcal-07-2013/run0010/run0010_by_dif2.raw
• /calice/tb-desy/native/desySiEcal-07-2013/run0017/run0017_by_dif5.raw
• /calice/tb-desy/native/desySiEcal-07-2013/run0016/run0016_by_dif1.raw.root
• /calice/tb-desy/native/desySiEcal-07-2013/run0015/run0015_by_dif5.raw
• /calice/tb-desy/native/desySiEcal-07-2013/run0014/run0014_sc_5.txt

otherwise, all files are present.

/grid/calice/tb-desy-siw-2012

This directory is considered finished

• 7013 files
• 6907 on dcache-se-desy.desy.de
• 106 on polgrid4.in2p3.fr

all reimported (to the SE "IN2P3-USER"), except

• 1020 zero-size files

/grid/calice/tb-fnal

This directory is considered finished

all done, except:

entries in LFC without any replicas on a SE:

• /calice/tb-fnal/raw/conv_v0402/Run510212.000.slcio
• /calice/tb-fnal/raw/conv_v0402/Run500818.000.slcio
• /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520332.tar.gz
• /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520297.tar.gz
• /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520273.tar.gz
• /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520196.tar.gz
• /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520278.tar.gz
• /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520326.tar.gz
• /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520300.tar.gz
• /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520369.tar.gz
• /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520280.tar.gz
• /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520301.tar.gz
• /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520293.tar.gz
• /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520257.tar.gz
• /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520274.tar.gz
• /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520248.tar.gz
• /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520311.tar.gz
• /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520319.tar.gz

/grid/calice/tb-MCProduction

• all "bad"

• LFC_contents_grid_calice_tb-MCProduction: contents of tb-MCProduction folder on the LFC

• LFC_contents_grid_calice_tb-MCProduction_SUMMARY.txt: contents of tb-MCProduction folder on the LFC (summary)

Too many files to reimport. Probably most are no longer used. If you want to transfer some of these files, please make an explicit request. They will otherwise be deleted.

no requests received, so these MC files have now been deleted

Clean-Up

consider deleting physical files not accessible from ilcdirac, to save storage space. These are (almost always) files with "bad" physical file names.

there is either another replica with a good file name which has been imported into ilcdirac catalogue, or they have been re-copied onto a SE in an ilcdirac-friendly way.

files under /calice/tb-desy/desySiEcal* , and stored at polgrid and lal SEs:

• delete_desySiEcal_polgrid.log: files for deletion
• delete_desySiEcal_lal.log: files for deletion

files under /calice/tb-fnal, stored on desy SE:

• delete_tb-fnal_desy.log: files to delete on tb-fnal, at desy

files under /calice/tb-cern, stored on LAL and DESY SEs:

• delete_tb-cern_lal.log: files to delete from tb-cern
• delete_tb-cern_desy.log: files to delete from tb-cern

In addition, we plan to delete all files under tb-MCProduction.

DanielJeansExternal1 - 2020-08-03