CALICE data is being transferred into the ilcdirac catalogue.

we'll update the status on this page.

a little explanation.

  • SE = Storage Element (where the physical file is stored)
  • LFN=Logical File Name (the name in the catalogue, which points to the physical file(s)/replica(s) )
  • PFN=Physical File Name (name of replica on a SE)

a "good" case:

  • LFN: /grid/calice/tb-cern/native/cernAhcal2012/Runs_04112012.tgz
  • PFN: srm://dcache-se-desy.desy.de/pnfs/desy.de/calice/tb-cern/native/cernAhcal2012/Runs_04112012.tgz

the PFN is just the LFN (with some SE-dependent prefix).

in this case, we can just create a catalogue entry in dirac.

a "bad" case:

  • LFN: /grid/calice/tb-desy-siw-2012/data_skiroc_HVok/sk_beam_sc31_120_3003_19h_noW_3GeV.raw
  • PFN: srm://polgrid4.in2p3.fr/dpm/in2p3.fr/home/calice/generated/2012-04-12/file9ab4a8cf-e837-40b9-b84e-d61d28c1cedc

PFN bears no relation to the LFN: this type of file must physically re-copied using dirac tools.

progress

Plan to migrate the following directories:

/grid/calice/tb-cern

this directory considered complete

  • 157k files
  • 136k good, at DESY
  • 11k bad filenames

LFN with no replicas found:

  • /calice/tb-cern/raw/conv_v0402/Run350346.001.slcio
  • /calice/tb-cern/raw/conv_v0402/log/theconv_write.steer.330004.tar.gz
  • /calice/tb-cern/raw/conv_v0402/log_dbup_oct07/theconv_write.steer.330062.tar.gz
  • /calice/tb-cern/raw/conv_v0402/log_dbup_oct07/theconv_write.steer.300719.tar.gz
  • /calice/tb-cern/rec/rec_v0406/Run331536_rec.000.slcio
  • /calice/tb-cern/rec/rec_v0407pre/Run330794_rec.000.slcio
  • /calice/tb-cern/rec/rec_v0407track/Run330294_rec.000.slcio
  • /calice/tb-cern/data_quality/prod_005/reco/Reco350316.000.slcio
  • /calice/tb-cern/data_quality/prod_005/reco/Reco350357.000.slcio
  • /calice/tb-cern/data_quality/prod_005/reco/Reco350364.000.slcio
  • /calice/tb-cern/data_quality/prod_001/Reco350145_log.tar.gz

126 files with zero file size not imported:

  • /calice/tb-cern/rec/rec_v0407/Run330232_rec.000.slcio
  • and the remainder with filenames like /calice/tb-cern/data_quality/prod_005/reco/Reco3*.000.slcio

/grid/calice/gain

Eldwan

  • 67k files
  • 44k at DESY
  • 18k at in2p3
  • 4k at *.ac.uk
  • remainder at kek, ciemat

these are all BAD. Are they needed?

/grid/calice/ic_ahcal

Eldwan

155 files, all bad

/calice/ic_ahcal directory does not exist on the DESY SE, and is troublesome to create. Can we copy these somewhere else?

/grid/calice/reco_job

This directory is considered finished

29 files (all bad),

copied to IN2P3 SE, except the following, which have some problems (files not found on SE...)

  • /calice/reco_job/lfn_root40402libs.tgz_junk
  • /calice/reco_job/lfn_root40402libs.tgz

/grid/calice/SDHCAL

Gerald is working on it. running out of space.

  • 137k files
  • 129k at lyon
  • all BAD

/grid/calice/software

This directory is considered finished

all "bad"

all copied (to SE IN2P3-USER)

/grid/calice/status

This directory is considered finished

copied to IN2P3 SE

/grid/calice/statusFnal08

This directory is considered finished

copied to IN2P3 SE

/grid/calice/statusFnal08_SCECAL

This directory is considered finished

copied to IN2P3 SE

/grid/calice/statusFnal09

This directory is considered finished

copied to IN2P3 SE

/grid/calice/tb-desy

This directory is considered finished

almost all files transferred

exceptions:

entries in LFC without any replicas on a SE:

  • /calice/tb-desy/maps/native/490000/Run490003.004.bin
  • /calice/tb-desy/maps/native/490000/Run490067.001.bin
  • /calice/tb-desy/maps/native/470000/Run470011.003.bin
  • /calice/tb-desy/maps/native/450000/Run450011.000.bin
  • /calice/tb-desy/maps/native/450000/Run450033.000.bin
  • /calice/tb-desy/rec/rec_v0403-pre4/log/therec.steer.230097.tar.gz
  • /calice/tb-desy/native/desyAhcal2016/December2016/raw/run000007.raw
  • /calice/tb-desy/native/dat/Run240211.000.bin
  • /calice/tb-desy/native/dat/Run200070.000.bin
  • /calice/tb-desy/native/dat/Run200071.001.bin
  • /calice/tb-desy/native/dat/Run240210.000.bin
  • /calice/tb-desy/native/dat/Run200071.000.bin
  • /calice/tb-desy/native/dat/Run240046.000.bin
  • /calice/tb-desy/native/dat/Run240215.000.bin
  • /calice/tb-desy/native/dat/Run240136.000.bin

91 files under: * /calice/tb-desy/maps/native/450000 * /calice/tb-desy/maps/native/470000 which were stored at srm:/dcache01.tier2.hep.manchester.ac.uk/ (this SE seems to have been retired)

the following 28 files no longer reachable under srm:/grid05.lal.in2p3.fr/dpm/lal.in2p3.fr/home/calice/

  • /calice/tb-desy/native/desySiEcal-07-2013/run0026/run0026_sc_2.txt
  • /calice/tb-desy/native/desySiEcal-07-2013/run0027/run0027_sc_5.txt
  • /calice/tb-desy/native/desySiEcal-07-2013/run0027/run0027_by_dif2.raw
  • /calice/tb-desy/native/desySiEcal-07-2013/run0008/run0008_sc_2.txt
  • /calice/tb-desy/native/desySiEcal-07-2013/run0008/run0008_by_dif1.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0021/run0021_by_dif5.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0021/run0021_by_dif1.raw
  • /calice/tb-desy/native/desySiEcal-07-2013/run0021/run0021_sc_0.txt
  • /calice/tb-desy/native/desySiEcal-07-2013/run0004/run0004_by_dif6.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0005/run0005_by_dif6.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0007/run0007_sc_1.txt
  • /calice/tb-desy/native/desySiEcal-07-2013/run0001/first_run5_by_dif6.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0002/first_run6_by_dif6.raw
  • /calice/tb-desy/native/desySiEcal-07-2013/run0003/conf_runCC.xml
  • /calice/tb-desy/native/desySiEcal-07-2013/run0022/run0022_by_dif5.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0022/run0022_by_dif2.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0023/run0023_sc_0.txt
  • /calice/tb-desy/native/desySiEcal-07-2013/run0019/run0019_by_dif4.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0032/run0032_by_dif3.raw
  • /calice/tb-desy/native/desySiEcal-07-2013/run0013/run0013_by_dif5.raw
  • /calice/tb-desy/native/desySiEcal-07-2013/run0012/run0012_by_dif6.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0011/run0011_by_dif5.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0011/run0011_sc_3.txt
  • /calice/tb-desy/native/desySiEcal-07-2013/run0010/run0010_by_dif2.raw
  • /calice/tb-desy/native/desySiEcal-07-2013/run0017/run0017_by_dif5.raw
  • /calice/tb-desy/native/desySiEcal-07-2013/run0016/run0016_by_dif1.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0015/run0015_by_dif5.raw
  • /calice/tb-desy/native/desySiEcal-07-2013/run0014/run0014_sc_5.txt

otherwise, all files are present.

/grid/calice/tb-desy-siw-2012

This directory is considered finished

  • 7013 files
  • 6907 on dcache-se-desy.desy.de
  • 106 on polgrid4.in2p3.fr
all files are "badly" stored.

all reimported (to the SE "IN2P3-USER"), except

  • 1020 zero-size files

/grid/calice/tb-fnal

This directory is considered finished

all done, except:

entries in LFC without any replicas on a SE:

  • /calice/tb-fnal/raw/conv_v0402/Run510212.000.slcio
  • /calice/tb-fnal/raw/conv_v0402/Run500818.000.slcio
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520332.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520297.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520273.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520196.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520278.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520326.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520300.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520369.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520280.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520301.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520293.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520257.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520274.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520248.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520311.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520319.tar.gz

/grid/calice/tb-MCProduction

  • all "bad"

Too many files to reimport. Probably most are no longer used. If you want to transfer some of these files, please make an explicit request. They will otherwise be deleted.

no requests received, so these MC files have now been deleted

Clean-Up

consider deleting physical files not accessible from ilcdirac, to save storage space. These are (almost always) files with "bad" physical file names.

there is either another replica with a good file name which has been imported into ilcdirac catalogue, or they have been re-copied onto a SE in an ilcdirac-friendly way.

files under /calice/tb-desy/desySiEcal* , and stored at polgrid and lal SEs:

files under /calice/tb-fnal, stored on desy SE:

files under /calice/tb-cern, stored on LAL and DESY SEs:

In addition, we plan to delete all files under tb-MCProduction.

DanielJeansExternal1 - 2020-08-03

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatext LFC_contents_grid_calice_tb-MCProduction r1 manage 30729.1 K 2020-07-03 - 11:01 DanielJeansExternal contents of tb-MCProduction folder on the LFC
Texttxt LFC_contents_grid_calice_tb-MCProduction_SUMMARY.txt r1 manage 58.2 K 2020-07-03 - 11:01 DanielJeansExternal contents of tb-MCProduction folder on the LFC
Unknown file formatlog delete_desySiEcal_lal.log r1 manage 114.5 K 2020-08-03 - 07:14 DanielJeansExternal files for deletion
Unknown file formatlog delete_desySiEcal_polgrid.log r1 manage 107.7 K 2020-08-03 - 07:14 DanielJeansExternal files for deletion
Unknown file formatlog delete_tb-cern_desy.log r1 manage 2101.4 K 2020-08-03 - 08:07 DanielJeansExternal files to delete from tb-cern
Unknown file formatlog delete_tb-cern_lal.log r1 manage 24.2 K 2020-08-03 - 08:07 DanielJeansExternal files to delete from tb-cern
Unknown file formatlog delete_tb-fnal_desy.log r1 manage 1046.0 K 2020-08-03 - 07:29 DanielJeansExternal files to delete on tb-fnal, at desy

This topic: CALICE > WebHome > SoftwareMain > TransferToDIRAC
Topic revision: r17 - 2020-09-16 - DanielJeansExternal
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback