TWiki> CALICE Web>SoftwareMain>TransferToDIRAC (revision 15)EditAttachPDF

CALICE data is being transferred into the ilcdirac catalogue.

we'll update the status on this page.

a little explanation.

  • SE = Storage Element (where the physical file is stored)
  • LFN=Logical File Name (the name in the catalogue, which points to the physical file(s)/replica(s) )
  • PFN=Physical File Name (name of replica on a SE)

a "good" case:

  • LFN: /grid/calice/tb-cern/native/cernAhcal2012/Runs_04112012.tgz
  • PFN: srm://dcache-se-desy.desy.de/pnfs/desy.de/calice/tb-cern/native/cernAhcal2012/Runs_04112012.tgz

the PFN is just the LFN (with some SE-dependent prefix).

in this case, we can just create a catalogue entry in dirac.

a "bad" case:

  • LFN: /grid/calice/tb-desy-siw-2012/data_skiroc_HVok/sk_beam_sc31_120_3003_19h_noW_3GeV.raw
  • PFN: srm://polgrid4.in2p3.fr/dpm/in2p3.fr/home/calice/generated/2012-04-12/file9ab4a8cf-e837-40b9-b84e-d61d28c1cedc

PFN bears no relation to the LFN: this type of file must physically re-copied using dirac tools.

progress

Plan to migrate the following directories:

/grid/calice/tb-cern

this directory considered complete

  • 157k files
  • 136k good, at DESY
  • 11k bad filenames

LFN with no replicas found:

  • /calice/tb-cern/raw/conv_v0402/Run350346.001.slcio
  • /calice/tb-cern/raw/conv_v0402/log/theconv_write.steer.330004.tar.gz
  • /calice/tb-cern/raw/conv_v0402/log_dbup_oct07/theconv_write.steer.330062.tar.gz
  • /calice/tb-cern/raw/conv_v0402/log_dbup_oct07/theconv_write.steer.300719.tar.gz
  • /calice/tb-cern/rec/rec_v0406/Run331536_rec.000.slcio
  • /calice/tb-cern/rec/rec_v0407pre/Run330794_rec.000.slcio
  • /calice/tb-cern/rec/rec_v0407track/Run330294_rec.000.slcio
  • /calice/tb-cern/data_quality/prod_005/reco/Reco350316.000.slcio
  • /calice/tb-cern/data_quality/prod_005/reco/Reco350357.000.slcio
  • /calice/tb-cern/data_quality/prod_005/reco/Reco350364.000.slcio
  • /calice/tb-cern/data_quality/prod_001/Reco350145_log.tar.gz

126 files with zero file size not imported:

  • /calice/tb-cern/rec/rec_v0407/Run330232_rec.000.slcio
  • and the remainder with filenames like /calice/tb-cern/data_quality/prod_005/reco/Reco3*.000.slcio

/grid/calice/gain

Eldwan

  • 67k files
  • 44k at DESY
  • 18k at in2p3
  • 4k at *.ac.uk
  • remainder at kek, ciemat

these are all BAD. Are they needed?

/grid/calice/ic_ahcal

Eldwan

155 files, all bad

/calice/ic_ahcal directory does not exist on the DESY SE, and is troublesome to create. Can we copy these somewhere else?

/grid/calice/reco_job

This directory is considered finished

29 files (all bad),

copied to IN2P3 SE, except the following, which have some problems (files not found on SE...)

  • /calice/reco_job/lfn_root40402libs.tgz_junk
  • /calice/reco_job/lfn_root40402libs.tgz

/grid/calice/SDHCAL

Gerald is working on it. running out of space.

  • 137k files
  • 129k at lyon
  • all BAD

/grid/calice/software

This directory is considered finished

all "bad"

all copied (to SE IN2P3-USER)

/grid/calice/status

This directory is considered finished

copied to IN2P3 SE

/grid/calice/statusFnal08

This directory is considered finished

copied to IN2P3 SE

/grid/calice/statusFnal08_SCECAL

This directory is considered finished

copied to IN2P3 SE

/grid/calice/statusFnal09

This directory is considered finished

copied to IN2P3 SE

/grid/calice/tb-desy

This directory is considered finished

almost all files transferred

exceptions:

entries in LFC without any replicas on a SE:

  • /calice/tb-desy/maps/native/490000/Run490003.004.bin
  • /calice/tb-desy/maps/native/490000/Run490067.001.bin
  • /calice/tb-desy/maps/native/470000/Run470011.003.bin
  • /calice/tb-desy/maps/native/450000/Run450011.000.bin
  • /calice/tb-desy/maps/native/450000/Run450033.000.bin
  • /calice/tb-desy/rec/rec_v0403-pre4/log/therec.steer.230097.tar.gz
  • /calice/tb-desy/native/desyAhcal2016/December2016/raw/run000007.raw
  • /calice/tb-desy/native/dat/Run240211.000.bin
  • /calice/tb-desy/native/dat/Run200070.000.bin
  • /calice/tb-desy/native/dat/Run200071.001.bin
  • /calice/tb-desy/native/dat/Run240210.000.bin
  • /calice/tb-desy/native/dat/Run200071.000.bin
  • /calice/tb-desy/native/dat/Run240046.000.bin
  • /calice/tb-desy/native/dat/Run240215.000.bin
  • /calice/tb-desy/native/dat/Run240136.000.bin

91 files under: * /calice/tb-desy/maps/native/450000 * /calice/tb-desy/maps/native/470000 which were stored at srm:/dcache01.tier2.hep.manchester.ac.uk/ (this SE seems to have been retired)

the following 28 files no longer reachable under srm:/grid05.lal.in2p3.fr/dpm/lal.in2p3.fr/home/calice/

  • /calice/tb-desy/native/desySiEcal-07-2013/run0026/run0026_sc_2.txt
  • /calice/tb-desy/native/desySiEcal-07-2013/run0027/run0027_sc_5.txt
  • /calice/tb-desy/native/desySiEcal-07-2013/run0027/run0027_by_dif2.raw
  • /calice/tb-desy/native/desySiEcal-07-2013/run0008/run0008_sc_2.txt
  • /calice/tb-desy/native/desySiEcal-07-2013/run0008/run0008_by_dif1.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0021/run0021_by_dif5.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0021/run0021_by_dif1.raw
  • /calice/tb-desy/native/desySiEcal-07-2013/run0021/run0021_sc_0.txt
  • /calice/tb-desy/native/desySiEcal-07-2013/run0004/run0004_by_dif6.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0005/run0005_by_dif6.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0007/run0007_sc_1.txt
  • /calice/tb-desy/native/desySiEcal-07-2013/run0001/first_run5_by_dif6.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0002/first_run6_by_dif6.raw
  • /calice/tb-desy/native/desySiEcal-07-2013/run0003/conf_runCC.xml
  • /calice/tb-desy/native/desySiEcal-07-2013/run0022/run0022_by_dif5.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0022/run0022_by_dif2.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0023/run0023_sc_0.txt
  • /calice/tb-desy/native/desySiEcal-07-2013/run0019/run0019_by_dif4.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0032/run0032_by_dif3.raw
  • /calice/tb-desy/native/desySiEcal-07-2013/run0013/run0013_by_dif5.raw
  • /calice/tb-desy/native/desySiEcal-07-2013/run0012/run0012_by_dif6.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0011/run0011_by_dif5.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0011/run0011_sc_3.txt
  • /calice/tb-desy/native/desySiEcal-07-2013/run0010/run0010_by_dif2.raw
  • /calice/tb-desy/native/desySiEcal-07-2013/run0017/run0017_by_dif5.raw
  • /calice/tb-desy/native/desySiEcal-07-2013/run0016/run0016_by_dif1.raw.root
  • /calice/tb-desy/native/desySiEcal-07-2013/run0015/run0015_by_dif5.raw
  • /calice/tb-desy/native/desySiEcal-07-2013/run0014/run0014_sc_5.txt

otherwise, all files are present.

/grid/calice/tb-desy-siw-2012

This directory is considered finished

  • 7013 files
  • 6907 on dcache-se-desy.desy.de
  • 106 on polgrid4.in2p3.fr
all files are "badly" stored.

all reimported (to the SE "IN2P3-USER"), except

  • 1020 zero-size files

/grid/calice/tb-fnal

This directory is considered finished

all done, except:

entries in LFC without any replicas on a SE:

  • /calice/tb-fnal/raw/conv_v0402/Run510212.000.slcio
  • /calice/tb-fnal/raw/conv_v0402/Run500818.000.slcio
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520332.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520297.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520273.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520196.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520278.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520326.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520300.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520369.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520280.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520301.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520293.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520257.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520274.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520248.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520311.tar.gz
  • /calice/tb-fnal/raw/conv_v0402/log/theconv_write.steer.520319.tar.gz

/grid/calice/tb-MCProduction

  • all "bad"

Too many files to reimport. Probably most are no longer used. If you want to transfer some of these files, please make an explicit request.

Clean-Up

consider deleting physical files not accessible from ilcdirac, to save storage space. These are (almost always) files with "bad" physical file names.

there is either another replica with a good file name which has been imported into ilcdirac catalogue, or they have been re-copied onto a SE in an ilcdirac-friendly way.

files under /calice/tb-desy/desySiEcal* , and stored at polgrid and lal SEs:

files under /calice/tb-fnal, stored on desy SE:

files under /calice/tb-fcern, stored on LAL and DESY SEs:

DanielJeansExternal1 - 2020-08-03

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatext LFC_contents_grid_calice_tb-MCProduction r1 manage 30729.1 K 2020-07-03 - 11:01 DanielJeansExternal contents of tb-MCProduction folder on the LFC
Texttxt LFC_contents_grid_calice_tb-MCProduction_SUMMARY.txt r1 manage 58.2 K 2020-07-03 - 11:01 DanielJeansExternal contents of tb-MCProduction folder on the LFC
Unknown file formatlog delete_desySiEcal_lal.log r1 manage 114.5 K 2020-08-03 - 07:14 DanielJeansExternal files for deletion
Unknown file formatlog delete_desySiEcal_polgrid.log r1 manage 107.7 K 2020-08-03 - 07:14 DanielJeansExternal files for deletion
Unknown file formatlog delete_tb-cern_desy.log r1 manage 2101.4 K 2020-08-03 - 08:07 DanielJeansExternal files to delete from tb-cern
Unknown file formatlog delete_tb-cern_lal.log r1 manage 24.2 K 2020-08-03 - 08:07 DanielJeansExternal files to delete from tb-cern
Unknown file formatlog delete_tb-fnal_desy.log r1 manage 1046.0 K 2020-08-03 - 07:29 DanielJeansExternal files to delete on tb-fnal, at desy
Edit | Attach | Watch | Print version | History: r17 < r16 < r15 < r14 < r13 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r15 - 2020-08-03 - DanielJeansExternal
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CALICE All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback