T0-->T1 Data Transfer tests in preparation for Run2

Motivation

Test T0->T1 transfers at the rates expected in Run2, taking into account the relevant changes in Tier-0 and Tier-1 processing, namely:

  • Tier-0 processing runs against EOS
  • Prompt reconstruction shared with Tier-1s
  • Disk/tape separation at Tier-1s
  • RECO is a temporary disk-only data format

Assumptions for the test

  • Event sizes/rates for 2015 are taken from the presentation at the Fall 2014 O&C week - 3.0 MB/evt RECO, 0.63 MB/evt AOD
  • RAW is produced on EOS/T2_CH_CERN and exported in three copies: one custodial copy on tape at T0_CH_CERN_MSS; one custodial copy on tape distributed across T1_*_MSS; one disk copy distributed across T1_*_Disk
  • PromptRECO is produced 50% on EOS/T2_CH_CERN and 50% at T1s; the T2_CH_CERN fraction is exported in one copy distributed across T1_*_Disk
  • PromptAOD is produced 50% on EOS/T2_CH_CERN and 50% at T1s; the T2_CH_CERN fraction is exported in two copies: one custodial copy distributed across T1_*_MSS; one disk copy distributed across T1_*_Disk
  • The dataset transfers to the Tier-1s are distributed in proportion to their 2015 pledges (either by tape, disk or CPU pledge share - it doesn't matter, since the relative shares don't differ by more than a few percent)
  • The rates are calculated assuming a worst-case scenario, with no rerouting in PhEDEx, i.e. each RAW & AOD file is exported from T2_CH_CERN to T1 twice (once to Disk and once to MSS)

Procedure for the test

  • Set the LoadTest rates for T2_CH_CERN-->T0_CH_CERN_MSS, T2_CH_CERN-->T1_*_Disk, T2_CH_CERN-->T1_*_MSS according to the targets in the table
    • Increasing gradually in steps, e.g. 25%-50%-75%-100%

Timeline for the test

  • Nov 11th-17th
  • Repeat Nov 18th-24th in case of issues
  • 2 Round Feb 16th-19th (Increasing in 2 steps 50%-100%)
  • 3 Round April 2015
  • 4 Round July 7th - 2015
  • 5 Round October 16th - 2015
  • 6 Round January 25th - 2016

Progress

March 2017

January 2016

October 2015

July 2015

April 2015

Feb 2015

Site 16Feb 17Feb 18Feb
FNAL_MSS 288/350 266/350 353/350
FNAL_Disk 218/550 65/550 40/550
CNAF_MSS 38/38 73/76 73/76
CNAF_Disk 86/90 147/180 174/180
RAL_MSS 22/24 46/47 46/47
RAL_Disk 53/56 103/111 89/111
KIT_MSS 29/30 56/59 51/59
KIT_Disk 60/60 117/124 105/124
PIC_MSS 14/15 29/30 29/30
PIC_Disk 35/36 68/71 69/71
JINR_MSS - -
JINR_Disk 47/67 53/133 91/133
Transfer Quality Plot Plot Plot
* FNAL_Disk: Outgoing transfers and incoming transfers(Staging Test) in the production instance affected the test.

Nov 2014

Values: Actual Rate/Target Rate
Site 11Nov 12Nov 13Nov 14Nov 15Nov 16Nov
FNAL_MSS 59/60 114/120 155/180 31/235 252/235 71/235
FNAL_Disk 157/140 262/280 386/420 451/556 471/556 496/556
CNAF_MSS 19/19 37/38 55/57 72/76 75/76 74/76
CNAF_Disk 51/45 86/90 127/135 172/180 172/180 174/180
RAL_MSS 12/12 23/24 35/36 46/47 46/47 46/47
RAL_Disk 31/28 54/56 80/84 106/111 107/111 107/111
IN2P3_MSS 11/11 21/22 32/33 37/44 43/44 43/44
IN2P3_Disk 29/26 45/52 75/78 86/104 101/104 100/104
KIT_MSS 13/15 32/30 9/45 60/59 51/59 64/59
KIT_Disk 31/30 58/60 48/90 140/124 120/124 119/124
PIC_MSS 8/8 16/16 22/24 29/30 30/30 29/30
PIC_Disk 21/18 35/36 52/54 69/71 68/71 70/71
JINR_MSS - - - - - -
JINR_Disk 38/34 65/68 97/102 120/133 122/133 117/133
Transfer Quality Plot Plot Plot Plot Plot Plot

Site Target Rate 18Nov 19Nov 20Nov 21Nov 22Nov 23Nov
FNAL_MSS 235 MB/s 245 212 99 5 123 268
FNAL_Disk 556 MB/s 305 187 214 257 110 278
CNAF_MSS 76 MB/s 73 74 74 74 73 75
CNAF_Disk 180 MB/s 158 164 175 182 184 172
RAL_MSS 47 MB/s 47 46 47 46 46 46
RAL_Disk 111 MB/s 106 108 107 107 108 106
IN2P3_MSS 44 MB/s 47 43 42 43 43 43
IN2P3_Disk 104 MB/s 97 105 100 82 104 106
KIT_MSS 59 MB/s 1 49 40 48 1 7
KIT_Disk 124 MB/s 96 59 18 68 6 20
PIC_MSS 30 MB/s 22 5 25 15 35 30
PIC_Disk 71 MB/s 34 5 50 34 73 70
JINR_MSS 40 MB/s - - - - - -
JINR_Disk 133 MB/s 108 100 113 109 120 110
Transfer Quality   Plot Plot Plot Plot Plot Plot

Opened tickets

-- NicoloMagini - 2014-11-10

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng FromFNALProd.png r2 r1 manage 41.5 K 2015-02-19 - 23:53 JorgeAlbertoDiazCruz  
PNGpng T0FNALDebug.png r1 manage 43.6 K 2015-02-19 - 23:54 JorgeAlbertoDiazCruz  
PNGpng ToFNALProd.png r1 manage 45.2 K 2015-02-19 - 23:54 JorgeAlbertoDiazCruz  
Edit | Attach | Watch | Print version | History: r32 < r31 < r30 < r29 < r28 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r32 - 2017-03-18 - SiddharthMNarayanan
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback