Test on lxplus

  • CMSSW 3_3_2
  • PAT
  • rfio:/castor/cern.ch/cms/store/relval/CMSSW_3_3_2/RelValQCD_Pt_80_120/GEN-SIM-RECO/MC_31X_V9-v2/0000/3A005AC9-58C8-DE11-A2F5-001A9281173A.root

noPatch

std real 6m22.034s user 4m52.267s sys 0m14.462s

app-only real 5m52.468s user 4m46.947s sys 0m16.439s

Patch std real 6m6.318s user 4m45.188s sys 0m11.386s

app-only 5m26.532s user 4m44.177s sys 0m9.440s

Test from Ian

1 file

Unpatched

standard

real 7m45.230s user 5m28.089s sys 0m19.066s

<Catalog>trivialcatalog_file:/afs/cern.ch/cms/SITECONF/local/PhEDEx/storage.xml?protocol=rfio</Catalog>
  <PFN>rfio:/castor/cern.ch/cms/store/relval/CMSSW_3_3_3/RelValZTT/GEN-SIM-RECO/STARTUP31X_V8-v1/0003/50FF8E3B-1FD2-DE11-806E-001D09F29321.root</PFN>
 <counter-value subsystem='rfio' counter-name='close' num-operations='1' num-successful-operations='1' total-megabytes='0' total-msecs='36.044' min-msecs='36.044' max-msecs='36.044'/>
 <counter-value subsystem='rfio' counter-name='construct' num-operations='1' num-successful-operations='1' total-megabytes='0' total-msecs='0.002048' min-msecs='0.002048' max-msecs='0.002048'/>
 <counter-value subsystem='rfio' counter-name='destruct' num-operations='1' num-successful-operations='1' total-megabytes='0' total-msecs='0.083968' min-msecs='0.083968'max-msecs='0.083968'/>
 <counter-value subsystem='rfio' counter-name='open' num-operations='1' num-successful-operations='1' total-megabytes='0' total-msecs='21328.4' min-msecs='21328.4' max-msecs='21328.4'/>
 <counter-value subsystem='rfio' counter-name='position' num-operations='23197' num-successful-operations='23197' total-megabytes='0' total-msecs='159.077' min-msecs='0.002048' max-msecs='0.977152'/>
 <counter-value subsystem='rfio' counter-name='prefetch' num-operations='0' num-successful-operations='0' total-megabytes='0' total-msecs='0' min-msecs='0' max-msecs='0'/>
 <counter-value subsystem='rfio' counter-name='read' num-operations='23189' num-successful-operations='23189' total-megabytes='205.262' total-msecs='78801.1' min-msecs='0.004864' max-msecs='370.958'/>
 <counter-value subsystem='rfio' counter-name='readv' num-operations='0' num-successful-operations='0' total-megabytes='0' total-msecs='0' min-msecs='0' max-msecs='0'/>
 <counter-value subsystem='rfio' counter-name='stagein' num-operations='1' num-successful-operations='1' total-megabytes='0' total-msecs='138.138' min-msecs='138.138' max-msecs='138.138'/>
 <counter-value subsystem='rfio' counter-name='write' num-operations='0' num-successful-operations='0' total-megabytes='0' total-msecs='0' min-msecs='0' max-msecs='0'/>
 <counter-value subsystem='rfio' counter-name='writev' num-operations='0' num-successful-operations='0' total-megabytes='0' total-msecs='0' min-msecs='0' max-msecs='0'/>
 <counter-value subsystem='tstoragefile' counter-name='close' num-operations='2' num-successful-operations='2' total-megabytes='0' total-msecs='36.5289' min-msecs='0.367872' max-msecs='36.161'/>
 <counter-value subsystem='tstoragefile' counter-name='construct' num-operations='2' num-successful-operations='2' total-megabytes='0' total-msecs='27349.4' min-msecs='25.793' max-msecs='27323.6'/>
 <counter-value subsystem='tstoragefile' counter-name='flush' num-operations='4' num-successful-operations='4' total-megabytes='0' total-msecs='132.602' min-msecs='0.699904' max-msecs='60.055'/>
 <counter-value subsystem='tstoragefile' counter-name='read' num-operations='23189' num-successful-operations='23189' total-megabytes='205.262' total-msecs='78927.5' min-msecs='0.00896' max-msecs='370.963'/>
 <counter-value subsystem='tstoragefile' counter-name='read-actual' num-operations='23189' num-successful-operations='23189' total-megabytes='205.262' total-msecs='78873.7' min-msecs='0.006912' max-msecs='370.961'/>
 <counter-value subsystem='tstoragefile' counter-name='seek' num-operations='30810' num-successful-operations='30810' total-megabytes='0' total-msecs='1209.98' min-msecs='0.002048' max-msecs='7.50387'/>
 <counter-value subsystem='tstoragefile' counter-name='stat' num-operations='2' num-successful-operations='2' total-megabytes='0' total-msecs='1.7751' min-msecs='0.768' max-msecs='1.0071'/>
 <counter-value subsystem='tstoragefile' counter-name='write' num-operations='7621' num-successful-operations='7621' total-megabytes='28.0065' total-msecs='4026.23' min-msecs='0.0128' max-msecs='695.06'/>
 <counter-value subsystem='tstoragefile' counter-name='write-actual' num-operations='7621' num-successful-operations='7621' total-megabytes='28.0065' total-msecs='3998.85' min-msecs='0.009984' max-msecs='695.056'/>
 <counter-value subsystem='tstoragefile' counter-name='write-via-cache' num-operations='7621' num-successful-operations='0' total-megabytes='0' total-msecs='0' min-msecs='0' max-msecs='0'/>

app-only, cache 20MB

real 6m55.493s user 5m23.696s sys 0m14.068s

  <PFN>rfio:/castor/cern.ch/cms/store/relval/CMSSW_3_3_3/RelValZTT/GEN-SIM-RECO/STARTUP31X_V8-v1/0003/50FF8E3B-1FD2-DE11-806E-001D09F29321.root</PFN>
 <counter-value subsystem='rfio' counter-name='close' num-operations='1' num-successful-operations='1' total-megabytes='0' total-msecs='28.1582' min-msecs='28.1582' max-msecs='28.1582'/>
 <counter-value subsystem='rfio' counter-name='construct' num-operations='1' num-successful-operations='1' total-megabytes='0' total-msecs='0.002048' min-msecs='0.002048' max-msecs='0.002048'/>
 <counter-value subsystem='rfio' counter-name='destruct' num-operations='1' num-successful-operations='1' total-megabytes='0' total-msecs='0.071168' min-msecs='0.071168'max-msecs='0.071168'/>
 <counter-value subsystem='rfio' counter-name='open' num-operations='1' num-successful-operations='1' total-megabytes='0' total-msecs='34863.1' min-msecs='34863.1' max-msecs='34863.1'/>
 <counter-value subsystem='rfio' counter-name='position' num-operations='67097' num-successful-operations='67097' total-megabytes='0' total-msecs='272.413' min-msecs='0'max-msecs='1.75104'/>
 <counter-value subsystem='rfio' counter-name='prefetch' num-operations='0' num-successful-operations='0' total-megabytes='0' total-msecs='0' min-msecs='0' max-msecs='0'/>
 <counter-value subsystem='rfio' counter-name='read' num-operations='1244' num-successful-operations='1244' total-megabytes='26.4522' total-msecs='4536.04' min-msecs='0.004864' max-msecs='298.966'/>
 <counter-value subsystem='rfio' counter-name='readv' num-operations='29' num-successful-operations='29' total-megabytes='588.405' total-msecs='30722.5' min-msecs='164.845' max-msecs='1610.25'/>
 <counter-value subsystem='rfio' counter-name='stagein' num-operations='1' num-successful-operations='1' total-megabytes='0' total-msecs='136.314' min-msecs='136.314' max-msecs='136.314'/>
 <counter-value subsystem='rfio' counter-name='write' num-operations='0' num-successful-operations='0' total-megabytes='0' total-msecs='0' min-msecs='0' max-msecs='0'/>
 <counter-value subsystem='rfio' counter-name='writev' num-operations='0' num-successful-operations='0' total-megabytes='0' total-msecs='0' min-msecs='0' max-msecs='0'/>
 <counter-value subsystem='tstoragefile' counter-name='close' num-operations='2' num-successful-operations='2' total-megabytes='0' total-msecs='28.6321' min-msecs='0.361984' max-msecs='28.2701'/>
 <counter-value subsystem='tstoragefile' counter-name='construct' num-operations='2' num-successful-operations='2' total-megabytes='0' total-msecs='40476.6' min-msecs='39.8551' max-msecs='40436.7'/>
 <counter-value subsystem='tstoragefile' counter-name='flush' num-operations='4' num-successful-operations='4' total-megabytes='0' total-msecs='139.393' min-msecs='0.78592' max-msecs='64.9188'/>
 <counter-value subsystem='tstoragefile' counter-name='read' num-operations='23189' num-successful-operations='23189' total-megabytes='205.262' total-msecs='34287.4' min-msecs='0' max-msecs='1563.24'/>
 <counter-value subsystem='tstoragefile' counter-name='read-actual' num-operations='1244' num-successful-operations='1244' total-megabytes='26.4522' total-msecs='4539.55' min-msecs='0.006912' max-msecs='298.969'/>
 <counter-value subsystem='tstoragefile' counter-name='read-async' num-operations='1' num-successful-operations='0' total-megabytes='0' total-msecs='0' min-msecs='0' max-msecs='0'/>
 <counter-value subsystem='tstoragefile' counter-name='read-via-cache' num-operations='21955' num-successful-operations='21945' total-megabytes='178.81' total-msecs='20802.7' min-msecs='0' max-msecs='1563.23'/>
 <counter-value subsystem='tstoragefile' counter-name='readv-actual' num-operations='29' num-successful-operations='29' total-megabytes='588.405' total-msecs='30722.7' min-msecs='164.848' max-msecs='1610.26'/>
 <counter-value subsystem='tstoragefile' counter-name='seek' num-operations='74710' num-successful-operations='74710' total-megabytes='0' total-msecs='522.992' min-msecs='0' max-msecs='8.32205'/>
 <counter-value subsystem='tstoragefile' counter-name='stat' num-operations='2' num-successful-operations='2' total-megabytes='0' total-msecs='1.17299' min-msecs='0.577024' max-msecs='0.595968'/>
 <counter-value subsystem='tstoragefile' counter-name='write' num-operations='7621' num-successful-operations='7621' total-megabytes='28.0065' total-msecs='1928.56' min-msecs='0.012032' max-msecs='294.78'/>
 <counter-value subsystem='tstoragefile' counter-name='write-actual' num-operations='7621' num-successful-operations='7621' total-megabytes='28.0065' total-msecs='1903.28' min-msecs='0.00896' max-msecs='294.776'/>
 <counter-value subsystem='tstoragefile' counter-name='write-via-cache' num-operations='7621' num-successful-operations='0' total-megabytes='0' total-msecs='0' min-msecs='0' max-msecs='0'/>

Patched

app-only, cache 20MB

real 7m57.612s user 5m59.863s sys 0m14.241s

  <PFN>rfio:/castor/cern.ch/cms/store/relval/CMSSW_3_3_3/RelValZTT/GEN-SIM-RECO/STARTUP31X_V8-v1/0003/50FF8E3B-1FD2-DE11-806E-001D09F29321.root</PFN>
 <counter-value subsystem='rfio' counter-name='close' num-operations='1' num-successful-operations='1' total-megabytes='0' total-msecs='36.9211' min-msecs='36.9211' max-msecs='36.9211'/>
 <counter-value subsystem='rfio' counter-name='construct' num-operations='1' num-successful-operations='1' total-megabytes='0' total-msecs='0.002048' min-msecs='0.002048' max-msecs='0.002048'/>
 <counter-value subsystem='rfio' counter-name='destruct' num-operations='1' num-successful-operations='1' total-megabytes='0' total-msecs='0.10496' min-msecs='0.10496' max-msecs='0.10496'/>
 <counter-value subsystem='rfio' counter-name='open' num-operations='1' num-successful-operations='1' total-megabytes='0' total-msecs='29745.5' min-msecs='29745.5' max-msecs='29745.5'/>
 <counter-value subsystem='rfio' counter-name='position' num-operations='70515' num-successful-operations='70515' total-megabytes='0' total-msecs='324.715' min-msecs='0'max-msecs='0.740096'/>
 <counter-value subsystem='rfio' counter-name='prefetch' num-operations='0' num-successful-operations='0' total-megabytes='0' total-msecs='0' min-msecs='0' max-msecs='0'/>
 <counter-value subsystem='rfio' counter-name='read' num-operations='8618' num-successful-operations='8618' total-megabytes='300.085' total-msecs='21343.7' min-msecs='0.00512' max-msecs='208.051'/>
 <counter-value subsystem='rfio' counter-name='readv' num-operations='0' num-successful-operations='0' total-megabytes='0' total-msecs='0' min-msecs='0' max-msecs='0'/>
 <counter-value subsystem='rfio' counter-name='stagein' num-operations='1' num-successful-operations='1' total-megabytes='0' total-msecs='172.258' min-msecs='172.258' max-msecs='172.258'/>
 <counter-value subsystem='rfio' counter-name='write' num-operations='0' num-successful-operations='0' total-megabytes='0' total-msecs='0' min-msecs='0' max-msecs='0'/>
 <counter-value subsystem='rfio' counter-name='writev' num-operations='0' num-successful-operations='0' total-megabytes='0' total-msecs='0' min-msecs='0' max-msecs='0'/>
 <counter-value subsystem='tstoragefile' counter-name='close' num-operations='2' num-successful-operations='2' total-megabytes='0' total-msecs='37.311' min-msecs='0.244992' max-msecs='37.066'/>
 <counter-value subsystem='tstoragefile' counter-name='construct' num-operations='2' num-successful-operations='2' total-megabytes='0' total-msecs='36834.3' min-msecs='20.866' max-msecs='36813.5'/>
 <counter-value subsystem='tstoragefile' counter-name='flush' num-operations='4' num-successful-operations='4' total-megabytes='0' total-msecs='143.849' min-msecs='0.660224' max-msecs='64.3251'/>
 <counter-value subsystem='tstoragefile' counter-name='read' num-operations='28585' num-successful-operations='28585' total-megabytes='462.743' total-msecs='33055.9' min-msecs='0.009984' max-msecs='1439.67'/>
 <counter-value subsystem='tstoragefile' counter-name='read-actual' num-operations='8618' num-successful-operations='8618' total-megabytes='300.085' total-msecs='21365.7' min-msecs='0.006912' max-msecs='208.054'/>
 <counter-value subsystem='tstoragefile' counter-name='read-async' num-operations='1' num-successful-operations='0' total-megabytes='0' total-msecs='0' min-msecs='0' max-msecs='0'/>
 <counter-value subsystem='tstoragefile' counter-name='read-via-cache' num-operations='21955' num-successful-operations='19967' total-megabytes='162.658' total-msecs='10705.6' min-msecs='0.010752' max-msecs='1439.67'/>
 <counter-value subsystem='tstoragefile' counter-name='readv-actual' num-operations='11' num-successful-operations='11' total-megabytes='0' total-msecs='11162.7' min-msecs='120.806' max-msecs='1435.67'/>
 <counter-value subsystem='tstoragefile' counter-name='seek' num-operations='78128' num-successful-operations='78128' total-megabytes='0' total-msecs='543.02' min-msecs='0.001792' max-msecs='4.57318'/>
 <counter-value subsystem='tstoragefile' counter-name='stat' num-operations='2' num-successful-operations='2' total-megabytes='0' total-msecs='1.4231' min-msecs='0.648192' max-msecs='0.774912'/>
 <counter-value subsystem='tstoragefile' counter-name='write' num-operations='7621' num-successful-operations='7621' total-megabytes='28.0065' total-msecs='788.102' min-msecs='0.012032' max-msecs='72.1101'/>
 <counter-value subsystem='tstoragefile' counter-name='write-actual' num-operations='7621' num-successful-operations='7621' total-megabytes='28.0065' total-msecs='761.395' min-msecs='0.009984' max-msecs='72.107'/>
 <counter-value subsystem='tstoragefile' counter-name='write-via-cache' num-operations='7621' num-successful-operations='0' total-megabytes='0' total-msecs='0' min-msecs='0' max-msecs='0'/>

CRAB tests:

First Round

Sites: CIEMAT, DESY, KNU, Pisa, Rome

UCSD has deleted the data smile

CIEMAT

Observations (see tables and images below for details):

  • Patched version reads more data from read but no data from readv. Also, less data from cache

PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20-QCD_Pt80+Summer09.MC_31X_V3.v1+GEN.SIM.RECO-10000-CMSSW_3_3_2-200912100002/res/crab_fjr_1.xml:
         <Metric Name="tstoragefile-read-total-megabytes" Value="887.932"/>
PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20-QCD_Pt80+Summer09.MC_31X_V3.v1+GEN.SIM.RECO-10000-CMSSW_3_3_2_patched-200912100012/res/crab_fjr_1.xml:
         <Metric Name="tstoragefile-read-total-megabytes" Value="2206.66"/>
PAT1/CIEMAT-PAT1-QCD_Pt80+Summer09.MC_31X_V3.v1+GEN.SIM.RECO-10000-CMSSW_3_3_2-200912090012/res/crab_fjr_1.xml 
         <Metric Name="tstoragefile-read-total-megabytes" Value="887.932"/>


PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20-QCD_Pt80+Summer09.MC_31X_V3.v1+GEN.SIM.RECO-10000-CMSSW_3_3_2-200912100002/res/crab_fjr_1.xml:
         <Metric Name="dcap-readv-total-megabytes" Value="5801.12"/>
PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20-QCD_Pt80+Summer09.MC_31X_V3.v1+GEN.SIM.RECO-10000-CMSSW_3_3_2_patched-200912100012/res/crab_fjr_1.xml:
         <Metric Name="dcap-readv-total-megabytes" Value="0"/>

PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20-QCD_Pt80+Summer09.MC_31X_V3.v1+GEN.SIM.RECO-10000-CMSSW_3_3_2-200912100002/res/crab_fjr_1.xml:
         <Metric Name="tstoragefile-read-via-cache-total-megabytes" Value="823.815"/>
PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20-QCD_Pt80+Summer09.MC_31X_V3.v1+GEN.SIM.RECO-10000-CMSSW_3_3_2_patched-200912100012/res/crab_fjr_1.xml:
         <Metric Name="tstoragefile-read-via-cache-total-megabytes" Value="782"/>

In this case, if total-megabytes = read+readv, unpatched CMSSW read 6688.9 vs patched, which read 2206.66. Anyway, why standard CMSSW reads only 887.932?

  • Number of operations strange in patched: actual-readv greater than total-readv? But actually patched version did not read anything

PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20--CMSSW_3_3_2-200912100002/res/crab_fjr_1.xml
            <Metric Name="dcap-read-num-successful-operations" Value="10350"/>
PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20--CMSSW_3_3_2_patched-200912100012/res/crab_fjr_1.xml
            <Metric Name="dcap-read-num-successful-operations" Value="59525"/>
PAT1/CIEMAT-PAT1--CMSSW_3_3_2-200912090012/res/crab_fjr_1.xml 
         <Metric Name="dcap-read-num-successful-operations" Value="133875"/>


PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20--CMSSW_3_3_2-200912100002/res/crab_fjr_1.xml
            <Metric Name="dcap-readv-num-successful-operations" Value="278"/>
PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20--CMSSW_3_3_2_patched-200912100012/res/crab_fjr_1.xml
            <Metric Name="dcap-readv-num-successful-operations" Value="0"/>

PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20--CMSSW_3_3_2-200912100002/res/crab_fjr_1.xml
            <Metric Name="tstoragefile-readv-actual-num-successful-operations" Value="278"/>
PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20--CMSSW_3_3_2_patched-200912100012/res/crab_fjr_1.xml
            <Metric Name="tstoragefile-readv-actual-num-successful-operations" Value="42"/>

For what about reading operations in dcap, standard>patched>unpatched

in the correspondent logfile there is patch's output, so I'm confident of the right version used:

Using my custom readv impl for 1961 buffers
Number of reads performed 909

Same opened files:

PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20--CMSSW_3_3_2-200912100002/res/CMSSW_1.stderr:10-Dec-2009 00:14:13 CET  Successfully opened file dcap://dchadm.ciemat.es:22125/pnfs/0010/8051F6A9-DC80-DE11-A06C-001F29C7BDEE.root
PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20--CMSSW_3_3_2-200912100002/res/CMSSW_1.stderr:10-Dec-2009 00:19:57 CET  Successfully opened file dcap://dchadm.ciemat.es:22125/pnfs/0010/E468BDA8-E680-DE11-9146-001F29C7BDEE.root
PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20--CMSSW_3_3_2-200912100002/res/CMSSW_1.stderr:10-Dec-2009 00:25:19 CET  Successfully opened file dcap://dchadm.ciemat.es:22125/pnfs/0010/5AA4EB9C-E580-DE11-8BB4-001F29C9D4A8.root
PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20--CMSSW_3_3_2-200912100002/res/CMSSW_1.stderr:10-Dec-2009 00:30:30 CET  Successfully opened file dcap://dchadm.ciemat.es:22125/pnfs/0010/92D7AC2F-E580-DE11-9527-001F29C9E4D0.root
PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20--CMSSW_3_3_2-200912100002/res/CMSSW_1.stderr:10-Dec-2009 00:35:41 CET  Successfully opened file dcap://dchadm.ciemat.es:22125/pnfs/0010/8E5FCC4B-EE80-DE11-9B5B-001F29C79C66.root
PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20--CMSSW_3_3_2_patched-200912100012/res/CMSSW_1.stderr:10-Dec-2009 00:24:20 CET  Successfully opened file dcap://dchadm.ciemat.es:22125/pnfs/0010/8051F6A9-DC80-DE11-A06C-001F29C7BDEE.root
PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20--CMSSW_3_3_2_patched-200912100012/res/CMSSW_1.stderr:10-Dec-2009 00:30:32 CET  Successfully opened file dcap://dchadm.ciemat.es:22125/pnfs/0010/E468BDA8-E680-DE11-9146-001F29C7BDEE.root
PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20--CMSSW_3_3_2_patched-200912100012/res/CMSSW_1.stderr:10-Dec-2009 00:36:26 CET  Successfully opened file dcap://dchadm.ciemat.es:22125/pnfs/0010/5AA4EB9C-E580-DE11-8BB4-001F29C9D4A8.root
PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20--CMSSW_3_3_2_patched-200912100012/res/CMSSW_1.stderr:10-Dec-2009 00:42:15 CET  Successfully opened file dcap://dchadm.ciemat.es:22125/pnfs/0010/92D7AC2F-E580-DE11-9527-001F29C9E4D0.root
PAT1/CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20--CMSSW_3_3_2_patched-200912100012/res/CMSSW_1.stderr:10-Dec-2009 00:47:49 CET  Successfully opened file dcap://dchadm.ciemat.es:22125/pnfs/0010/8E5FCC4B-EE80-DE11-9B5B-001F29C79C66.root

Similar conclusions from JRA, where even less data is read ( 9213 vs 851 ):

JRA1/CIEMAT-JRA1_RHAUTO_CHAPP_CACHE20-QCD_Pt80+Summer09.MC_31X_V3.v1+GEN.SIM.RECO-15000-CMSSW_3_3_2-200912100112/res/crab_fjr_1.xml:
         <Metric Name="tstoragefile-read-total-megabytes" Value="413.641"/>
JRA1/CIEMAT-JRA1_RHAUTO_CHAPP_CACHE20-QCD_Pt80+Summer09.MC_31X_V3.v1+GEN.SIM.RECO-15000-CMSSW_3_3_2_patched-200912100112/res/crab_fjr_1.xml:
         <Metric Name="tstoragefile-read-total-megabytes" Value="851.546"/>
JRA1/CIEMAT-JRA1_RHAUTO_CHAPP_CACHE20-QCD_Pt80+Summer09.MC_31X_V3.v1+GEN.SIM.RECO-15000-CMSSW_3_3_2-200912100112/res/crab_fjr_1.xml:
         <Metric Name="dcap-readv-total-megabytes" Value="8800.92"/>
JRA1/CIEMAT-JRA1_RHAUTO_CHAPP_CACHE20-QCD_Pt80+Summer09.MC_31X_V3.v1+GEN.SIM.RECO-15000-CMSSW_3_3_2_patched-200912100112/res/crab_fjr_1.xml:
         <Metric Name="dcap-readv-total-megabytes" Value="0"/>

Question: patched coalescence read does not falls into readv statistics, but in read ones? for i in `ls ../../PAT1/*20091212* | grep QCD | cut -d: -f1`; do crab_getJobStatistics_2.py $i; done

  CIEMAT-JRA1 CMSSW_3_3_2 20091209 CIEMAT-JRA1_RHAUTO_CHAPP_CACHE20 CMSSW_3_3_2 20091210 CIEMAT-JRA1_RHAUTO_CHAPP_CACHE20 CMSSW_3_3_2_patched 20091210
Success 100.0% (20 / 20) 100.0% (20 / 20) 100.0% (20 / 20)
CrabCpuPercentage 46.95 +- 13.45 52.40 +- 13.27 49.35 +- 17.80
CrabSysCpuTime 22.11 +- 5.36 26.36 +- 7.17 15.80 +- 3.75
CrabUserCpuTime 379.82 +- 91.70 382.52 +- 92.65 379.26 +- 89.35
CrabWrapperTime 933.05 +- 410.51 854.05 +- 328.90 985.70 +- 546.70
ExeTime 923.15 +- 410.73 845.95 +- 328.91 977.70 +- 546.67
StageoutTime -1.00 +- 0.00 -1.00 +- 0.00 -1.00 +- 0.00
StageoutTime -1.00 +- 0.00 -1.00 +- 0.00 -1.00 +- 0.00
WrapperTime 933.05 +- 410.51 854.05 +- 328.90 985.70 +- 546.70
OPEN
dcap-open-num-successful-operations 7.20 +- 1.72 7.20 +- 1.72 7.20 +- 1.72
dcap-open-num-operations 7.20 +- 1.72 7.20 +- 1.72 7.20 +- 1.72
dcap-open-total-msecs 709.48 +- 1092.91 401.93 +- 311.94 261.14 +- 64.53
dcap-open-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
READ
dcap-read-num-successful-operations 167939.90 +- 41688.94 487.60 +- 118.89 44958.50 +- 9947.28
tstoragefile-read-actual-num-successful-operations 167939.90 +- 41688.94 487.60 +- 118.89 44958.50 +- 9947.28
tstoragefile-read-async-num-successful-operations // 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-num-successful-operations 167939.90 +- 41688.94 167939.90 +- 41688.94 199265.75 +- 50083.28
tstoragefile-read-via-cache-num-successful-operations // 167452.30 +- 41570.29 154307.25 +- 41369.40
dcap-read-num-operations 167939.90 +- 41688.94 487.60 +- 118.89 44958.50 +- 9947.28
tstoragefile-read-actual-num-operations 167939.90 +- 41688.94 487.60 +- 118.89 44958.50 +- 9947.28
tstoragefile-read-async-num-operations // 7.20 +- 1.72 7.20 +- 1.72
tstoragefile-read-num-operations 167939.90 +- 41688.94 167939.90 +- 41688.94 199265.75 +- 50083.28
tstoragefile-read-via-cache-num-operations // 167599.90 +- 41606.79 167599.90 +- 41606.79
dcap-read-total-msecs 524731.72 +- 381653.60 7538.77 +- 5281.07 585896.02 +- 534415.49
tstoragefile-read-actual-total-msecs 524900.68 +- 381661.78 7539.59 +- 5281.15 585951.37 +- 534413.43
tstoragefile-read-async-total-msecs // 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-total-msecs 525352.87 +- 381683.74 431667.93 +- 260231.12 1118167.86 +- 1033962.19
tstoragefile-read-via-cache-total-msecs // 261860.84 +- 159376.23 531968.84 +- 499547.15
dcap-read-total-megabytes 381.61 +- 94.39 17.56 +- 4.14 440.60 +- 110.78
tfile-read-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 -53.68 +- 14.44
tstoragefile-read-actual-total-megabytes 381.61 +- 94.39 17.56 +- 4.14 440.60 +- 110.78
tstoragefile-read-async-total-megabytes // 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-total-megabytes 381.61 +- 94.39 381.61 +- 94.39 776.16 +- 200.45
tstoragefile-read-via-cache-total-megabytes // 364.05 +- 90.37 335.56 +- 90.01
READV
dcap-readv-num-successful-operations 0.00 +- 0.00 382.55 +- 94.50 0.00 +- 0.00
tstoragefile-readv-actual-num-successful-operations // 382.55 +- 94.50 19.55 +- 4.99
dcap-readv-num-operations 0.00 +- 0.00 382.55 +- 94.50 0.00 +- 0.00
tstoragefile-readv-actual-num-operations // 382.55 +- 94.50 19.55 +- 4.99
dcap-readv-total-msecs 0.00 +- 0.00 443930.80 +- 274228.34 0.00 +- 0.00
tstoragefile-readv-actual-total-msecs // 443932.06 +- 274228.37 531060.23 +- 499587.99
dcap-readv-total-megabytes 0.00 +- 0.00 8044.66 +- 1992.83 0.00 +- 0.00
tstoragefile-readv-actual-total-megabytes // 8044.66 +- 1992.83 0.00 +- 0.00
SEEK
tstoragefile-seek-num-successful-operations 167939.90 +- 41688.94 502992.10 +- 124866.02 521172.90 +- 132863.60
tstoragefile-seek-num-operations 167939.90 +- 41688.94 502992.10 +- 124866.02 521172.90 +- 132863.60
tstoragefile-seek-total-msecs 549.82 +- 136.33 1136.91 +- 441.22 1095.08 +- 325.02
tstoragefile-seek-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00

  CIEMAT-PAT1 CMSSW_3_3_2 20091209 CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20 CMSSW_3_3_2 20091210 CIEMAT-PAT1_RHAUTO_CHAPP_CACHE20 CMSSW_3_3_2_patched 20091210
Success 100.0% (20 / 20) 100.0% (20 / 20) 100.0% (20 / 20)
CrabCpuPercentage 69.25 +- 10.02 85.05 +- 4.94 82.20 +- 9.14
CrabSysCpuTime 24.23 +- 6.07 22.20 +- 4.92 18.04 +- 3.65
CrabUserCpuTime 1410.01 +- 301.23 1301.10 +- 274.28 1324.93 +- 285.60
CrabWrapperTime 2075.50 +- 514.52 1551.45 +- 329.82 1636.55 +- 422.12
ExeTime 2064.80 +- 514.35 1543.00 +- 329.18 1626.70 +- 421.49
StageoutTime -1.00 +- 0.00 -1.00 +- 0.00 -1.00 +- 0.00
StageoutTime -1.00 +- 0.00 -1.00 +- 0.00 -1.00 +- 0.00
WrapperTime 2075.50 +- 514.52 1551.45 +- 329.82 1636.55 +- 422.12
OPEN
dcap-open-num-successful-operations 5.30 +- 1.10 5.30 +- 1.10 5.30 +- 1.10
file-open-num-successful-operations 1.00 +- 0.00 1.00 +- 0.00 1.00 +- 0.00
dcap-open-num-operations 5.30 +- 1.10 5.30 +- 1.10 5.30 +- 1.10
file-open-num-operations 1.00 +- 0.00 1.00 +- 0.00 1.00 +- 0.00
dcap-open-total-msecs 318.68 +- 198.81 573.54 +- 1084.55 1208.24 +- 1995.41
file-open-total-msecs 0.17 +- 0.03 0.14 +- 0.03 0.16 +- 0.03
dcap-open-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
file-open-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
READ
dcap-read-num-successful-operations 127473.15 +- 26728.12 9854.30 +- 2063.79 62810.05 +- 12763.84
file-read-num-successful-operations 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-actual-num-successful-operations 127473.15 +- 26728.12 9854.30 +- 2063.79 62810.05 +- 12763.84
tstoragefile-read-async-num-successful-operations // 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-num-successful-operations 127473.15 +- 26728.12 127473.15 +- 26728.12 164129.80 +- 35251.38
tstoragefile-read-via-cache-num-successful-operations // 117618.85 +- 24664.45 101319.75 +- 24396.43
dcap-read-num-operations 127473.15 +- 26728.12 9854.30 +- 2063.79 62810.05 +- 12763.84
file-read-num-operations 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-actual-num-operations 127473.15 +- 26728.12 9854.30 +- 2063.79 62810.05 +- 12763.84
tstoragefile-read-async-num-operations // 5.30 +- 1.10 5.30 +- 1.10
tstoragefile-read-num-operations 127473.15 +- 26728.12 127473.15 +- 26728.12 164129.80 +- 35251.38
tstoragefile-read-via-cache-num-operations // 117721.75 +- 24685.83 117721.75 +- 24685.83
dcap-read-total-msecs 631151.80 +- 335548.55 22837.94 +- 5844.38 277248.28 +- 253618.98
file-read-total-msecs 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-actual-total-msecs 631390.15 +- 335562.15 22854.37 +- 5846.76 277354.88 +- 253628.45
tstoragefile-read-async-total-msecs // 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-total-msecs 631654.39 +- 335623.66 199784.36 +- 98284.49 457056.52 +- 496925.98
tstoragefile-read-via-cache-total-msecs // 109057.91 +- 59834.11 179360.70 +- 244842.39
dcap-read-total-megabytes 848.64 +- 177.67 62.16 +- 12.79 1308.66 +- 285.91
file-read-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
tfile-read-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 -414.38 +- 100.36
tstoragefile-read-actual-total-megabytes 848.64 +- 177.67 62.16 +- 12.79 1308.66 +- 285.91
tstoragefile-read-async-total-megabytes // 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-total-megabytes 848.64 +- 177.67 848.64 +- 177.67 1988.43 +- 447.83
tstoragefile-read-via-cache-total-megabytes // 786.48 +- 164.93 679.78 +- 163.69
READV
dcap-readv-num-successful-operations 0.00 +- 0.00 265.70 +- 55.37 0.00 +- 0.00
file-readv-num-successful-operations 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-readv-actual-num-successful-operations // 265.70 +- 55.37 37.90 +- 8.89
dcap-readv-num-operations 0.00 +- 0.00 265.70 +- 55.37 0.00 +- 0.00
file-readv-num-operations 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-readv-actual-num-operations // 265.70 +- 55.37 37.90 +- 8.89
dcap-readv-total-msecs 0.00 +- 0.00 181847.67 +- 97483.75 0.00 +- 0.00
file-readv-total-msecs 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-readv-actual-total-msecs // 181848.63 +- 97483.83 178288.59 +- 244695.22
dcap-readv-total-megabytes 0.00 +- 0.00 5539.61 +- 1156.86 0.00 +- 0.00
file-readv-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-readv-actual-total-megabytes // 5539.61 +- 1156.86 0.00 +- 0.00
SEEK
tstoragefile-seek-num-successful-operations 127736.30 +- 26776.44 363076.90 +- 76126.71 383434.45 +- 83744.30
tstoragefile-seek-num-operations 127736.30 +- 26776.44 363076.90 +- 76126.71 383434.45 +- 83744.30
tstoragefile-seek-total-msecs 609.22 +- 133.62 923.87 +- 299.08 1148.98 +- 382.96
tstoragefile-seek-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00

DESY

  • For JRA, cannot tell much more, some jobs aborted:

  DESY-JRA1 CMSSW_3_3_2 20091209 DESY-JRA1_RHAUTO_CHAPP_CACHE20 CMSSW_3_3_2 20091210 DESY-JRA1_RHAUTO_CHAPP_CACHE20 CMSSW_3_3_2_patched 20091210
Success 100.0% (20 / 20) 100.0% (14 / 14) 100.0% (8 / 8)
CrabCpuPercentage 37.00 +- 9.51 40.64 +- 5.71 50.00 +- 8.14
CrabSysCpuTime 26.00 +- 8.51 30.17 +- 9.76 18.95 +- 3.09
CrabUserCpuTime 459.88 +- 114.14 432.93 +- 131.22 473.84 +- 59.23
CrabWrapperTime 1444.30 +- 566.48 1181.14 +- 418.48 1020.62 +- 224.33
ExeTime 1433.00 +- 567.13 1169.64 +- 418.11 1008.62 +- 223.53

  • For PAT, full statistics. Note that standard test has been performed on a different day, so actual differences in CpuPerf are greater. Thi can be seen also from the Overview plots:

DESY-PAT-PatchVsStd-Overview.jpg

  DESY-PAT1 CMSSW_3_3_2 20091209 DESY-PAT1_RHAUTO_CHAPP_CACHE20 CMSSW_3_3_2 20091210 DESY-PAT1_RHAUTO_CHAPP_CACHE20 CMSSW_3_3_2_patched 20091210
Success 100.0% (20 / 20) 100.0% (20 / 20) 100.0% (20 / 20)
CrabCpuPercentage 62.35 +- 10.68 69.50 +- 4.95 74.70 +- 7.18
CrabSysCpuTime 25.44 +- 5.78 33.21 +- 9.21 22.93 +- 5.45
CrabUserCpuTime 1518.81 +- 320.71 1618.00 +- 330.25 1616.26 +- 340.54
CrabWrapperTime 2575.30 +- 740.51 2385.05 +- 513.02 2213.10 +- 510.22
OPEN
file-open-num-successful-operations 1.00 +- 0.00 1.00 +- 0.00 1.00 +- 0.00
gsidcap-open-num-successful-operations 5.30 +- 1.10 5.30 +- 1.10 5.30 +- 1.10
file-open-num-operations 1.00 +- 0.00 1.00 +- 0.00 1.00 +- 0.00
gsidcap-open-num-operations 5.30 +- 1.10 5.30 +- 1.10 5.30 +- 1.10
file-open-total-msecs 0.19 +- 0.01 0.15 +- 0.02 0.19 +- 0.01
gsidcap-open-total-msecs 1221.82 +- 2120.00 1761.46 +- 2209.34 651.83 +- 1014.96
file-open-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
gsidcap-open-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
READ
file-read-num-successful-operations 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
gsidcap-read-num-successful-operations 127473.15 +- 26728.12 9854.30 +- 2063.79 62810.05 +- 12763.84
tstoragefile-read-actual-num-successful-operations 127473.15 +- 26728.12 9854.30 +- 2063.79 62810.05 +- 12763.84
tstoragefile-read-async-num-successful-operations // 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-num-successful-operations 127473.15 +- 26728.12 127473.15 +- 26728.12 164129.80 +- 35251.38
tstoragefile-read-via-cache-num-successful-operations // 117618.85 +- 24664.45 101319.75 +- 24396.43
file-read-num-operations 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
gsidcap-read-num-operations 127473.15 +- 26728.12 9854.30 +- 2063.79 62810.05 +- 12763.84
tstoragefile-read-actual-num-operations 127473.15 +- 26728.12 9854.30 +- 2063.79 62810.05 +- 12763.84
tstoragefile-read-async-num-operations // 5.30 +- 1.10 5.30 +- 1.10
tstoragefile-read-num-operations 127473.15 +- 26728.12 127473.15 +- 26728.12 164129.80 +- 35251.38
tstoragefile-read-via-cache-num-operations // 117721.75 +- 24685.83 117721.75 +- 24685.83
file-read-total-msecs 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
gsidcap-read-total-msecs 1027560.58 +- 541833.40 43650.38 +- 14315.92 546842.93 +- 267359.70
tstoragefile-read-actual-total-msecs 1027830.42 +- 541863.90 43668.26 +- 14317.64 546983.03 +- 267369.32
tstoragefile-read-async-total-msecs // 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-total-msecs 1028177.75 +- 541979.00 707557.88 +- 216727.09 883638.25 +- 407277.32
tstoragefile-read-via-cache-total-msecs // 391303.59 +- 119388.78 336162.03 +- 155298.26
file-read-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
gsidcap-read-total-megabytes 848.64 +- 177.67 62.16 +- 12.79 1308.66 +- 285.91
tfile-read-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 -414.38 +- 100.36
tstoragefile-read-actual-total-megabytes 848.64 +- 177.67 62.16 +- 12.79 1308.66 +- 285.91
tstoragefile-read-async-total-megabytes // 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-total-megabytes 848.64 +- 177.67 848.64 +- 177.67 1988.43 +- 447.83
tstoragefile-read-via-cache-total-megabytes // 786.48 +- 164.93 679.78 +- 163.69
READV
file-readv-num-successful-operations 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
gsidcap-readv-num-successful-operations 0.00 +- 0.00 265.70 +- 55.37 0.00 +- 0.00
tstoragefile-readv-actual-num-successful-operations // 265.70 +- 55.37 37.90 +- 8.89
file-readv-num-operations 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
gsidcap-readv-num-operations 0.00 +- 0.00 265.70 +- 55.37 0.00 +- 0.00
tstoragefile-readv-actual-num-operations // 265.70 +- 55.37 37.90 +- 8.89
file-readv-total-msecs 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
gsidcap-readv-total-msecs 0.00 +- 0.00 686676.62 +- 210195.01 0.00 +- 0.00
tstoragefile-readv-actual-total-msecs // 686677.62 +- 210195.14 334639.96 +- 155176.47
file-readv-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
gsidcap-readv-total-megabytes 0.00 +- 0.00 5539.61 +- 1156.86 0.00 +- 0.00
tstoragefile-readv-actual-total-megabytes // 5539.61 +- 1156.86 0.00 +- 0.00
SEEK
tstoragefile-seek-num-successful-operations 127736.30 +- 26776.44 363076.90 +- 76126.71 383434.45 +- 83744.30
tstoragefile-seek-num-operations 127736.30 +- 26776.44 363076.90 +- 76126.71 383434.45 +- 83744.30
tstoragefile-seek-total-msecs 699.11 +- 150.48 1037.44 +- 262.53 1569.04 +- 365.44
tstoragefile-seek-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00

KNU

The passage standard->unpatched->patched improve performances for PAT, while not significant results for JRA.

  KNU-JRA1 CMSSW_3_3_2 20091209 KNU-JRA1_RHAUTO_CHAPP_CACHE20 CMSSW_3_3_2 20091210 KNU-JRA1_RHAUTO_CHAPP_CACHE20 CMSSW_3_3_2_patched 20091210
Success 100.0% (20 / 20) 100.0% (20 / 20) 100.0% (20 / 20)
CrabCpuPercentage 55.25 +- 3.48 49.75 +- 5.07 59.45 +- 6.28
CrabSysCpuTime 69.20 +- 17.07 87.11 +- 22.58 45.33 +- 11.79
CrabUserCpuTime 770.44 +- 205.50 785.14 +- 190.51 792.86 +- 196.20
CrabWrapperTime 1532.85 +- 385.40 1788.65 +- 451.83 1423.50 +- 358.22
ExeTime 1520.20 +- 385.31 1775.65 +- 451.68 1411.10 +- 358.04
StageoutTime -1.00 +- 0.00 -1.00 +- 0.00 -1.00 +- 0.00
StageoutTime -1.00 +- 0.00 -1.00 +- 0.00 -1.00 +- 0.00
WrapperTime 1532.85 +- 385.40 1788.65 +- 451.83 1423.50 +- 358.22
OPEN
dcap-open-num-successful-operations 7.20 +- 1.72 7.20 +- 1.72 7.20 +- 1.72
dcap-open-num-operations 7.20 +- 1.72 7.20 +- 1.72 7.20 +- 1.72
dcap-open-total-msecs 6266.72 +- 1814.26 5133.48 +- 2092.33 4684.86 +- 1325.55
dcap-open-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
READ
dcap-read-num-successful-operations 167939.90 +- 41688.94 487.60 +- 118.89 44958.50 +- 9947.28
tstoragefile-read-actual-num-successful-operations 167939.90 +- 41688.94 487.60 +- 118.89 44958.50 +- 9947.28
tstoragefile-read-async-num-successful-operations // 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-num-successful-operations 167939.90 +- 41688.94 167939.90 +- 41688.94 199265.75 +- 50083.28
tstoragefile-read-via-cache-num-successful-operations // 167452.30 +- 41570.29 154307.25 +- 41369.40
dcap-read-num-operations 167939.90 +- 41688.94 487.60 +- 118.89 44958.50 +- 9947.28
tstoragefile-read-actual-num-operations 167939.90 +- 41688.94 487.60 +- 118.89 44958.50 +- 9947.28
tstoragefile-read-async-num-operations // 7.20 +- 1.72 7.20 +- 1.72
tstoragefile-read-num-operations 167939.90 +- 41688.94 167939.90 +- 41688.94 199265.75 +- 50083.28
tstoragefile-read-via-cache-num-operations // 167599.90 +- 41606.79 167599.90 +- 41606.79
dcap-read-total-msecs 679871.14 +- 174818.29 10332.25 +- 2872.63 562335.92 +- 182207.85
tstoragefile-read-actual-total-msecs 680693.32 +- 175021.31 10334.96 +- 2873.29 562554.86 +- 182246.81
tstoragefile-read-async-total-msecs // 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-total-msecs 681507.08 +- 175174.70 922777.51 +- 252795.83 1014521.10 +- 334707.68
tstoragefile-read-via-cache-total-msecs // 563495.93 +- 153047.97 450896.89 +- 156616.37
dcap-read-total-megabytes 381.61 +- 94.39 17.56 +- 4.14 440.60 +- 110.78
tfile-read-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 -53.68 +- 14.44
tstoragefile-read-actual-total-megabytes 381.61 +- 94.39 17.56 +- 4.14 440.60 +- 110.78
tstoragefile-read-async-total-megabytes // 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-total-megabytes 381.61 +- 94.39 381.61 +- 94.39 776.16 +- 200.45
tstoragefile-read-via-cache-total-megabytes // 364.05 +- 90.37 335.56 +- 90.01
READV
dcap-readv-num-successful-operations 0.00 +- 0.00 382.55 +- 94.50 0.00 +- 0.00
tstoragefile-readv-actual-num-successful-operations // 382.55 +- 94.50 19.55 +- 4.99
dcap-readv-num-operations 0.00 +- 0.00 382.55 +- 94.50 0.00 +- 0.00
tstoragefile-readv-actual-num-operations // 382.55 +- 94.50 19.55 +- 4.99
dcap-readv-total-msecs 0.00 +- 0.00 924098.31 +- 253256.14 0.00 +- 0.00
tstoragefile-readv-actual-total-msecs // 924102.07 +- 253257.23 447184.66 +- 155855.96
dcap-readv-total-megabytes 0.00 +- 0.00 8044.66 +- 1992.83 0.00 +- 0.00
tstoragefile-readv-actual-total-megabytes // 8044.66 +- 1992.83 0.00 +- 0.00
SEEK
tstoragefile-seek-num-successful-operations 167939.90 +- 41688.94 502992.10 +- 124866.02 521172.90 +- 132863.60
tstoragefile-seek-num-operations 167939.90 +- 41688.94 502992.10 +- 124866.02 521172.90 +- 132863.60
tstoragefile-seek-total-msecs 1997.94 +- 516.86 4032.49 +- 1002.43 4247.09 +- 1075.59
tstoragefile-seek-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00

  KNU-PAT1 CMSSW_3_3_2 20091209 KNU-PAT1_RHAUTO_CHAPP_CACHE20 CMSSW_3_3_2 20091210 KNU-PAT1_RHAUTO_CHAPP_CACHE20 CMSSW_3_3_2_patched 20091210
Success 100.0% (20 / 20) 100.0% (20 / 20) 100.0% (20 / 20)
CrabCpuPercentage 70.00 +- 6.33 78.60 +- 7.78 88.25 +- 1.22
CrabSysCpuTime 76.84 +- 17.56 80.66 +- 19.43 52.74 +- 10.97
CrabUserCpuTime 2738.00 +- 569.52 2679.40 +- 553.50 2595.25 +- 564.85
CrabWrapperTime 3948.30 +- 767.41 3434.85 +- 639.30 2994.40 +- 645.29
ExeTime 3936.65 +- 767.55 3423.05 +- 638.87 2983.50 +- 645.20
StageoutTime -1.00 +- 0.00 -1.00 +- 0.00 -1.00 +- 0.00
StageoutTime -1.00 +- 0.00 -1.00 +- 0.00 -1.00 +- 0.00
WrapperTime 3948.30 +- 767.41 3434.85 +- 639.30 2994.40 +- 645.29
OPEN
dcap-open-num-successful-operations 5.30 +- 1.10 5.30 +- 1.10 5.30 +- 1.10
file-open-num-successful-operations 1.00 +- 0.00 1.00 +- 0.00 1.00 +- 0.00
dcap-open-num-operations 5.30 +- 1.10 5.30 +- 1.10 5.30 +- 1.10
file-open-num-operations 1.00 +- 0.00 1.00 +- 0.00 1.00 +- 0.00
dcap-open-total-msecs 4794.81 +- 1773.35 7268.24 +- 2505.35 5261.60 +- 1864.49
file-open-total-msecs 0.27 +- 0.02 0.27 +- 0.02 0.37 +- 0.37
dcap-open-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
file-open-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
READ
dcap-read-num-successful-operations 127473.15 +- 26728.12 9854.30 +- 2063.79 62810.05 +- 12763.84
file-read-num-successful-operations 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-actual-num-successful-operations 127473.15 +- 26728.12 9854.30 +- 2063.79 62810.05 +- 12763.84
tstoragefile-read-async-num-successful-operations // 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-num-successful-operations 127473.15 +- 26728.12 127473.15 +- 26728.12 164129.80 +- 35251.38
tstoragefile-read-via-cache-num-successful-operations // 117618.85 +- 24664.45 101319.75 +- 24396.43
dcap-read-num-operations 127473.15 +- 26728.12 9854.30 +- 2063.79 62810.05 +- 12763.84
file-read-num-operations 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-actual-num-operations 127473.15 +- 26728.12 9854.30 +- 2063.79 62810.05 +- 12763.84
tstoragefile-read-async-num-operations // 5.30 +- 1.10 5.30 +- 1.10
tstoragefile-read-num-operations 127473.15 +- 26728.12 127473.15 +- 26728.12 164129.80 +- 35251.38
tstoragefile-read-via-cache-num-operations // 117721.75 +- 24685.83 117721.75 +- 24685.83
dcap-read-total-msecs 802135.07 +- 223131.22 51779.57 +- 14581.39 309617.25 +- 78664.87
file-read-total-msecs 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-actual-total-msecs 802795.84 +- 223239.99 51831.94 +- 14590.36 309927.37 +- 78723.77
tstoragefile-read-async-total-msecs // 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-total-msecs 803432.34 +- 223281.19 352964.86 +- 84835.11 487080.86 +- 120601.05
tstoragefile-read-via-cache-total-msecs // 186862.06 +- 44728.84 176198.05 +- 48736.72
dcap-read-total-megabytes 848.64 +- 177.67 62.16 +- 12.79 1308.66 +- 285.91
file-read-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
tfile-read-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 -414.38 +- 100.36
tstoragefile-read-actual-total-megabytes 848.64 +- 177.67 62.16 +- 12.79 1308.66 +- 285.91
tstoragefile-read-async-total-megabytes // 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-read-total-megabytes 848.64 +- 177.67 848.64 +- 177.67 1988.43 +- 447.83
tstoragefile-read-via-cache-total-megabytes // 786.48 +- 164.93 679.78 +- 163.69
READV
dcap-readv-num-successful-operations 0.00 +- 0.00 265.70 +- 55.37 0.00 +- 0.00
file-readv-num-successful-operations 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-readv-actual-num-successful-operations // 265.70 +- 55.37 37.90 +- 8.89
dcap-readv-num-operations 0.00 +- 0.00 265.70 +- 55.37 0.00 +- 0.00
file-readv-num-operations 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-readv-actual-num-operations // 265.70 +- 55.37 37.90 +- 8.89
dcap-readv-total-msecs 0.00 +- 0.00 303042.82 +- 72083.43 0.00 +- 0.00
file-readv-total-msecs 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-readv-actual-total-msecs // 303045.08 +- 72083.82 173213.82 +- 48143.91
dcap-readv-total-megabytes 0.00 +- 0.00 5539.61 +- 1156.86 0.00 +- 0.00
file-readv-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
tstoragefile-readv-actual-total-megabytes // 5539.61 +- 1156.86 0.00 +- 0.00
SEEK
tstoragefile-seek-num-successful-operations 127736.30 +- 26776.44 363076.90 +- 76126.71 383434.45 +- 83744.30
tstoragefile-seek-num-operations 127736.30 +- 26776.44 363076.90 +- 76126.71 383434.45 +- 83744.30
tstoragefile-seek-total-msecs 1895.39 +- 404.45 3335.72 +- 706.47 3422.28 +- 739.64
tstoragefile-seek-total-megabytes 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00

Second round

Sites:

  • Brunel (dpm/rfio)
  • KNU (dCache)
  • Nebraska (bestman)
  • Pisa (dCache)
  • Wisconsin (bestman)

Dataset: /QCD_Pt80/Summer09-MC_31X_V3_AODSIM-v1/AODSIM

Commands


for i in `ls ../../PAT1/*20091212* | grep QCD | cut -d: -f1`; do crab_getJobStatistics_2.py $i; done
crab_cfrSamples_2.py KNU*

NOTES

  • User_ReadkBJob = 1024*(tstoragefile-read-total-megabytes + (.*)-readv-total-megabytes)/NJobs does it make sense? If this holds,
then patched CMSSW actually reads less than unpatched in the application-only settings, but still reads twice than standard settings. Need to find overlaps.

  • User_Wall-Read_CpuPercentage = (CrabWrapperTime - tstoragefile-read-total-msecs)/CrabWrapperTime

  • 100% error patched at Brunel - castor issue? Seems so (checked in two differend days):

ms::Exception caught in cmsRun
---- EventProcessorFailure BEGIN
EventProcessingStopped
---- ScheduleExecutionFailure BEGIN
ProcessingStopped
---- FileReadError BEGIN
---- FatalRootError BEGIN
Fatal Root Error: @SUB=TBasket::ReadBasketBuffers
basket: has fNevBuf=0 but fEntryOffset=0, pos=219806488, len=4338, fNbytes=-1, fObjlen=4352, trying to repair
---- FatalRootError END

  • More seek operations in application-only and lazy-download wrt standard settings

  KNU-PAT1 CMSSW_3_3_2 20091212 KNU-PAT1_RHAUTO_CHAPP_CACHE20 CMSSW_3_3_2 20091212 KNU-PAT1_RHAUTO_CHAPP_CACHE20 CMSSW_3_3_2_patched 20091212 KNU-PAT1_RHAUTO_CHLD_CACHE20 CMSSW_3_3_2 20091212
Success 100.0% (20 / 20) 100.0% (20 / 20) 100.0% (20 / 20) 100.0% (20 / 20)
WrapperTime 3117.50 +- 129.18 2428.50 +- 227.38 2739.50 +- 291.32 2529.50 +- 286.07
ExeTime 3107.00 +- 129.79 2418.00 +- 228.06 2729.00 +- 289.33 2518.00 +- 285.38
UserTime 2399.50 +- 135.77 2270.00 +- 220.12 2424.50 +- 254.82 2404.00 +- 246.82
CpuPercentage 79.70 +- 3.30 95.60 +- 0.70 90.95 +- 7.81 97.05 +- 2.40
User_ReadkBEvt 89.95 +- 0.50 325.85 +- 2.10 169.15 +- 23.00 89.95 +- 0.50
User_Wall-Read_CpuPercentage 83.05 +- 3.07 92.65 +- 1.01 89.95 +- 9.71 97.70 +- 2.06
StageoutTime 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
SysTime 74.50 +- 7.40 40.00 +- 5.92 42.50 +- 8.29 36.50 +- 6.54
CACHE
local-cache-read-total-msecs // // // 29400.00 +- 32555.18
local-cache-readv-total-msecs // // // 3000.00 +- 0.00
local-cache-read-num-operations // // // 99995.00 +- 0.00
local-cache-readv-num-successful-operations // // // 0.50 +- 0.00
local-cache-read-total-megabytes // // // 878.65 +- 2.52
local-cache-readv-total-megabytes // // // 0.50 +- 0.00
local-cache-read-num-successful-operations // // // 9999.50 +- 0.00
local-cache-readv-num-operations // // // 5.00 +- 0.00
READ
tstoragefile-read-prefetch-to-cache-total-msecs // // // 30900.00 +- 31479.99
tstoragefile-read-prefetch-to-cache-num-successful-operations // // // 9999.50 +- 0.00
tstoragefile-read-actual-num-successful-operations 9999.50 +- 0.00 9999.50 +- 0.00 9999.50 +- 0.00 9999.50 +- 0.00
tstoragefile-read-num-operations 99995.00 +- 0.00 99995.00 +- 0.00 99995.00 +- 0.00 99995.00 +- 0.00
dcap-read-num-operations 99995.00 +- 0.00 10309.50 +- 5.89 72975.00 +- 18127.74 15.00 +- 0.00
file-read-num-operations 5.00 +- 0.00 5.00 +- 0.00 5.00 +- 0.00 5.00 +- 0.00
tstoragefile-read-async-total-msecs // 3000.00 +- 0.00 3000.00 +- 0.00 29700.00 +- 31088.74
tstoragefile-read-actual-num-operations 99995.00 +- 0.00 10309.50 +- 5.89 72975.00 +- 18127.74 99995.00 +- 0.00
tstoragefile-read-via-cache-total-msecs // 3000.00 +- 0.00 48000.00 +- 75215.69 //
tstoragefile-read-async-num-successful-operations // 0.50 +- 0.00 0.50 +- 0.00 9999.50 +- 0.00
tstoragefile-read-via-cache-total-megabytes // 828.75 +- 2.70 517.05 +- 150.67 //
tstoragefile-read-prefetch-to-cache-total-megabytes // // // 828.75 +- 2.70
tstoragefile-read-actual-total-msecs 526800.00 +- 107047.47 19800.00 +- 5878.78 243300.00 +- 257507.50 30000.00 +- 32338.83
tstoragefile-read-prefetch-to-cache-num-operations // // // 99995.00 +- 0.00
tstoragefile-read-total-megabytes 878.65 +- 2.52 878.65 +- 2.52 1652.00 +- 224.53 878.65 +- 2.52
tfile-read-total-megabytes 0.50 +- 0.00 0.50 +- 0.00 0.00 +- 0.00 0.50 +- 0.00
dcap-read-total-megabytes 878.65 +- 2.52 50.00 +- 1.50 1134.90 +- 73.97 1809.10 +- 85.05
file-read-total-megabytes 0.50 +- 0.00 0.50 +- 0.00 0.50 +- 0.00 0.50 +- 0.00
tstoragefile-read-async-total-megabytes // 0.50 +- 0.00 0.50 +- 0.00 2188.35 +- 13.31
tstoragefile-read-total-msecs 527400.00 +- 107823.19 100500.00 +- 18383.42 292200.00 +- 324475.21 63300.00 +- 63369.63
file-read-total-msecs 3000.00 +- 0.00 3000.00 +- 0.00 3000.00 +- 0.00 3000.00 +- 0.00
dcap-read-total-msecs 526200.00 +- 107190.30 19800.00 +- 5878.78 243000.00 +- 257182.43 54000.00 +- 56460.61
tstoragefile-read-num-successful-operations 9999.50 +- 0.00 9999.50 +- 0.00 9999.50 +- 0.00 9999.50 +- 0.00
file-read-num-successful-operations 0.50 +- 0.00 0.50 +- 0.00 0.50 +- 0.00 0.50 +- 0.00
dcap-read-num-successful-operations 9999.50 +- 0.00 9999.50 +- 0.00 9999.50 +- 0.00 15.75 +- 0.77
tstoragefile-read-via-cache-num-successful-operations // 9999.50 +- 0.00 9999.50 +- 0.00 //
tstoragefile-read-via-cache-num-operations // 99995.00 +- 0.00 99995.00 +- 0.00 //
tstoragefile-read-async-num-operations // 5.00 +- 0.00 5.00 +- 0.00 22622.00 +- 537.33
tstoragefile-read-actual-total-megabytes 878.65 +- 2.52 50.00 +- 1.50 1134.90 +- 73.97 878.65 +- 2.52
READV
tstoragefile-readv-actual-num-successful-operations // 111.90 +- 0.86 29.25 +- 8.07 //
tstoragefile-readv-actual-total-megabytes // 2303.30 +- 20.74 0.50 +- 0.00 //
file-readv-total-msecs 3000.00 +- 0.00 3000.00 +- 0.00 3000.00 +- 0.00 3000.00 +- 0.00
dcap-readv-total-msecs 3000.00 +- 0.00 77100.00 +- 11738.40 3000.00 +- 0.00 3000.00 +- 0.00
tstoragefile-readv-actual-total-msecs // 77100.00 +- 11738.40 46800.00 +- 75110.32 //
dcap-readv-num-successful-operations 0.50 +- 0.00 111.90 +- 0.86 0.50 +- 0.00 0.50 +- 0.00
file-readv-num-successful-operations 0.50 +- 0.00 0.50 +- 0.00 0.50 +- 0.00 0.50 +- 0.00
file-readv-total-megabytes 0.50 +- 0.00 0.50 +- 0.00 0.50 +- 0.00 0.50 +- 0.00
dcap-readv-total-megabytes 0.50 +- 0.00 2303.30 +- 20.74 0.50 +- 0.00 0.50 +- 0.00
file-readv-num-operations 5.00 +- 0.00 5.00 +- 0.00 5.00 +- 0.00 5.00 +- 0.00
dcap-readv-num-operations 5.00 +- 0.00 115.00 +- 0.00 5.00 +- 0.00 5.00 +- 0.00
tstoragefile-readv-actual-num-operations // 115.00 +- 0.00 29.50 +- 8.05 //

  Nebraska-PAT1 CMSSW_3_3_2 20091212 Nebraska-PAT1_RHAUTO_CHAPP_CACHE20 CMSSW_3_3_2 20091212 Nebraska-PAT1_RHAUTO_CHAPP_CACHE20 CMSSW_3_3_2_patched 20091212 Nebraska-PAT1_RHAUTO_CHLD_CACHE20 CMSSW_3_3_2 20091212
Success 100.0% (20 / 20) 100.0% (20 / 20) 100.0% (20 / 20) 100.0% (20 / 20)
WrapperTime 3055.50 +- 301.01 2638.00 +- 366.93 2934.00 +- 374.67 2937.00 +- 381.07
ExeTime 2975.00 +- 228.41 2578.00 +- 291.96 2832.00 +- 304.40 2814.50 +- 301.75
UserTime 2529.50 +- 228.05 2390.00 +- 180.93 2526.50 +- 181.45 2624.50 +- 218.35
CpuPercentage 85.75 +- 2.09 93.95 +- 5.46 90.55 +- 5.62 94.45 +- 4.26
User_ReadkBEvt 89.95 +- 0.50 325.85 +- 2.10 169.15 +- 23.00 89.95 +- 0.50
User_Wall-Read_CpuPercentage 87.30 +- 1.86 93.90 +- 1.85 92.55 +- 1.83 97.80 +- 0.90
StageoutTime 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00 0.00 +- 0.00
SysTime 23.00 +- 6.00 19.00 +- 4.90 23.00 +- 4.00 22.50 +- 4.33
READ
tstoragefile-read-prefetch-to-cache-total-msecs // // // 36600.00 +- 5161.40
tstoragefile-read-prefetch-to-cache-num-successful-operations // // // 9999.50 +- 0.00
tstoragefile-read-actual-num-successful-operations 9999.50 +- 0.00 9999.50 +- 0.00 9999.50 +- 0.00 9999.50 +- 0.00
tstoragefile-read-actual-total-megabytes 878.65 +- 2.52 50.00 +- 1.50 1134.90 +- 73.97 878.65 +- 2.52
tstoragefile-read-async-total-msecs // 3000.00 +- 0.00 3000.00 +- 0.00 36900.00 +- 4752.89
tstoragefile-read-actual-num-operations 99995.00 +- 0.00 10309.50 +- 5.89 72975.00 +- 18127.74 99995.00 +- 0.00
tstoragefile-read-via-cache-total-msecs // 3000.00 +- 0.00 39600.00 +- 11208.93 //
tstoragefile-read-async-num-successful-operations // 0.50 +- 0.00 0.50 +- 0.00 9999.50 +- 0.00
tstoragefile-read-via-cache-total-megabytes // 828.75 +- 2.70 517.05 +- 150.67 //
tstoragefile-read-prefetch-to-cache-total-megabytes // // // 828.75 +- 2.70
tstoragefile-read-actual-total-msecs 372600.00 +- 40400.99 3300.00 +- 1307.67 169800.00 +- 65879.89 24000.00 +- 23043.44
tstoragefile-read-prefetch-to-cache-num-operations // // // 99995.00 +- 0.00
tstoragefile-read-total-megabytes 878.65 +- 2.52 878.65 +- 2.52 1652.00 +- 224.53 878.65 +- 2.52
tfile-read-total-megabytes 0.50 +- 0.00 0.50 +- 0.00 0.00 +- 0.00 0.50 +- 0.00
file-read-total-megabytes 878.65 +- 2.52 50.00 +- 1.50 1134.90 +- 73.97 878.65 +- 2.52
tstoragefile-read-async-total-megabytes // 0.50 +- 0.00 0.50 +- 0.00 2188.35 +- 13.31
tstoragefile-read-total-msecs 372900.00 +- 40395.42 79500.00 +- 21539.50 210900.00 +- 58273.41 62700.00 +- 23274.66
file-read-total-msecs 372000.00 +- 40092.39 3300.00 +- 1307.67 169500.00 +- 65586.20 23700.00 +- 21755.69
tstoragefile-read-num-successful-operations 9999.50 +- 0.00 9999.50 +- 0.00 9999.50 +- 0.00 9999.50 +- 0.00
file-read-num-successful-operations 9999.50 +- 0.00 9999.50 +- 0.00 9999.50 +- 0.00 9999.50 +- 0.00
tstoragefile-read-via-cache-num-successful-operations // 9999.50 +- 0.00 9999.50 +- 0.00 //
tstoragefile-read-via-cache-num-operations // 99995.00 +- 0.00 99995.00 +- 0.00 //
tstoragefile-read-async-num-operations // 5.00 +- 0.00 5.00 +- 0.00 22622.00 +- 537.33
tstoragefile-read-num-operations 99995.00 +- 0.00 99995.00 +- 0.00 99995.00 +- 0.00 99995.00 +- 0.00
file-read-num-operations 99995.00 +- 0.00 10309.50 +- 5.89 72975.00 +- 18127.74 99995.00 +- 0.00
READV
tstoragefile-readv-actual-num-successful-operations // 111.90 +- 0.86 29.25 +- 8.07 //
tstoragefile-readv-actual-total-megabytes // 2303.30 +- 20.74 0.50 +- 0.00 //
file-readv-total-msecs 3000.00 +- 0.00 74700.00 +- 21253.47 3000.00 +- 0.00 3000.00 +- 0.00
tstoragefile-readv-actual-total-msecs // 74700.00 +- 21253.47 37500.00 +- 9463.09 //
file-readv-num-successful-operations 0.50 +- 0.00 111.90 +- 0.86 0.50 +- 0.00 0.50 +- 0.00
file-readv-total-megabytes 0.50 +- 0.00 2303.30 +- 20.74 0.50 +- 0.00 0.50 +- 0.00
file-readv-num-operations 5.00 +- 0.00 115.00 +- 0.00 5.00 +- 0.00 5.00 +- 0.00
tstoragefile-readv-actual-num-operations // 115.00 +- 0.00 29.50 +- 8.05 //

-- LeonardoSala - 07-Dec-2009

Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r8 - 2009-12-16 - unknown
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback