September 2012 Reports

To the main

28th September 2012 (Friday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s
  • Reprocessing at T1s and selected T2s

  • T0:
    • CERN : cleaning TMPDIR on lxbatch (GGUS:86039 )
    • CERN : problem of protocol supported by EOS (GGUS:86226) only gsiftp is supported which prevent access to the DATA from our framework..

  • T1 :
    • SARA: SE misconfiguration (GGUS:86509); most probably disk is full

27th September 2012 (Thursday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s
  • Reprocessing at T1s and selected T2s

  • T0:
    • CERN : cleaning TMPDIR on lxbatch (GGUS:86039 )
    • CERN : problem of protocol supported by EOS (GGUS:86226) only gsiftp is supported which prevent access to the DATA from our framework..

  • T1 :
    • CNAF: SE downtime: Solved
    • SARA: wrong TURLs returned from SARA's SRM (GGUS:86435): Solved

26th September 2012 (Wednesday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s
  • Reprocessing at T1s and selected T2s

  • T0:
    • CERN : cleaning TMPDIR on lxbatch (GGUS:86039 )
    • CERN : problem of protocol supported by EOS (GGUS:86226) only gsiftp is supported which prevent access to the DATA from our framework..

  • T1 :
    • CNAF: SE downtime
    • SARA: wrong TURLs returned from SARA's SRM (GGUS:86435)

25th September 2012 (Tuesday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s
  • Reprocessing at T1s and selected T2s

  • T0:
    • CERN : cleaning TMPDIR on lxbatch (GGUS:86039 )
    • CERN : problem of protocol supported by EOS (GGUS:86226) only gsiftp is supported which prevent access to the DATA from our framework..

  • T1 :
    • CNAF: SE downtime

24th September 2012 (Monday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s
  • Reprocessing at T1s and selected T2s

  • T0:
    • CERN : cleaning TMPDIR on lxbatch (GGUS:86039 )
    • CERN : problem of protocol supported by EOS (GGUS:86226) only gsiftp is supported which prevent access to the DATA from our framework..

  • T1 :
    • CNAF: SE downtime

21st September 2012 (Friday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s
  • Reprocessing at T1s and selected T2s : 12k jobs running

  • T0:

  • T1 :
    • CERN : problem of protocol supported by EOS (GGUS:86226) only gsiftp is supported which prevent access to teh DATA from our framework..

20th September 2012 (Thursday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s
  • Reprocessing at T1s and selected T2s : 12k jobs running
  • Dashboard availability : We see NIKHEF and PIC unavailable in the dashboard as of this morning. This is most probably not a real problem but jobs are timing out. The reason is that we are submitting with role lhcb_prod and the jobs are dying in the queues b/c we run heavily reprocessing at the moment and sites are fully loaded.

  • T0:

  • T1 :
    • CERN : problem of protocol supported by EOS (GGUS:86226) only gsiftp is supported which prevent access to teh DATA from our framework..

19th September 2012 (Wednesday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s
  • Reprocessing at T1s and selected T2s : 12k jobs running

  • T0:

  • T1 :
    • IN2P3 : downtime finished

18th September 2012 (Tuesday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s
  • Reprocessing at T1s and selected T2s : started

  • T0:

  • T1 :
    • IN2P3 : downtime tomorrow
    • GRIDKA : system tape overloaded (GGUS:86194) On the LHCb system the size in the configuration service was incorrect and we put too many request. We decrease the number and the number of request should decrease in the coming hours.
    • CERN : castor intervention
    • PIC : downtime tomorrow

17th September 2012 (Monday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s
  • Reprocessing at T1s and selected T2s : started

  • T0:

  • T1 :
    • IN2P3 : downtime tomorrow
    • PIC : downtime tomorrow

14th September 2012 (Friday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s
  • Validation of reprocessing at T1s and selected T2s : OK
  • validation of stripping started

  • T0:

  • T1 :
    • IN2P3 : problem of jobs stalled : investigate if it is not a problem CPU time left not properly reported..

13th September 2012 (Thursday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s
  • Validation of reprocessing at T1s and selected T2s : OK
  • validation of stripping started

  • T0:
    • CERN : cleaning TMPDIR on lxbatch (GGUS:86039 ) CASTOR intervention

  • T1 :
    • CERN : FTS transfer problem between CERN-GRIDKA (GGUS:86025)

12th September 2012 (wednesday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s
  • Validation of reprocessing at T1s and selected T2s : OK
  • validation of stripping started

  • T0:

  • T1 :
    • CERN : FTS transfer problem between CERN-GRIDKA (GGUS:86025)
    • SARA : one node was not beahving properly and all LHCbpilot were aborted.
    • PIC : tape repack in progress

11th September 2012 (Tuesday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s
  • Validation of reprocessing at T1s and selected T2s : OK
  • validation of stripping started

  • T0: Problem of stratum 1 for CVMFS which was full. Problem understood

  • T1 :

10th September 2012 (Monday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s
  • Validation of reprocessing at T1s and selected T2s : OK
  • validation of stripping will start soon

  • T0: Problem of stratum 1 for CVMFS which was full

  • T1 :

7 September 2012 (Friday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s
  • Validation of replrocessing at T1s and selected T2s

  • T0:

  • T1 :

5 September 2012 (Wednesday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s

  • T0:

  • T1 :
    • IN2P3: Jobs failures, cannot load shared library, site was banned for production (GGUS:85644); Wrong version of tcmalloc with link to AFS and wrong version of gcc; Fixed
    • GRIDKA: Pilots failed at two CEs (GGUS:85716); Probably fixed
    • PIC: Pilots aborted (GGUS:85746); Medium queues were removed, but they are still in BDII; Fixed

4 September 2012 (Tuesday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s

  • T0:

  • T1 :
    • IN2P3: Jobs failures, cannot load shared library, site was banned for production (GGUS:85644); Wrong version of tcmalloc with link to AFS and wrong version of gcc
    • GRIDKA: Pilots failed at two CEs (GGUS:85716); Still failed
    • PIC: Pilots aborted (GGUS:85746); Medium queues were removed, but they are still in BDII

3 September 2012 (Monday)

  • Running user analysis, prompt reconstruction and stripping at T0 and T1s
  • Simulation at T2s

  • T0:
    • constant rate of aborted pilot observed during last week GGUS:85385; Solved
    • a drop of 50TB in the disk occupancy GGUS:85708; Solved

  • T1 :
    • IN2P3: Jobs failures, cannot load shared library, site was banned for production (GGUS:85644); Under investigation
    • GRIDKA: Pilots failed at two CEs (GGUS:85716)

-- JoelClosier - 10-Oct-2012

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2012-12-05 - JoelClosier
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback