June 2014 Reports

30 June 2014 (Monday)

• Main activity: MC and User jobs
• T0:
  • Brazilian proxy problem on EOS (GGUS:106434)
• T1:
  • GridKa : CVMFS problem on WN (GGUS:106405) ongoing
  • RAL : GGUS:105571 - inconsistent storage information publishing actually a sensor problem (mixup of tape and disk info)?

26 June 2014 (Thursday)

• Main activity: User jobs continuing, test reprocessing done and preparing for more, larger MC productions running again
• T0:
  • CASTORLHCB down on monday due to a single user hitting it. Fine now after the user was banned.
  • Brazilian proxy problem on EOS (GGUS:106434)
• T1:
  • GridKa : CVMFS problem on WN (GGUS:106405) ongoing
  • RAL : GGUS:105571 - inconsistent storage information publishing actually a sensor problem (mixup of tape and disk info)?

23 June 2014 (Monday)

• Main activity: User jobs continuing, test reprocessing done and preparing for more, larger MC productions running again
• T0: CASTORLHCB seems down currently (https://cern.service-now.com/service-portal/view-incident.do?n=INC0583679)
• T1:
  • IN2P3 : Was publishing file protocol after dCache upgrade. After alarm ticket (GGUS:106368) fixed very fast on Friday.
  • GridKa : CVMFS problem on WN (GGUS:106405)
• Various sites :
  • BDII and SRM publish inconsistent storage information (RAL - GGUS:105571 , CBPF - GGUS:105572)
  • Multiple sites (SARA, RRCKI) provided extra storage last Friday - thanks!

19 June 2014 (Thursday)

• Main activity: User jobs continuing, test reprocessing done and preparing for more, larger MC productions running again
• T0: NTR
• T1: NTR

16 June 2014 (Monday)

• Main activity: User jobs continuing, reprocessing campaign restarted, expect a lot more MC jobs later this week
• T0:
• T1:
  • CNAF : still checking files after DT

12 June 2014 (Thursday)

• Main activity: MC and user jobs, reprocessing campaign about to start today or tomorrow
• T0:
• T1:
  • CNAF : back from their DT, in contact with LHCb DataMgmt team to restore missing files

10 June 2014 (Tuesday)

• Main activity: MC and user jobs, low amount of activities over the long week-end
• T0:
• T1:
  • CNAF : problem of storage, LFC dump given to site for consistency checks
  • RAL: Problems with ARC CEs publishing wrong values for MaxCPUTime in the BDII (GGUS:106059), fixed
  • RAL: It seems the local batch system returns 0 for the CPU time left for a landed pilot, all jobs failing (GGUS:)
  • RAL & PIC in DT today

05 June 2014 (Thursday)

• Main activity: MC and user jobs. Moving VOBOXes (physical machine to VM)
• T0: add to the trusted list new VOBOX for LFC
• T1: CNAF : problem of storage.

02 June 2014 (Monday)

• Main activity: MC and user jobs
• T0: NTR
• T1:
  • GRIDKA:
    • File access problems reported for user jobs, couldn't be reproduced at the site, investigation ongoing
    • Issue with one WN CVMFS is fixed.

-- JoelClosier - 31 Mar 2014