Week of 130923

WLCG Operations Call details

To join the call, at 15.00 CE(S)T, by default on Monday and Thursday (at CERN in 513 R-068), do one of the following:

1. Dial +41227676000 (Main) and enter access code 0119168, or
2. To have the system call you, click here

The scod rota for the next few weeks is at ScodRota

WLCG Availability, Service Incidents, Broadcasts, Operations Web

VO Summaries of Site Usability				SIRs	Broadcasts	Operations Web
ALICE	ATLAS	CMS	LHCb	WLCG Service Incident Reports	Broadcast archive	Operations Web

General Information

General Information			GGUS Information	LHC Machine Information
CERN IT status board	WLCG Baseline Versions	WLCG Blogs	GgusInformation	Sharepoint site - LHC Page 1

Monday

Attendance:

• local: AndreaV/SCOD, Victor/Grid, Ken/CMS, MariaD/GGUS, Belinda/Storage, Ivan/Dashboard
• remote: Michael/BNL, Onno/NLT1, John/RAL, Sang-Un/KISTI, Wei-Jen/ASGC, Christian/NDGF, Lisa/FNAL, Rob/OSG, Rolf/IN2P3, Salvatore/CNAF, Doug/ATLAS

Experiments round table:

• ATLAS reports (raw view) -
  • Central Services
    • FT3-Pilot (GGUS:97359) Problem with a cached proxy affected functional tests to all sites. (solved)
    • AFS - ~13:03 on 19-Sept spurious rm process on /afs/cern.ch/atlas/offline/* removed RW areas including panda client areas needed by Hammer Cloud. Computing operations restored the needed area from tape promptly when alerted 20-Sept. Exact cause of rm is unknown.(INC:388802) ATLAS investigation continues.
  • T0
    • NTR
  • T1
    • NTR

• CMS reports (raw view) -
  • It has been fairly quiet until just a few hours ago!
  • Appears to be a CVMFS problem at KIT, see GGUS:97505 and GGUS:97506.
  • SAV:139882 and SAV:139885 appear to indicate EOS transition teething pains at FNAL.
  • No one seems to be complaining about network connectivity to Russia, nor has anyone given me any update on the status.
  • Why does CERN external networking show as degraded to 60% when it appears that all services are green? The service has been degraded for a week; I think the usual culprit is the (irrelevant) link to the Wigner center, but at the moment that looks green too. [MariaD: this may be due to the many interventions last week. AndreaV: will follow up - opened INC:391013 after the meeting]

• ALICE -
  • NTR

• LHCb reports (raw view) -
  • Main activity are MC productions
  • Fall incremental stripping campaign will be launched coming Monday 30 Sep (see also WLCG Operations Coordination meeting, 19 Sep)
  • T0: ntr
  • T1:
    • IN2P3: Currently in DT for sl6 upgrade [Rolf/!IN2P3: sorry we did a mistake, IN2P3 is marked in DT today since yesterday evening, but actually the downtime and interventions will start this evening]
    • SARA: Currently in DT
    • GRIDKA: In DT as of 14.00 today

Sites / Services round table:

• Michael/BNL: ntr
• Onno/NLT1:
  • today's maintenance was successful
  • announcement, on October 8 we will move from LHCOPN to LHCONE
• John/RAL: disk server issue for ALICE, being fixed
• Sang-Un/KISTI: ntr
• Wei-Jen/ASGC: ntr
• Christian/NDGF: ntr
• Lisa/FNAL: ntr
• Rob/OSG: ntr
• Rolf/IN2P3: nta
• Salvatore/CNAF: ntr
• Pavel/KIT [via email]: CVMFS problems are not yet quite understood and under investigation

• Belinda/Storage:
  • issue with non-resolving IP address for EOS ALICE over the weekend, now fixed but a more robust fix is being developed (this is a known issue)
  • EOS ATLAS crashed this morning, fixed by restarting, investigations ongoing
• Victor/Grid: ntr
• Ivan/Dashboard: ntr
• MariaD/GGUS: Due to personal circumstances in the GGUS team, GGUS release will be postponed from this Wednesday 2013/09/25. You shall be informed about the new release date. It will be decided a.s.a.p.

AOB: none

Thursday

Attendance:

• local:
• remote:

Experiments round table:

• ATLAS reports (raw view) -

• CMS reports (raw view) -

• ALICE -

• LHCb reports (raw view) -

Sites / Services round table:

AOB: