Week of 230313

WLCG Operations Call details

• The connection details for remote participation are provided on this agenda page.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to the wlcg-scod list (at cern.ch) to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Maarten (ALICE, WLCG), Panos (CMS, WLCG)
• remote: Kate (DB, chair), Julia (WLCG), Darren (RAL), Federico (LHCb), Peter (ATLAS), Andrew (TRIUMF), Xavier (KIT), Laurence (computing), Dave (FNAL), Douglas (BNL), David (IN2P3), Christian (NDGF), Pepe (PIC)

Experiments round table:

• ATLAS reports ( raw view) -
  • Issues
    • Issue with transfers from UK to INFN-T1 (GGUS:160759), INFN-T1 upgraded to IGTF 1.119 this morning, transfers now succeed. Also, other implementations are not affected so has this been understood with Storm developers?
    • Taiwan-LCG2 tickets, transfer errors GGUS:158820, systemic DPM issues GGUS:160691, DPM migration and decommission (in progress) GGUS:160808

Maarten commented that the issues was caused by the certificated being generated using the latest version of open-ssl, where a default behaviour has changed causing issues with a seemingly proper certificate. The certificate had to be recreated.

• CMS reports ( raw view) -
  • XRootd daylight-savings bug hit US sites again Sunday; US sites need to restart xrootd services to get servers into the right timezone; European sites will need to restart in two weeks; fix should have been in v5 but old ticket is still open, CMS is following up with the xrootd team. The relevant issue and potential fix can be found here: https://github.com/xrootd/xrootd/issues/1955 https://github.com/xrootd/xrootd/pull/1958
  • IAM security incident:
    • Forensics have been completed, as reported from both the CERN and OSG security teams. No trace of malicious activity has been identified.

• ALICE
  • Normal activity levels, no major issues
  • Sites will be informed of the ALICE token configuration details this week
    • Normal jobs were already tested OK at KIT last week

• LHCb reports ( raw view) -
  • Adding MP queues for T0 and T1s

Sites / Services round table:

• ASGC:
• BNL: NTR
• CNAF: NTR
• EGI:
• FNAL: DCache upgrade planned for Tue, Mar 21st. Storage will be in downtime.
• IN2P3: Reminder that site will be in scheduled maintenance tomorrow. Downtimes during the morning as follow:
  • dCache upgrade ( GOC DB #33659)
  • tape system software upgrade ( GOCDB #33666)
• JINR:
• KISTI:
• KIT: We're updating the HPSS tape backend, so on Thursday between 10 and 18 o'clock CET, CMS and LHCb cannot stage data.
• NDGF: NTR
• NL-T1:
• NRC-KI:
• OSG:
• PIC: Tape library intervention was cancelled, it will be re-scheduled.
• RAL: Last week Antares (Tape service) was down for a upgrade to the Spectra Logic Tape Robot. This was scheduled for Monday (6th) until Friday (10th) lunchtime. Due to a problem with the installation the downtime had to be extended until Tomorrow (14th). We are hopeful that the work will be completed this afternoon and we can end the downtime slightly early, this evening. For those sites that also have a Spectra Logic Library and are interested, we can circulate the more technical description of the problem and its solution.

• TRIUMF: NTR

• CERN computing services: Lxbatch capacity to be restored OTG0076391
• CERN storage services:
• CERN databases: NTR
• GGUS: NTR
• Monitoring:
• Middleware: NTR
• Networks: NTR
• Security:

AOB: