Week of 190722

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Portal
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Alberto (Monitoring), Borja (Chair, Monitoring), Julia (WLCG), Olga (Computing), Petr (ATLAS), Remy (Storage), Vincent (Security)
• remote: Alexander (NL-T1), Christian (NDGF), Christoph (CMS), Darren (RAL), Dave (FNAL), David (IN2P3), Di (TRIUMF), Marcelo (CNAF), Sang-Un (KISTI), Sprezzi (LHCb)

Experiments round table:

• ATLAS reports ( raw view) -
  • running with ~ 360k job slots (CERN-P1 back for grid production)
  • pilot2 (+containers) migration almost done for grid sites
    • few sites with problem - currently without containers (some with pilot1)
    • specific PQ configuration could cause troubles with big output and memory usage
  • sites not yet on CentOS7 & singularity reminded by GGUS ticket
    • in a month stop brokering analysis jobs to SL6 queues, offline in two months
    • in two months stop brokering production jobs to SL6 queues, offline in three months

• CMS reports ( raw view) -
  • CPU utilization ~225k cores
    • Usual mix ~25% Analysis, 75% Production
    • Slightly lower this week, likely because of HLT contribution of 15k cores 'only'
  • Only items worth reporting
    • Lost some files from tape at CNAF (GGUS:142345) and CC-IN2P3 (GGUS:142362)
  • Otherwise no major issues

• ALICE -
  • NTR

• LHCb reports ( raw view) -
  • Activity:
    • MC, user jobs and staging. Validation for reprocessing of 2011 and 2012 data finished, will start productions soon.
    • Almost all sites agreed to split aliases for end-point of disk and tape. Still missing SARA (split is difficult form their side). Important to have all sites with split aliases otherwise need to maintain an hack in the code.
  • Issues:
    • RAL:
      • GGUS:142350; problem accessing files on ECHO. Under investigation.
      • GGUS:142337; issues with killed pilots. Under investigation

Sites / Services round table:

• ASGC: NC
• BNL: NTR
• CNAF: NTR
• EGI: NC
• FNAL: NTR
• IN2P3: NTR
• JINR: Our networking experts still working on WAN problems.
• KISTI: NTR
• KIT: NTR
• NDGF: NTR
• NL-T1: NTR
• NRC-KI: NC
• OSG: NC
• PIC: NC
• RAL: NTR
• TRIUMF: NTR

• CERN computing services:
  • OTG:0051477 - Subset of Batch workers were affected over the weekend by connectivity issues.
  • OTG:0051185 - Hypervisor reboot schedule to mitigate MDS Intel CPU vulnerabilities
    • Will proceed by availability zone, from 18th July. Please watch OTG for updates.
    • Expect next set of CERN CEs to be rebooted on Wednesday, OTG:0051289 and OTG:0051294
• CERN storage services:
  • CASTOR : Nameserver DB upgrade tomorrow 23rd of July, all CASTOR instances will be impacted. OTG:0051004
• CERN databases: NC
• GGUS: NTR
• Monitoring:
  • Final availability reports for June 2019 sent around
• MW Officer: NC
• Networks: NTR
• Security: Nothing To Report

AOB: