Week of 200824

WLCG Operations Call details

  • For remote participation we use the Vidyo system. Instructions can be found here.

General Information

  • The purpose of the meeting is:
    • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
    • to announce or schedule interventions at Tier-1 sites;
    • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
    • to provide important news about the middleware;
    • to communicate any other information considered interesting for WLCG operations.
  • The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
  • The SCOD rota for the next few weeks is at ScodRota
  • Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes



  • local: -
  • remote: Andrew (NL-T1), Andrew (TRIUMF), Borja (Chair, Monitoring), Christoph (CMS), David (IN2P3), Dave (FNAL), Eric (Storage), Julia (WLCG), Maarten (ALICE), Michal (ATLAS), Stefano (LHCb), Xin (BNL)

Experiments round table:

  • ATLAS reports ( raw view) -
    • Activities:
      • Ongoing reprocessing
    • Issues:
      • short-term mass-blacklisting of queues on Monday evening
        • AGIS API broken by unrelated CRIC update
      • jobs at CERN queues failing with “Network is unreachable” (GGUS:148283)
        • There where some diskservers which were not available since they were assigned a private IP address - fixed
      • monitoring infrastructure problems
        • Tuesday (OTG:0058488) - caused by network incident affecting nodes in cern-geneva-a zone
        • Saturday (RQF:1637766)
      • CERN frontier down (GGUS:148322) on Thursday
        • misconfigured tomcat port
        • it did not cause increase in failure rate (frontiers at Lyon and Triumf were working)

  • CMS reports ( raw view) -
    • Excuted a tape deletion campaign at Tier-1 sites over the last two weeks. All went according to plans.
    • Again good CPU utilization for CMS: 200k+ CPU cores for production and ~45k cores for analysis
    • No major issues

  • ALICE -
    • NTR

  • LHCb reports ( raw view) -
    • Activity:
      • Usual MC, user and WG production.
    • Issues:
      • SARA/NIKHEF, GGUS:148321 : Timeout during transfers from SARA-BUFFER. Already investigated last week and now the problem is much mitigated but still there. Will updated ticket.

Sites / Services round table:

  • ASGC: NC
  • BNL: As announced before, BNL dCache is down today and tomorrow for upgrade to v6.2.
  • EGI: NC
  • IN2P3: IN2P3-CC will be in maintenance on September 22th, a Tuesday. As usual details will be available one week before the event.
  • KIT: Draining arc-5-kit.gridka.de starting September 2nd (GOC-DB:29271), because we want to shut it down as the first of the legacy ARC-CEs in favor of the new HTCondor-CEs.
  • NDGF: NC
  • NL-T1:
    • Nikhef: one of the redundant switches between our main xen host cluster and the storage which holds the VMs has been having issues after the power interuption we suffered on 14th August. We had several unexpected downtimes last week as the failing switch rebooted and due to firmware issues this also caused the working switch to reboot, killing our VMs. At the moment the failed switch has been removed and our VMs are running normally, but without the redundant network links. An outage is scheduled for tomorrow to allow for more work on the failed switch and possibly replacing it.
  • NRC-KI: NC
  • OSG: NC
  • PIC: NTR
  • RAL: NC

  • CERN computing services: NC
  • CERN storage services:
  • CERN databases: NC
  • Monitoring:
    • SAM3 dashboard brought back after some users request, new deadline is 31st of August
    • Last week to submit recomputation request for July reports since they are going to be sent next Monday
    • Tuesday (OTG:0058488) - Problems caused by network incident affecting nodes in cern-geneva-a zone
    • Saturday, monit-grafana servers not accessible for a short period of time
  • MW Officer: NC
  • Networks: NC
  • Security: NTR


This topic: LCG > WebHome > WLCGCommonComputingReadinessChallenges > WLCGOperationsMeetings > WLCGOpsMeetingWeek200824
Topic revision: r19 - 2020-08-25 - NikolayVoytishin
This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback