Week of 200113

WLCG Operations Call details

  • For remote participation we use the Vidyo system. Instructions can be found here.

General Information

  • The purpose of the meeting is:
    • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
    • to announce or schedule interventions at Tier-1 sites;
    • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
    • to provide important news about the middleware;
    • to communicate any other information considered interesting for WLCG operations.
  • The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
  • The SCOD rota for the next few weeks is at ScodRota
  • General information about the WLCG Service can be accessed from the Operations Portal
  • Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes



  • local: Julia (WLCG), Maarten (WLCG, ALICE), Kate (WLCG, chair), Renato (LHCb), Vincent (security), Ivan (ATLAS), Borja (monitoring), Olga (computing), Joao (storage)
  • remote: Onno (NL-T1), Elena (CNAF), Jens (NDGF), Xavier (KIT), Darren (RAL), Christoph (CMS), Di (TRIUMF), Dave M (FNAL), Xin (BNL)

Experiments round table:

  • ATLAS reports ( raw view) -
    • High ADCR DB load over the weekend due to a network issue. Investigation - ongoing.

  • CMS reports ( raw view) -
    • Good CPU utilization: 200k cores production + 50k cores analysis
    • No major issues
    • Just one item to mention: CERN EOS appears to have lost the subscription to the data federation: GGUS:144780 (in progress)

  • ALICE -
    • NTR

  • LHCb reports ( raw view) -
    • Activity:
      • Running MC simulation, WG productions, user analysis.
    • Issues:
      • RAL: failing access by protocol to files in ECHO

Sites / Services round table:

  • ASGC: nc
  • BNL: NTR
  • EGI: nc
  • IN2P3: nc
  • KISTI: nc
  • KIT:
    • No major issues during Christmas or New Year period.
    • Currently working on updating OS and GPFS version of our centralized GPFS cluster. That is supposed to be transparent, but occasionally, few dCache pools encounter I/O errors and need to be restarted.
    • ARC-CE-4, currently in downtime, will be replaced with a different machine this week (keeping the hostname).
  • NL-T1: NTR
  • NRC-KI: nc
  • OSG: nc
  • PIC: nc
  • RAL: NTR
  • TRIUMF: Last Wednesday (Jan 8) there was a 7-hour scheduled downtime, in this downtime we upgraded the DDN storage controller firmware, updated dcache version to 5.2.12 and also updated the majority of our nodes to the latest kernel and rebooted them. However in the new version of dCache, there are some configuration needed for VOMS proxy authentication, thus caused the xrootd doors had not worked properly, we only found this Thursday.

  • CERN computing services: NTR
  • CERN storage services:
    • EOSALICEDAQ namespace migration planned for Wednesday, 15th 10:00AM; OTG0054144 (1h downtime)
    • CASTOR stager database update planned for all instances.
      • Oracle 11 to Oracle 19.
      • The stagers will be down for the day (09h00 to 16h00).
      • Planned dates are: Mon 3rd Feb for CMS, Wed 5th Feb for Public, Mon 10th Feb for ATLAS, Wed 12th Feb for LHCb and Mon 17th Feb for ALICE.
      • Will be validated on PPS instance beforehand.
  • CERN databases: NTR
  • Monitoring: NTR
  • MW Officer: nc
  • Networks: nc
  • Security: NTR


Edit | Attach | Watch | Print version | History: r17 < r16 < r15 < r14 < r13 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r17 - 2020-01-13 - KateDziedziniewicz
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback