Week of 200316

WLCG Operations Call details

  • For remote participation we use the Vidyo system. Instructions can be found here.

General Information

  • The purpose of the meeting is:
    • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
    • to announce or schedule interventions at Tier-1 sites;
    • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
    • to provide important news about the middleware;
    • to communicate any other information considered interesting for WLCG operations.
  • The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
  • The SCOD rota for the next few weeks is at ScodRota
  • General information about the WLCG Service can be accessed from the Operations Portal
  • Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes



  • remote: Kate (WLCG, DB, chair), Julia (WLCG), Maarten (WLCG, ALICE), Onno (NL-T1), Elena (CNAF), Gavin (computing), Vincent (security), Darren (RAL), Dave M (FNAL), Christoph (CMS), David B (IN2P3), Andrew (TRIUMF), Cristi (storage), Pepe (PIC), Xin (BNL), Renato (LHCb)

Experiments round table:

  • CMS reports ( raw view) -
    • CPU utilization a bit lower than usual: ~180k for production, 60k for analysis
      • Have some processing constrains to run I/O intense work at sites that host input data (pileup for 'classical' event mixing in particular)
    • Otherwise no major issues

  • ALICE -
    • NTR

  • LHCb reports ( raw view) -
    • Activity:
      • Stripping campaign ongoing, occupying most of T0/1 capacity (no T2s)
      • Staging for 2016 is almost finished. Staging for 2017 is ongoing.
    • Issues:
      • NTR

Sites / Services round table:

  • ASGC: nc
  • BNL: NTR
  • EGI: nc
  • IN2P3:
    • Following stage 3 declaration in France, IN2P3-CC has switched to best effort mode. It will strive to ensure the full continuity of service in the respect of the sanitary rules and following the instructions given by CNRS. As a result, the maintenance scheduled tomorrow is postponed. Downtime in GOCDB will be updated accordingly by the end of the day.
    • Issue:
      • ALICE opened a ticket for our ALICE voboxes (GGUS #146052). We have a problem with CERN CA CRL update for services with fetch-crl + local squid combinaison. A ticket has been opened to CERN's CA team : INC2356750 Are there other sites with the same problem?
Maarten commented that Alice observed the issues (CMS is also affected). CERN CA should check file serving. Vincent commented that it might be related to CFI servers migrated to the new system. Anyway this needs to be checked by CERN CA.
  • KIT: Announced a storage downtime for 1st of April, where we have to update the firmware of our storage systems. All storage will be offline, so all VOs will be affected!
  • NDGF: Nothing to report
  • NL-T1: NTR
  • NRC-KI:
  • OSG: nc
  • PIC: State of alarm announced in Spain. PIC staff is working from home, all services are supported. HTConder and DCache were upgrade remotely.
  • RAL: NTR

  • CERN computing services: NTR
  • CERN storage services:
  • CERN databases: NTR
  • Monitoring:
    • Final availability reports for February sent around
    • Investigating situation with ATLAS SSB metrics not being updated in the database (OTG:0055444)
  • MW Officer:
  • Networks: NTR
  • Security: Security Alert send last Friday, please check your mails


Edit | Attach | Watch | Print version | History: r23 < r22 < r21 < r20 < r19 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r23 - 2020-03-16 - KateDziedziniewicz
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback