Week of 200309

WLCG Operations Call details

  • The meeting is Vidyo only until further notice due to COVID-19 considerations.

  • For remote participation we use the Vidyo system. Instructions can be found here.

General Information

  • The purpose of the meeting is:
    • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
    • to announce or schedule interventions at Tier-1 sites;
    • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
    • to provide important news about the middleware;
    • to communicate any other information considered interesting for WLCG operations.
  • The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
  • The SCOD rota for the next few weeks is at ScodRota
  • General information about the WLCG Service can be accessed from the Operations Portal
  • Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

Monday

Attendance:

  • local:
  • remote: Julia (WLCG), Kate (WLCG, DB, chair), Eric C (storage), Elena (CNAF), Ivan (ATLAS), Onno (NL-T1), Olga (computing), Vincent (security), Andrew (TRIUMF), Borja (monitoring), Christoph (CMS), Gavin (computing), Cristi (storage), Darren (RAL), DaveM (FNAL), Xin (BNL), Darren (NeIC), Jens (NDGF)

Experiments round table:

  • CMS reports ( raw view) -
    • CMS presently struggling a bit with disk space usage
      • We have a number of MC requests that request large output data tiers
    • Otherwise business as usual

  • ALICE -
    • NTR

Sites / Services round table:

  • ASGC: nc
  • BNL: SE (dCache) downtime is scheduled for 03/25~03/27 (48 hours), for dCache upgrade to version 5.2.
  • CNAF: on March 10th from 6.00 to 17.00 (UTC) there is a scheduled disk downtime to HW intervention only for ALICE and ATLAS, tape will be not affected. GOCDB 28499
  • EGI: nc
  • FNAL: NTR
  • IN2P3: nc -- [After the meeting] Apologies for not being connected. IN2P3-CC will be in downtime on Tuesday 17th March for the whole day: CREAMCEs will be decommissioned and HTCondorCE in downtime for upgrade. SEs will be off also.
  • JINR: nc
  • KISTI: nc
  • KIT: NTR
  • NDGF: NTR
  • NL-T1:
    • Last week Surfsara had two network issues in the dCache cluster causing instabilities.
    • During the weekend a dCache poolnode crashed; we're investigating this.
  • NRC-KI: nc
  • OSG: nc
  • PIC: We will be in OUTAGE from 16-03-2020 08:00 (PIC local time) until 16-03-2020 17:00 (PIC local time), for dCache upgrade and HTCondor upgrade. Additionally, on Tuesday 7th of April from 5:00 to 23:00 PIC Time we will be in downtime, due to the yearly electrical maintenance in the building.
  • RAL: NTR
  • TRIUMF: NTR

  • CERN computing services: NTR
  • CERN storage services:
    • All CASTOR instances will be impacted by OTG:0054761. A manual restart of DB backed CASTOR daemons might be required.
  • CERN databases:
    • (reminder from last week) Oracle and Database on Demand databases will be unavailable on the 11th of March in the morning, due to a network intervention
      • ACCLOG/ACCMEAS databases down due to network intervention OTG:0054762
      • DBoD instances down due to network intervention for about 1h OTG:0054760
      • Oracle databases unavailable for a few minutes due to network intervention OTG:0054761
  • GGUS: NTR
  • Monitoring: NTR
  • MW Officer:
  • Networks: NTR
  • Security: NTR

AOB:

Edit | Attach | Watch | Print version | History: r23 < r22 < r21 < r20 < r19 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r23 - 2020-03-10 - XinZhao
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback