WLCGOpsMeetingWeek200810 < LCG

LCG Web>WLCGCommonComputingReadinessChallenges>WLCGOperationsMeetings>WLCGOpsMeetingWeek200810 (2020-08-10, MaartenLitmaath) (raw view)

EditAttachPDF

---+!! Week of 200810

%TOC%

---++ WLCG Operations Call details

   * At CERN the meeting room is [[https://maps.cern.ch/mapsearch/mapsearch.htm?n=%5b%27513/R-068%27%5d][513-R-068]].

   * For remote participation we use the Vidyo system. Instructions can be found [[https://indico.cern.ch/conferenceDisplay.py?confId=287280][here]].

---++ General Information
   * The purpose of the meeting is:
      * to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
      * to announce or schedule interventions at Tier-1 sites;
      * to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
      * to provide important news about the middleware;
      * to communicate any other information considered interesting for WLCG operations.
   * The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
   * The SCOD rota for the next few weeks is at ScodRota
   * Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOT.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

---++ Best practices for scheduled downtimes 

   * [[BestPracticesForSchedDT][Best practices for scheduled downtimes]]

---++ Monday

Attendance:
   * local: 
   * remote: Miro (Chair), Alberto (Monitoring), Borja (Monitoring), Darren (RAL), David (IN2P3), Julia (WLCG), Maarten (ALICE), Nils (Compute), Pavlo (ATLAS), Vincent (Security), Xin (BNL), Xavier (KIT), Dave (FNAL), Andrei (DB), Dennis (NL-T1), Pepe (PIC), Vladimir (LHCb), Jens (NDGF)

Experiments round table:

   * ATLAS [[AtlasComputing.ADCOperationsWeeklySummaries2020][reports]] ( [[AtlasComputing.ADCOperationsWeeklySummaries2020?raw=on][raw view]]) -
      * FZK-LCG2: storages issues, impact from high activity of ALICE (GGUS:148164)
      * IN2P3-CC: frontier issues
      * INFN-ROMA1: hardware issues with one of the storage servers (GGUS:148165)


   * CMS [[CMS.FacOps_WLCGdailyreports][reports]] ( [[CMS.FacOps_WLCGdailyreports?raw=on][raw view]]) -
      * Overall very good CPU utilization: reached ~330k cores (~280k production, ~50k analysis) during the weekend, could be a new record for CMS
      * No big problems

   * ALICE -
      * NTR

   * LHCb [[LHCb.ProductionOperationsWLCGdailyReports][reports]] ( [[LHCb.ProductionOperationsWLCGdailyReports?raw=on][raw view]]) -
      * Activity:
         * Usual MC, user and WG production. 
      * Issues:
         * IN2P3 decreased number of running job : Fixed
         * RAL FTS3 transfers issue : GGUS:148187 : Fixed

Sites / Services round table:

   * ASGC: NC
   * BNL: downtime scheduled for Aug. 24~25, dCache upgrade to v6.2, and enabling SRR. 
   * CNAF: NTR
   * EGI: NC
   * FNAL: NTR
   * !IN2P3:
      * Modifications in HTCondor configuration resulted in a decrease in the number of jobs for LHCb. It has been fixed.
      * Frontier issues caused by heavy load of traffic + a disk space problem. Now fixed. Disk space will be increased during the next maintenance.
   * JINR: NC
   * KISTI: NTR
   * KIT:
      * Downtime for replacing the CMS dCache door node with two new (virtual) machines was extended from 10:00 to 13:30 CEST, because firewall ACLs and DNS records could not be updated faster.
      * Following this intervention, !CMS@GridKa was down for almost two days, till Thursday morning, where we eventually found a misconfigured router, that made the CMS MC SAM tests fail.
      * Storage performance was dragged down by a load hot spot on two servers caused by extreme Alice activity. This indicates an issue with how Alice data is distributed, but we need to investigate this a bit more. We had to put an I/O limit for Alice on GPFS to improve the situation for literally all other clients.
      * Site downtime that was planned for September 26th already has been shifted to October 6th. Formal GOC-DB downtime announcement still pending.
   * NDGF: NTR
   * NL-T1: Sara-matrix dCache upgrade on Wednesday 19th: https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=29218
   * NRC-KI: NC
   * OSG: NC
   * PIC: NTR
   * !RAL: NTR
   * TRIUMF: NTR

   * CERN computing services: Nothing To Report.
   * CERN storage services: NC
   * CERN databases: Nothing To Report
   * GGUS: NTR
   * Monitoring:
      * SiteMon draft availability reports for July sent around
      * Exceptionally, We would like to extend recomputation request period until the end of August
   * MW Officer: NC
   * Networks: NC
   * Security: NTR

AOB:

Topic revision: r20 - 2020-08-10 - MaartenLitmaath

LCG Wikis

LCG Service
Coordination

LCG Grid
Deployment

LCG
Apps Area

Public webs

Welcome Guest

- Cern Search
- TWiki Search
- Google Search
LCG All webs

Copyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback