WLCGOpsMeetingWeek190722 < LCG

LCG Web>WLCGCommonComputingReadinessChallenges>WLCGOperationsMeetings>WLCGOpsMeetingWeek190722 (2019-07-22, PetrVokac) (raw view)

EditAttachPDF

---+!! Week of 190722

%TOC%

---++ WLCG Operations Call details

   * At CERN the meeting room is [[https://maps.cern.ch/mapsearch/mapsearch.htm?n=%5b%27513/R-068%27%5d][513-R-068]].

   * For remote participation we use the Vidyo system. Instructions can be found [[https://indico.cern.ch/conferenceDisplay.py?confId=287280][here]].

---++ General Information
   * The purpose of the meeting is:
      * to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
      * to announce or schedule interventions at Tier-1 sites;
      * to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
      * to provide important news about the middleware;
      * to communicate any other information considered interesting for WLCG operations.
   * The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
   * The SCOD rota for the next few weeks is at ScodRota
   * General information about the WLCG Service can be accessed from the [[https://wlcg-ops.web.cern.ch/][Operations Portal]]
   * Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOT.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

---++ Best practices for scheduled downtimes

   * [[BestPracticesForSchedDT][Best practices for scheduled downtimes]]

---++ Monday

Attendance:
   * local: Alberto (Monitoring), Borja (Chair, Monitoring), Julia (WLCG), Olga (Computing), Petr (ATLAS), Remy (Storage), Vincent (Security)
   * remote: Alexander (NL-T1), Christian (NDGF), Christoph (CMS), Darren (RAL), Dave (FNAL), David (IN2P3), Di (TRIUMF), Marcelo (CNAF), Sang-Un (KISTI), Sprezzi (LHCb)

Experiments round table:

   * ATLAS [[AtlasComputing.ADCOperationsWeeklySummaries2019][reports]] ( [[AtlasComputing.ADCOperationsWeeklySummaries2019?raw=on][raw view]]) -
      * running with ~ 360k job slots (CERN-P1 back for grid production)
      * pilot2 (+containers) migration almost done for grid sites
         * few sites with problem - currently without containers (some with pilot1)
         * specific PQ configuration could cause troubles with big output and memory usage
      * [[AtlasComputing.CentOS7Deployment][sites not yet on CentOS7 & singularity]] reminded by GGUS ticket
         * in a month stop brokering analysis jobs to SL6 queues, offline in two months
         * in two months stop brokering production jobs to SL6 queues, offline in three months

   * CMS [[CMS.FacOps_WLCGdailyreports][reports]] ( [[CMS.FacOps_WLCGdailyreports?raw=on][raw view]]) -
      * CPU utilization ~225k cores
         * Usual mix ~25% Analysis, 75% Production
         * Slightly lower this week, likely because of HLT contribution of 15k cores 'only'
      * Only items worth reporting
         * Lost some files from tape at CNAF (GGUS:142345) and CC-IN2P3 (GGUS:142362)
      * Otherwise no major issues

   * ALICE -
      * NTR

   * LHCb [[LHCb.ProductionOperationsWLCGdailyReports][reports]] ( [[LHCb.ProductionOperationsWLCGdailyReports?raw=on][raw view]]) -
      * Activity:
         * MC, user jobs and staging. Validation for reprocessing of 2011 and 2012 data finished, will start productions soon.
         * Almost all sites agreed to split aliases for end-point of disk and tape. Still missing SARA (split is difficult form their side). Important to have all sites with split aliases
   otherwise need to maintain an hack in the code.
      * Issues:
         * RAL:
            * GGUS:142350; problem accessing files on ECHO. Under investigation.
            * GGUS:142337; issues with killed pilots. Under investigation

Sites / Services round table:

   * ASGC: NC
   * BNL: NTR
   * CNAF: NTR
   * EGI: NC
   * FNAL: NTR
   * !IN2P3: NTR
   * JINR: Our networking experts still working on WAN problems.
   * KISTI: NTR
   * KIT: NTR
   * NDGF: NTR
   * NL-T1: NTR
   * NRC-KI: NC
   * OSG: NC
   * PIC: NC
   * !RAL: NTR
   * TRIUMF: NTR

   * CERN computing services:
      * OTG:0051477 - Subset of Batch workers were affected over the weekend by connectivity issues.
      * OTG:0051185 - Hypervisor reboot schedule to mitigate MDS Intel CPU vulnerabilities
         * Will proceed by availability zone, from 18th July. Please watch OTG for updates.
         * Expect next set of CERN CEs to be rebooted on Wednesday, OTG:0051289 and OTG:0051294
   * CERN storage services:
      * CASTOR : Nameserver DB upgrade tomorrow 23rd of July, all CASTOR instances will be impacted. OTG:0051004
   * CERN databases: NC
   * GGUS: NTR
   * Monitoring:
      * Final availability reports for June 2019 sent around
   * MW Officer: NC
   * Networks: NTR
   * Security: Nothing To Report

AOB:

Topic revision: r21 - 2019-07-22 - PetrVokac

LCG Wikis

LCG Service
Coordination

LCG Grid
Deployment

LCG
Apps Area

Public webs

Welcome Guest

- Cern Search
- TWiki Search
- Google Search
LCG All webs

Copyright &© 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback