Week of 180723

WLCG Operations Call details

  • For remote participation we use the Vidyo system. Instructions can be found here.

General Information

  • The purpose of the meeting is:
    • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
    • to announce or schedule interventions at Tier-1 sites;
    • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
    • to provide important news about the middleware;
    • to communicate any other information considered interesting for WLCG operations.
  • The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
  • The SCOD rota for the next few weeks is at ScodRota
  • General information about the WLCG Service can be accessed from the Operations Portal
  • Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes



  • local: Julia (WLCG), Kate (chair, WLCG, DB), Gavin (computing), Vincent (security), Borja (monitoring), Alberto (monitoring), Belinda (storage), Marian (network)
  • remote: Marcelo (CNAF), Stefano (LHCb), Dmytro (NDGF), Alexander (NL-T1), Di (Triumf), Christoph (CMS), Sang Un (KISTI), Dave M (FNAL), Victor (JINR), Darren (RAL), Vincenzo (EGI)

Experiments round table:

  • ATLAS reports ( raw view) -
    • Overall - no major issues and smooth running.
      • Grid running with around 260k-330k
        • Mixture of simulation (major part) and some data reprocessing
    • Problems:
      • Quiet week with a few smaller issues
      • Issue at INFN-T1 which had no production at the beginning of the week (now back to normal)
    • If comments : crc.shifter@cernNOSPAMPLEASE.ch

  • CMS reports ( raw view) -
    • High CPU usage remains
      • 240k cores (~190k for Production, 60k for Analysis)
    • File access issue at T1_FR_CCIN2P3 (GGUS:135931) appears solved

  • ALICE -
    • Apologies: ALICE operations experts will not attend today
    • NTR at least until today

  • LHCb reports ( raw view) -
    • Activity
      • Data reconstruction for 2018 data
      • User and MC jobs
    • Site Issues
      • CERN: File transfers problems. Looks like it is related to a problematic FTS server. Under investigation ( GGUS:136275 )

Sites / Services round table:

  • ASGC: nc
  • BNL: nc
  • EGI: NTR in addition to MW officer report
  • FNAL: Fri-Sat downtime went smoothly
  • IN2P3: nc
  • JINR:
    • Number of pilots dropped from usual 600 to 400 in the weekend. Investigating.
    • CEs were rebooted after update.
  • KIT: nc
  • NL-T1: NTR
  • NRC-KI: nc
  • OSG: nc
  • PIC: nc
  • RAL: Power testing work starting in RAL data centre this week. There are warning in GocDB.
  • TRIUMF: New DDN storage arrived and were installed at new data centre, will start to migrate data there after verifying it.

  • CERN computing services: NTR
  • CERN storage services: nothing to report.
  • CERN databases: NTR
  • GGUS:
    • A new release is planned for Wed this week
      • Release notes
      • A downtime has been scheduled for 05:30-07:00 UTC
      • Test alarms will be submitted as usual
      • The certificate used for signing alarm emails will be renewed (should be transparent)
  • Monitoring:
    • Final Site Availability June report sent to WLCG Office.
  • MW Officer:
    • UMD-4 updates on July 11 broke the SL6 CREAM CE
      • Tomcat could not start with the newer versions of canl-java, bouncy-castle and voms-api-java
      • This was not caught in the Staged Rollout, because CREAM itself was not updated
        • Will be handled better in the future
      • Several high-priority tickets were opened, e.g. GGUS:136074
      • CREAM developers quickly provided fixes in their own repository
        • To be included in a UMD-4 emergency update
      • Tickets were updated with workaround recipes in the meantime
  • Networks:
    • GGUS:135962 - Transfers from FNAL to DESY failing due to timeout. Initial investigation didn't confirm a network issue, SE suspected.
    • Oak Ridge National Lab (ALICE T2) to CERN - Network performance issues reported on LHCONE, identified as MTU mismatch
Christoph asked if the FNAL-DESY issue was investigated. Marian confirmed it was checked with PerfSonar by both Shawn and Eduardo and no network issues were confirmed (network was slow, but not to the point of issues). Marian can redo the tests now. Christoph confirmed the issue is still ongoing, Dave confirmed that FNAL team is working on it. Christoph asked to check for possible 8'000s timeouts as there is a possible pattern.

  • Security: NTR


Edit | Attach | Watch | Print version | History: r18 < r17 < r16 < r15 < r14 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r18 - 2018-07-23 - MaartenLitmaath
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback