Week of 170814

WLCG Operations Call details

  • At CERN the meeting room is 513 R-068.

  • For remote participation we use the Vidyo system. Instructions can be found here.

General Information

  • The purpose of the meeting is:
    • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
    • to announce or schedule interventions at Tier-1 sites;
    • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
    • to provide important news about the middleware;
    • to communicate any other information considered interesting for WLCG operations.
  • The meeting should run from 15:00 until 15:20, exceptionally to 15:30.
  • The SCOD rota for the next few weeks is at ScodRota
  • General information about the WLCG Service can be accessed from the Operations Web
  • Whenever a particular topic needs to be discussed at the daily meeting requiring information from site or experiments, it is highly recommended to announce it by email to wlcg-operations@cernSPAMNOTNOSPAMPLEASE.ch to make sure that the relevant parties have the time to collect the required information or invite the right people at the meeting.

Tier-1 downtimes

Experiments may experience problems if two or more of their Tier-1 sites are inaccessible at the same time. Therefore Tier-1 sites should do their best to avoid scheduling a downtime classified as "outage" in a time slot overlapping with an "outage" downtime already declared by another Tier-1 site supporting the same VO(s). The following procedure is recommended:

  1. A Tier-1 should check the downtimes calendar to see if another Tier-1 has already an "outage" downtime in the desired time slot.
  2. If there is a conflict, another time slot should be chosen.
  3. In case stronger constraints cannot allow to choose another time slot, the Tier-1 will point out the existence of the conflict to the SCOD mailing list and at the next WLCG operations call, to discuss it with the representatives of the experiments involved and the other Tier-1.

As an additional precaution, the SCOD will check the downtimes calendar for Tier-1 "outage" downtime conflicts at least once during his/her shift, for the current and the following two weeks; in case a conflict is found, it will be discussed at the next operations call, or offline if at least one relevant experiment or site contact is absent.

Links to Tier-1 downtimes




  • local: Michal (ATLAS), Kate (Chair, DB), Vincent (security), Jesus (storage), Julia (WLCG), Raja (LHCb), Vladimir (LHCb), Marian (network),
  • remote: Jens (NDGF), Kyle (OSG), Stefano (CMS), Sang Un (KISTI), Dave M (FNAL), Chihsun (ASGC), Gareth (RAL)

Experiments round table:

  • ATLAS reports ( raw view) -
    • Activities:
      • normal activities
      • overlay jobs - require a lot of conditions and can cause frontier overload - limited to 800 running score jobs
    • Problems
      • BNL stageout failures (GGUS:129978) - network misconfiguration - solved
      • lost heartbeats at RAL-LCG2-ECHO_MCORE (GGUS:129998) - under investigation.

  • CMS reports ( raw view) -
    • T0 had problems with EOS (ongoing), lack of space (~solved) and slow outgoing transfers (improved). Otherwise a normal week.
      • T0 A few serious issues with EOS are endangering operations. Ad hoc meetings held with CERN EOS experts, thanks for support !
        • failure to open files "machine not on the network" GGUS:129607 (ALARM) at times file download from EOS to WN fails since too little data can be transfered in the 3min window due to EOS congestion. CMSSW change needed (smaller data chunks), Brian made PR last week
        • files disappear from EOS GGUS:129937 possibly understood by EOS developers (?) a configuration change for CMS may have helped, still ongoing.
        • directory disappeared from EOS GGUS:129988 maybe same as previous one (?), still ongoing - namespace was restarted today to solve the issue
    • T1_US_FNAL slow in getting CERN data, affecting T0 space needs. GGUS:130020 Doing better now, but not fully satisfactory yet.

  • ALICE -
    • Apologies: ALICE operations will not attend today
    • Very high activity
    • CERN: Stratum-0 host cvmfs-alice-ocdb.cern.ch was down (GGUS:129993)
      • Thanks to all experts involved for the fast resolution!
    • CERN: HTCondor job status monitoring frequency has been lowered by a factor 10
      • To reduce the CPU load on the VOBOXes as well as the batch service

  • LHCb reports ( raw view) -
    • Activity
      • Monte Carlo simulation, data processing and user analysis
    • Site Issues
      • T0:
        • Problem with installation of python possibly broken on multiple WNs (GGUS:130018) - ongoing issue
      • T1:
        • Problems uploading to various SEs - For SARA, tracked in GGUS:129946. Now also seen in FZK, IN2P3 and PIC - to be tracked and tickets opened if needed. Uploads fail from workernodes to T1s

Sites / Services round table:

  • ASGC: ntr
  • BNL: nc
  • CNAF: NTR (written on Friday, no one will participate from CNAF on monday)
  • EGI: nc
  • FNAL: Downtime on Thursday morning to move FTS DB to more powerful hardware as it grew due to a missing "lock" file. Only FTS will be affected.
  • IN2P3: nc
  • JINR: nc
  • KISTI: ntr
  • KIT: nc
  • NDGF: ntr
  • NL-T1: nc
  • NRC-KI: nc
  • OSG: Issues with ticket assignment was solved and it was tracked to a secret DB.
  • PIC: nc
  • RAL: ntr
  • TRIUMF: nc

  • CERN computing services: nc
  • CERN storage services: ntr
  • CERN databases:
    • Latest stability and security patches will be deployed on castor databases hosts during the next technical stop, according to the following schedule:
      Databases Hosts Schedule
      alicestg/atlasstg/lhcbstg itrac5209/itrac5249/itrac5238 Mon 18.09 - 9:00/12:00
      cmsstg itrac5215/itrac5255 Wed 20.09 - 9:00/12:00
    • CMS databases will be patched. Final dates to be proposed when technical stop is confirmed
  • GGUS:
    • intermittent e-mail engine issues since the Remedy upgrade are being investigated
      • some ticket updates may arrive late occasionally
  • Monitoring:
    • Final reports for the July availability sent around
  • MW Officer: nc
  • Networks:
    • RNP (Brazilian NREN) reported that they're investigating CBPF transfer problems to CNAF, PIC and IN2P3 - CBPF related ticket for LHCb is GGUS:129561. A request was made to add WLCG networking to such tickets.
  • Security: NTR


Edit | Attach | Watch | Print version | History: r13 < r12 < r11 < r10 < r9 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r13 - 2017-08-14 - MaartenLitmaath
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback