WLCG Operations Coordination Minutes, Nov 14, 2019





  • local:
  • remote:
  • apologies:

Operations News

News from the CVMFS service team at CERN

In the next two weeks, CVMFS repositories hosted at CERN will gradually start using a new signing key.
Clients accessing such repositories must have available the public keys cern-it4.cern.ch.pub and
cern-it5.cern.ch.pub at /etc/cvmfs/keys/cern.ch/. To avoid service interruptions,
sites are recommended to upgrade the CVMFS client to the latest version (2.6.3-1) at their earliest convenience.
Minimum required version is 2.1.20. If using the cvmfs-config-default package, version 1.3-1 is the minimum required.
Sites in EGI or OSG can also switch to cvmfs-config-egi or cvmfs-config-osg, respectively.

Special topics

Central deployment of WLCG services and WLCG SLATE security WG

Middleware News

Tier 0 News

Tier 1 Feedback

Tier 2 Feedback

Experiments Reports


  • Sites please plan moving to CentOS / EL 7 as soon as possible
  • Normal to high activity
    • High analysis train activity in preparation for Quark Matter 2019, Nov 3-9, Wuhan
    • Continuing afterwards
  • No major issues


  • Smooth Grid production over the past weeks with ~330k concurrently running grid job slots with the usual mix of MC generation, simulation, reconstruction, derivation production and user analysis. In addition ~80k job slots from the HLT/Sim@CERN-P1 farm.
  • No major issues apart from the usual storage or transfer related problems at sites.
  • We are recommending to sites not use Letís Encrypt certificates
  • Follow-up discussions with Tier1 sites and dCache developers on data/tape carousel campaign
  • Plan to add Kubernetes to the set of critical services for ATLAS in https://twiki.cern.ch/twiki/bin/view/LCG/WLCGCritSvc#ATLAS


  • smooth running up to 260k cores during the last month
    • UltraLegacy re-processing
      • 2017: almost complete
      • 2018: RAW data staging has started
      • 2016: Only after 2017+2018 are finished
    • B-parked data reconstruction in the tails
    • completed 2018 heavy-ion data processing
    • large core usage end of October due to HPC resources/NERSC
    • fire protection at P5 restored and high-level trigger farm now available again
  • currently no plans to use/support Let's Encrypt certificates
  • do we all have a close enough understanding what we mean with the entries in the WLCG service catalogue?


  • smooth usage of grid resources in the last month, with an average of 101k jobs running
  • Main activity: Monte Carlo production (88%), user jobs (6%), Monte Carlo reconstruction (4%), work group productions (0.5%)
  • Small issues
    • file access problems and deleting files on ECHO at RAL
  • Singularity support
    • Still waiting for IN2P3
  • T2-D sites gradually migrating to CC7

Task Forces and Working Groups

Upgrade of the T1 storage instances for TPC

GDPR and WLCG services

Accounting TF

Archival Storage WG

Containers WG

CREAM migration TF

dCache upgrade TF

DPM upgrade TF

  • 25 sites upgraded and re-configured with DOME. Out of those 4 are not updated in CRIC
  • 9 upgraded but to be re-configured
  • 7 sites are migrating to different storage solutions
  • 12 still to go (upgrade and re-configuration). Most of them plan to accomplish the task by the end of this year

Information System Evolution TF

IPv6 Validation and Deployment TF

Detailed status here.

Machine/Job Features TF


MW Readiness WG

Network Throughput WG

Squid Monitoring and HTTP Proxy Discovery TFs

  • These TFs are finally nearing completion. The last few tasks should be completed soon.

Traceability WG

Action list

Creation date Description Responsible Status Comments

Specific actions for experiments

Creation date Description Affected VO Affected TF/WG Deadline Completion Comments

Specific actions for sites

Creation date Description Affected VO Affected TF/WG Deadline Completion Comments


This topic: LCG > WebHome > WLCGCommonComputingReadinessChallenges > WLCGOperationsWeb > WLCGOpsCoordination > WLCGOpsMinutes191114
Topic revision: r10 - 2019-11-14 - JuliaAndreeva
This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback