WLCG Operations Coordination Minutes - May 22nd, 2014

Agenda

• https://indico.cern.ch/event/313440/

Attendance

• local: Maria Alandes (chair), Nicolo Magini (secretary), Alberto Aimar, Simone Campana (ATLAS), Zbigniew.Baranowski, Marian Babik, Oliver Keeble, Maarten Litmaath (ALICE), Hassen Riahi
• remote: Christoph Wissing (CMS), Di Qing (TRIUMF), Jeremy Coles, Kyle Gross (OSG), Maite Barroso (Tier-0), Massimo Sgaravatto, Thomas Hartmann (KIT), Burt Holzmann (FNAL)

News

• 2014 WLCG Workshop in Barcelona (7-9 July):
  • The WLCG workshop agenda is now available in Indico. We will start contacting the speakers to define the contents and details of each talk, and we will also contact the experiments to start thinking about the experiment session. We would like to see part of each experiment session dedicated to the long term future and hear about computing model evolution for Run3/Run4.
  • Please, register to the conference if you are planning to come! Registration deadline is 9th of June.

• Simone comments that the ATLAS presentation for computing model evolution for Run3/Run4 would be more about questions than solutions.
• Alberto proposes a standard skeleton for experiment presentations, unlike last year when each experiment used a different format.

Middleware news and baseline versions

https://twiki.cern.ch/twiki/bin/view/LCG/WLCGBaselineVersions

• Introducing Andrea Manzi who will take over the maintenance of the baseline versions as WLCG Middleware Officer
• The latest DPM 1.8.8 has issues with FTS2 transfers (currently affecting only some CMS sites), more news in the DM section.

Tier-0 and Tier-1 Grid services

Storage deployment

• Burt comments that the upgrade of the FNAL tape instance is basically complete, tests are OK and the instance will be opened up later today.
• Nicolo asks if the compatibility issues between dCache 2.6 and Enstore, which were previously preventing the upgrade at PIC, have been solved. To be followed offline since PIC is not connected.
  • After the meeting, PIC confirms that they have been testing dCache 2.6 with the new Enstore software and everything is working as expected. The upgrade to dCache 2.6 is tentatively scheduled for June 10th, site downtime to be announced next week.

FTS deployment

• Now tracking FTS3 servers at RAL and CERN, which are running the latest release candidate.

LFC deployment

Oracle deployment

• Note: only Oracle instances with a direct impact on offline computing activities of LHC experiments are tracked here
• Note: an explicit entry for specific instances is needed only during upgrades, listing affected services. Otherwise sites may list a single entry.

• Zbigniew comments that the remaining Tier-1 upgrades are tied to the migration to the new GoldenGate replication technology. A workshop is scheduled in two weeks, agenda linked below.
  • https://indico.cern.ch/event/315902/other-view?view=standard

T0 news

• Quattor phase-out: CERN is currently migrating all centrally managed services from Quattor to a new Puppet based Configuration Management system. This migration is meant to be fully finished by 31st October 2014. On that date, all components of the Quattor infrastructure (CDB, CDBSQL, CDBWeb, SINDES, SWREP, SMS, LEAF tools and CLUMAN) will stop working.
• SHA-2 Certificates have been automatically added to all users in the 4 LHC VOs.

Other site news

Data management provider news

• DPM 1.8.8 was released last week including a new gridftp component, which has issues with the gridftp implementation in FTS2 (currently still used only by CMS); FTS3 transfers are OK. Since CMS is anyway pushing to switch to FTS3, the affected sites which have already upgraded are asked to change their PhEDEx configuration to use FTS3. Sites which have not yet upgraded are encouraged to wait for the DPM fix which is in testing and is probably going to be released next week.
• Nicolo will inform the CMS Tier-2s through the appropriate mailing list.

Experiments operations review and Plans

ALICE

• activities for Quark Matter 2014 have ramped down
• high production activity has taken over
• CERN
  • SLC6 job efficiencies:
    • various data analytics and comparison efforts ongoing
    • new VOBOX has been set up to target physical SLC6 hosts only

ATLAS

• MC production and analysis: stable load in the past week
  • MC prod workload is available till the start of DC14 (mid/endJune), but single-core only
  • occasional multi-core validation tasks
• rucio full chain testing starting now. Ramping up in the next 4/6 weeks to stress test the Rucio components. Still proper monitoring missing, more news in one week from now at the ADC weekly. This will not impact in terms of data transfers the normal ATLAS activities, and thus not even the other experiments nor sites.
• migration of sites from LFC to Rucio: all clouds migrated but US, which is ongoing. The CERN LFC is not used anymore. We will discuss with CERN-IT in the coming week to snaphot the DB and close the frontends.
• issue about rfc proxy support in condor cream submission. GGUS:105188

• About the catalog migration, Maarten suggests to keep the LFC running in parallel to compare during the migration and debug issues recently seen at NDGF about files that were never written on the storage but were in the ATLAS catalogs. Simone comments that this can be done for the next weeks, but in the meantime they can plan the decommissioning campaign.
• Condor is using a very old version of the CREAM client, the issue can be fixed repackaging with a recent version; the fix is currently in testing in the dev trunk. Simone and Maarten suggest to follow condor in the Middleware Readiness Working Group now that it is used as common middleware by two experiments. Maria opens an ACTION item on the middleware readiness working group to include condor and report at the next meeting about the status.

CMS

• High priority Production and Processings
  • Heavy Ion MC (almost done now)
  • Upgrade MC
  • CSA14 preparation (13 TeV MC)
• Made CMS SAM test for glexec critical on May 19th
• SAM test for xrootd fallback
  • Not yet critical
  • Still waiting (mainly) for RAL to fix some issues
• FTS3 for Phedex Debug transfers becoming mandatory now
  • Have sent tickets to sites
• Need to deprecate Savannah to GGUS bridge in GGUS Mai release (May 26th)
  • Relies on old CMS siteDB API, which is being decommissioned June 3rd
  • Changing CMS shifter instructions to use GGUS directly
  • Operations effort is moving to GGUS anyway

LHCb

• 5 GGUS tickets in total (3 x pilot problems, 1 x glitch in cvmfs, 1 x "brazilian proxies")
• Mostly MonteCarlo productions and user analysis during the last 2 weeks
• Problem with brazilian certificates not able to access dCache storage was understood and will be fixed by the product team

Ongoing Task Forces and Working Groups

Tracking tools evolution TF

• GGUS release on the 26th. The alarms for UK and USA will be done on the 27th. The rest, on the 26th
• The Savannah-GGUS bridge for CMS will be decomissioned in this release

FTS3 Deployment TF

• CMS opening GGUS tickets to sites to complete migration to FTS3 in PhEDEx Debug.

gLExec deployment TF

• 83 tickets closed and verified, 12 still open (-3)
• Deployment tracking page

• No follow up yet with ATLAS and LHCb on the adoption of glExec.
• On a related topic, Maarten comments that there have been many tickets for ARGUS instabilities recently, but the experts are actively following up, hopefully a solution will be found soon.

Machine/Job Features

• NTR

Middleware readiness WG

• 4th meeting took place on May 15
  • agenda
  • minutes will be announced
• next meeting on Wed July 2, 16:00-17:30 CEST
  • agenda

• Maarten comments that an internally developed solution is going to be used to track middleware versions instead of pakiti, since this would need to be extended anyway. A prototype is expected to be ready by the next meeting, to be deployed initially on the sites involved in middleware validation.
• Marian asks if there is a plan to extend the monitoring to the entire infrastructure. Maarten answers that this has been a long-time goal; the monitoring will be demonstrated on the validation infrastructure and then proposed for wider adoption e.g. at the MB.

Multicore deployment

• NTR

SHA-2 Migration TF

• introduction of the new VOMS servers
  • blocking issue - job submission to CREAM fails when the proxy was signed by a VOMS server with a SHA512 host certificate (GGUS:104768)
  • the fix has been put into the EMI-3 third-party repo on May 16
    • bouncycastle-mail-1.46-2
  • no official announcement yet about which node types are affected
    • CREAM and/or UI
    • others?
  • we also need the fix to become available in UMD
  • all sites then need to update their affected hosts
  • we will define a new timeline accordingly
• RFC proxies
  • ATLAS discovered that Condor still uses an old CREAM client that does not support RFC proxies
    • their pilot factories thus need to keep using legacy proxies for the time being
    • this matter will be followed up with the Condor devs
  • CMS intended to switch their SAM preprod instance to RFC proxies
    • that should still work now, but would fail when WMS submission is replaced by Condor-G submission

• Maria asks about the criticality of the issue with job submission to CREAM. Maarten comments that is top priority because it's blocking the introduction of the new VOMS servers; however currently users are not affected because the new VOMS servers are not yet in production.
• About the RFC proxy issue, Nicolo comments that he's not aware about such issues in the CMS CRAB3 tests, which are also using RFC proxies with Condor. Possibly related by different usage of Condor by the two VOs.

WMS decommissioning TF

• Maite asks about the timeline of the SAM WMS decommissioning by the end of June. Marian answers that the deployment of the new CondorG probes is late, currently finalizing the CMS probes with Andrea Sciaba'. The new probes will be in SAM preprod by beginning of June, so realistically the commissioning will complete by end of August.
• Maite proposes to check back on the status in mid summer; the hard deadline is 31st of October since the SAM WMS machines are Quattor-managed.

IPv6 validation and deployment TF

• We are asking sites to fill in the site survey: https://www.gridpp.ac.uk/wiki/2014_IPv6_WLCG_Site_Survey
• This should be edittable by anyone with a grid certificate but if they have problems please email responses to ipv6@hepixNOSPAMPLEASE.org.
• We would like as many sites as possible (and all Tier 1s) to fill it in before June 10th.
• Recent activity from IN2P3 who have joined the IPv6 testbed.

HTTP proxy discovery TF

• NTR

Network and transfer metrics WG

• Mandate:
  • Ensure all relevant network and transfer metrics are identified, collected and published
  • Ensure sites and experiments can better understand and fix networking issues
  • Enable use of network-aware tools to improve transfer efficiency and optimize experiment workflows
• Objectives:
  • Identify and continuously make available relevant transfer and network metrics
  • Document metrics and their use
  • Facilitate their integration in the middleware and/or experiment tool chain
  • Coordinate commissioning and maintenance of WLCG network monitoring
• Homepage at https://twiki.cern.ch/twiki/bin/view/LCG/NetworkTransferMetrics; linked from WLCG OPS coordination page
  • Added initial information, mandate and objectives proposed at the last WLCG ops coordination meeting
• Mailing list: wlcg-ops-coord-wg-metrics@cernNOSPAMPLEASE.ch
• Meetings:
  • Thu 22nd May, Internal meeting held in LAPP, Annecy
• Discussed members to invite and started contacting them
• Proposing to organize the work in two sub-groups:
  • Deployment, commissioning and maintenance of the tools providing network and transfer metrics (FAX, AAA, FTS, perfSONAR, etc.) - versions and configuration tracking (parameter tuning), operational issues, etc.
  • Higher-level services that will make use of the provided metrics (PhEDEx, Panda, Rucio) - technical aspects of existing metrics (latency, measurement methodology, API access, etc.), identification of missing metrics, data analytics on archived data
• Discussed current issues and short-term tasks in the perfSONAR deployment and commissioning objective:
  • Identified near-term operational tasks needed to finalize deployment of perfSONAR (tracking the versions, identifying and resolving firewall issues, etc.)
  • Discussed plans for the OSG data store hosting and archiving perfSONAR data
• Next steps:
  • Invite all the proposed members and organize kick-off meeting

• Concerning perfSONAR, Marian mentions that a new version is coming out with a REST API.
• Maria asks about the kick-off meeting for the working group; Marian answers that the date will be agreed when all WG members are invited, the aim is for June.

Action list

1. NEW on the middleware readiness working group: include Condor in the readiness verification; report at the next meeting about the status of the issue with RFC proxies.
2. NEW on the WLCG monitoring team: status of the CondorG probes for SAM to be able to decommission SAM WMS

AOB

• Reminder: the next WLCG Operations Coordination will be on June 5th

-- NicoloMagini - 19 May 2014