WLCG Tier1 Service Coordination Minutes - 7 July 2011

Attendance

• Local: Elisa, MariaA, MariaD, MariaG, Steve, Massimo, AndreaV, AndreaS, Cedric, Joel, Maarten, Jamie, Simone, Nicolo, Zsolt, Alessandro, Daniele)

• Remote: Stefane, Ron, Pavel, Felix, Alexander, Carlos, Michael, DaveD, John DeStefano, Cristina, Ulf, Luca, Gonzalo, Kyle, Gareth), Jhen Wei)

Action list review

Release update

Data Management & Other Tier1 Service Issues

CASTOR news

CERN operations

The CERNT3 and ATLAS deployments use the new internal scheduling. In testing prior to deployment we have validated the system to handle 5 times more requests per seconds than the original LSF-based scheduler. With ATLAS we validated the system in conditions at the level (or slighlty exceeding) the top-rate operations experienced. ATLAS and CASTOR operations agreed to continue to operate the system in the continuation of the LHC 2011 run.

Development

EOS news

• CMS imported approx. all CAF data into EOSCMS 0.5 M files and 0.6 PB . CMS switched CAF to EOS with fallback to CASTOR.
• ATLAS import is at 4.5 M files 0.9 PB (~300.000 files still to import).

xrootd news

dCache news

StoRM news

FTS news

• FTS 2.2.5 in gLite Staged Rollout: http://glite.cern.ch/staged_rollout
• FTS 2.2.6 in development: FTS 2.2.6 patch (see list of bugs at the end)

DPM news

• DPM 1.8.1 in EGI UMD Staged Rollout: changes
• A wlcg-preview repository has been set up which contains new middleware versions which have been extensively tested but have not been certified in a formal way neither for gLite nor for EMI
• Testing DPM version "1.8.2" (wlcg-preview release) with the help of ASGC. The main features coming with this version are:
  • fast dpm-drain
  • better filesystem selection algorithm (designed by Maarten)
  • central banning (Argus)

LFC news

• LFC 1.8.1 in EGI UMD Verification
• A wlcg-preview repository has been set up which contains new middleware versions which have been extensively tested but have not been certified in a formal way neither for gLite nor for EMI
• Currently running LFC version "1.8.2" (wlcg-preview release) at CERN in production for LHCb and it is also the version installed for the Atlas Central LFC entering production soon at CERN. Compared to 1.8.0, this LFC version fixes a problem reported by IN2P3 when running the replica instance of LFC for LHCb. This version also brings support for central banning (Argus)

LFC deployment

Site	Version	OS, n-bit	Backend	Upgrade plans
NDGF	1.7.4.7-1	Ubuntu 9.10 64-bit	MySQL	None
TRIUMF	1.7.3-1	SL5 64-bit	MySQL
FNAL	N/A			Not deployed at Fermilab
ASGC	1.8.0-1	SLC5 64-bit	Oracle	None
BNL	1.8.0-1	SL5, 64-bit	Oracle	None
CERN	1.7.3 64-bit	SLC4	Oracle	Upgrade to SLC5 64-bit pending
CNAF	1.7.4-7 (ATLAS, to be dismissed> 1.8.0-1 (LHCb, recently updated)	SL5 64-bit	Oracle
KIT	1.7.4-7	SL5 64-bit	Oracle	Oracle backend migration pending
NL-T1	1.7.4-7	CentOS5 64-bit	Oracle
PIC	1.7.4-7	SL5 64-bit	Oracle
RAL	1.7.4-7	SL5 64-bit	Oracle
IN2P3	1.8.0-1	SL5 64-bit	Oracle 11g	Oracle DB migrated to 11g on Feb. 8th

Experiment issues

WLCG Baseline Versions

• Release report: deployment status wiki page
• WLCG Baseline versions: table

GGUS issues

Review of recent / open SIRs and other open service issues

Conditions data access and related services

• A new LCGCMT_60c has been prepared for ATLAS. The motivation for this release is the upgrade to newer versions of all Persistency Framework packages (CORAL, COOL, POOL) and of the ROOT and oracle external dependencies, with multiple fixes and improvements in all of the above. The largest changes are in the new CORAL (2.3.16), including in particular several fixes for the most serious problems observed in its handling of network and database connection glitches. The new ROOT (5.28.00e) includes a more robust fix for the xrootd/kerberos issue responsible for the KDC flood in the last few weeks, while the new Oracle client configuration (11.2.0.1.0p3) includes a workaround for another kerberos-related bug in the Oracle client, that redefines symbols conflicting with those in the system libraries. The full release notes are available at https://twiki.cern.ch/twiki/bin/view/Persistency/PersistencyReleaseNotes.
• A new frontier client version 2.8.2 has been released, including some performance optimizations that split into chunks the data received via http. The new client is now used in the CMS HLT system. While it was produced too late for its inclusion in the LCG60c release for ATLAS, this is now being tested in the CORAL nightlies and will be included in the next CORAL release prepared for ATLAS offline.

Database services

• Experiment reports:
  • ALICE:
  • ATLAS:
    • Instance 2 of Atlas Offline (ADCR) database has rebooted on Friday (01.07) around 00:50 due to ORA-00600 error. Database instance was back in operation after few minutes. Rootcause of this problem was an Oracle bug.
    • New schema has been added to the ATLAS AMI Streams replication setup from IN2P3 to CERN on Tuesday 05.07. Due to some troubles at IN2P3 replication downtime has been extended to ~2h.
    • Some intervention by ATLAS developer on one of ATLAS conditions data schema caused aborts of replication on Wednesday (06/07) at midnight.
    • Following up on ATLAS request, replication for Atlas Conditions to PiC will be discontinued on Tuesday 12.07 14:00.
  • CMS:
    • Node 2 of CMS Offline database has rebooted on Monday 27th of June around 7:50PM due to excessive memory consumption caused by one of CMS applications (Phedex). Database instance was back in operation after few minutes. Rootcause of this problem is under investigation.
    • CMS integration database (INT2R) has been successfully upgaded to Oracle 11.2.0.2 on Tuesday 5th of July afternoon.
  • LHCb:
    • Misconfiguration of Advance Queuing in LHCb replication to SARA has been fixed on Monday 27th (10AM). Whole replication LHCb T1s streaming had to be stopped for 1 hour.

• Site reports:

AOB

-- AndreaSciaba - 06-Jul-2011