WLCG Tier1 Service Coordination Minutes - 17 March 2011

Attendance

Action list review

Release update

Data Management & Other Tier1 Service Issues

Site	Status	Recent changes	Planned changes
CERN	CASTOR 2.1.10 (all) SRM 2.9-4 (ALICE, CMS, LHCb), SRM 2.10 (ATLAS) xrootd 2.1.9-7	SRM 2.10 rolled out for ATLAS, having issues, investigating them
ASGC	CASTOR 2.1.7-19 (stager, nameserver) CASTOR 2.1.8-14 (tapeserver) SRM 2.8-2	17/2: unscheduled downtime from 06:50 to 09:40 UTC due to a network issue which caused data transfers from EU and US te be slow	1/3: "at risk" intervention from 06:00 to 10:00 UTC on tape system for data centre construction works; tapes may be inaccessible, D1T0 data unaffected
BNL	dCache 1.9.5-23 (PNFS, Postgres 9)	None	None
CNAF	StoRM 1.5.6-3 (ATLAS, CMS, LHCb,ALICE)
FNAL	dCache 1.9.5-23 (PNFS) Scalla xrootd 2.9.1/1.4.2-4 Oracle Lustre 1.8.3	None	None
IN2P3	dCache 1.9.5-24 (Chimera) on all core servers and pool nodes	Upgrade from 1.9.5-22 to 1.9.5-24 on 2011-02-08	None
KIT	dCache (admin nodes): 1.9.5-15 (Chimera), 1.9.5-24 (PNFS) dCache (pool nodes): 1.9.5-9 through 1.9.5-24
NDGF	dCache 1.9.12
NL-T1	dCache 1.9.5-23 (Chimera) (SARA), DPM 1.7.3 (NIKHEF)
PIC	dCache 1.9.5-24 (PNFS, Postgres 9)
RAL	CASTOR 2.1.9-6 (stagers) 2.1.9-1 (tape servers) SRM 2.8-6	NS now upgraded to 2.1.10-0	during March we want to upgrade: castor clients on WNs to 2.1.9-6 and stagers to 2.1.10-0
TRIUMF	dCache 1.9.5-21 with Chimera namespace

CASTOR news

— CERN operations

— Development

xrootd news

dCache news

StoRM news

FTS news

• FTS 2.2.5 in certification: FTS 2.2.5 release notes
• FTS 2.2.6 in development: FTS 2.2.6 patch (see list of bugs at the end)

DPM news

• DPM 1.8.0-1 for gLite 3.1: waiting for rebuild of the meta package with the correct VOMS libraries (1.9.10-14)
  • Edinburgh involved as Early Adopter site

LFC news

• LFC 1.8.0-1 for gLite 3.1: waiting for rebuild of the meta package with the correct VOMS libraries (1.9.10-14)

LFC deployment

Experiment issues

WLCG Baseline Versions

• WLCG Baseline versions: table

Status of open GGUS tickets

GGUS - Service Now interface: update

Review of recent / open SIRs and other open service issues

Conditions data access and related services

COOL, CORAL and POOL

• A new LCGCMT_60b has been prepared for ATLAS. The main motivation for this release is the upgrade to newer versions of the ROOT (5.28.00b) and Qt (4.6.3p2) external dependencies. The new release also includes a few bug fixes enhancements in CORAL and POOL, while it is based on the same COOL code base as LCGCMT_60a. The full release notes are available at https://twiki.cern.ch/twiki/bin/view/Persistency/PersistencyReleaseNotes

Frontier/Squid

• ATLAS weekly Frontier meetings

Database services

• Experiment reports:
  • ALICE:
    • Intervention on Monday 7th from 9:15 to 10:15 in order to fix an electricity problem with one of the power boxes used by the Alice online database. The database was unavailable during the intervention.
  • ATLAS:
    • Muon Calibration data replication from 3 Muon sites: Michigan, Rome and Munich to CERN (ATLAS offline database) moved to production.
    • New schemas have been added to the ATLAS conditions Streams replication setup on Tuesday 15th March.
  • CMS:
    • CMS integration database affected by Oracle bug 7612454: More "direct path read" operations/OERI:kcblasm. Fix patch applied and tested. Recently, same issue affecting one application on CMS offline production database (CMSR). Waiting for green light from CMS to apply it in production (rolling intervention).
    • CMS online production database got stuck during a scheduled rolling reboot on Friday 4th February at 14:10, as part of the disk replacement. The hang was caused by an ASM locking and the database was unavailable for the users during 45 min.
    • CMS offline production database (CMSR) crashed on Friday 11th March on 10:20 due to a local power cut in the critical area of the CERN CC. Several disk arrays went down which in turn caused failure of the whole database. After the power was restored the database still could not be restarted due to failure of 2 disks containing mirror copies of data. The service was restarted on the standby hardware at 12:30. Unfortunately applications connecting from outside CERN could not reach the database until 3:10 when the CERN central firewall was re-configured to allow for such access. Detailed postmortem is available here https://twiki.cern.ch/twiki/bin/view/DB/PostMortem11Mar11
    • CMS offline production database (CMSR), running on standby hardware, failed on Tuesday 15th March at 11:45. The failure was caused by an issue in the SAN which is still being investigated. The database has been failed over to the original hardware. The service was fully restored at 13:20.
  • LHCB:
    • On Saturday 26th of February around 1am, the LHCB offline production database hung due to a problem with the controller for one of disk arrays attached to it. The service was fully recovered by 2:10am. The faulty controller has been replaced by the vendor.

• Site reports:

AOB

-- JamieShiers - 14-Mar-2011