WLCG Tier1 Service Coordination Minutes - March 31 2011

Attendance

• Local: Ian F, Oliver, Gavin, Stephane, Hans, Maria, Jamie, Maarten, Maite, Arne, Massim, Stefan, Eva, Giuseppe, Andrea S, Luca, Alexei, Simone, Dario, Dirk, Nicolo
• Remote: Michael, Gonzalo, Elena, Jon, Patrick, Rolf, Elizabeth, Carlos, Andrew, Felix, John de Stefano, Carmine, Jhen-Wei, Tiju, Christian, Pierre

Action list review

Release update

Data Management & Other Tier1 Service Issues

Site	Status	Recent changes	Planned changes
CERN	CASTOR 2.1.10 (all) SRM 2.10-x xrootd: ALICE 2.1.10 update 1, others: 2.1.9-7	We are going to have all SRM on 2.10-2 due to issues on 2.10-1. Central services CASTOR central services (VMGR, VDQM and Cupv) will be upgraded to version 2.1.10-1. The goal of the change is to enable support for tapes greater than 2TB
ASGC	CASTOR 2.1.7-19 (stager, nameserver) CASTOR 2.1.8-14 (tapeserver) SRM 2.8-2 DPM 1.8.0-1	31/03: 3-hour downtime for works to electrical system and construction	None
BNL	dCache 1.9.5-23 (PNFS, Postgres 9)	None	None
CNAF	StoRM 1.5.6-3 SL4 (CMS, LHCb,ALICE) StoRM 1.6 SL5 (ATLAS)	29/03: upgraded StoRM Atlas end-point to 1.6., the first one on SL5	1.6 SL5 deployment on the other end-points will be done during the next technical stop of LHC
FNAL	dCache 1.9.5-23 (PNFS) httpd=1.9.5.-25 Scalla xrootd 2.9.1/1.4.2-4 Oracle Lustre 1.8.3	DNS round robin srm service deployed today	none
IN2P3	dCache 1.9.5-24 (Chimera) on all core servers and pool nodes
KIT	dCache (admin nodes): 1.9.5-15 (Chimera), 1.9.5-24 (PNFS) dCache (pool nodes): 1.9.5-9 through 1.9.5-24
NDGF	dCache 1.9.12
NL-T1	dCache 1.9.5-23 (Chimera) (SARA), DPM 1.7.3 (NIKHEF)
PIC	dCache 1.9.5-25 (PNFS, Postgres 9)	none	none
RAL	CASTOR 2.1.10-0 2.1.9-1 (tape servers) SRM 2.8-6	Upgraded all instances to 2.1.10	none
TRIUMF	dCache 1.9.5-21 with Chimera namespace	None	None

CASTOR news

CERN operations

Development

xrootd news

dCache news

StoRM news

FTS news

• FTS 2.2.5 in certification: FTS 2.2.5 release notes
• FTS 2.2.6 in development: FTS 2.2.6 patch (see list of bugs at the end)

DPM news

• DPM 1.8.0-1 for gLite 3.1: waiting for rebuild of the meta package with the correct VOMS libraries (1.9.10-14)
  • Edinburgh verified the memory leak is gone when those libraries are installed instead of the ones currently provided by the patch

LFC news

• LFC 1.8.0-1 for gLite 3.1: waiting for rebuild of the meta package with the correct VOMS libraries (1.9.10-14)

LFC deployment

Experiment issues

WLCG Baseline Versions

• Release report: deployment status wiki page
• WLCG Baseline versions: table

Status of open GGUS tickets

GGUS - Service Now interface: update

Review of recent / open SIRs and other open service issues

Conditions data access and related services

Database services

• Experiment reports:
  • ALICE:
    • NTR
  • ATLAS:
    • New schemas have been added to the ATLAS conditions Streams replication setup on Tuesday 15th of March
  • CMS:
    • The CMS offline production database (CMSR) crashed on Friday 11th March on 10:20 due to a local power cut in the critical area of the CERN CC. Due to the power cut several disk arrays went down which in turn caused failure of the whole database. After the power was restored the database still could not be restarted due to failure of 2 disks containing mirror copies of data. A decission has been taken to fail over the service to the standby hardware. The service was restarted on the standby hardware at 12:30. Unfortunately applications connecting from outside CERN could not reach the database until 3:10 when the CERN central firewall was re-configured to allow for such access. Detailed postmortem is available here https://twiki.cern.ch/twiki/bin/view/DB/PostMortem11Mar11
    • The CMS offline production database (CMSR), running on standby hardware, failed on Tuesday 15th March at 11:45. The failure was caused by an issue in the SAN. The database has been failed over to the original hardware which it was running on until 11th of March. The service was fully restored at 13:20. The SAN issue was caused by a bug in Emulex driver, a fixed version has been identified in the newer kernel (2.6.18-238.1.1.el5).
    • Sine one week CMS PVSS replication is suffering from repetitive data latency caused by degradation of apply process performance. The root cause is not obvious and not yet known. Some extra workload from certain schemas has been observed but not big enough to saturate streams throughput. The problematic schemas have been separated from main replication by new streams configuration. Also a service request to the Oracle Support has been opened.
  • LHCb:

• Site reports:

AOB

-- JamieShiers - 30-Mar-2011