WLCG Tier1 Service Coordination Minutes - 20th January 2011

Attendance

local(); remote ();

Release update

Data Management & Other Tier1 Service Issues

Site	Status	Recent changes	Planned changes
CERN	CASTOR 2.1.10 (CMS, ATLAS and ALICE) CASTOR 2.1.9-9 (LHCb) SRM 2.9-4 (all) xrootd 2.1.9-7	Oracle changes (all instances): upgrade on Jan the 6th to 10.2.0.5	LHCb will be upgraded to 2.1.10
ASGC	CASTOR 2.1.7-19 (stager, nameserver) CASTOR 2.1.8-14 (tapeserver) SRM 2.8-2	14/1: 4h "at risk" intervention on tape system due to construction of electrical power system in data center	None
BNL	dCache 1.9.5-23 (PNFS, Postgres 9)	1.9.4 upgraded to 1.9.5-23; PG 8.3 to PG 9	None
CNAF	StoRM 1.5.6-3 (ATLAS, CMS, LHCb,ALICE)		upgrade OS to SL5 within February
FNAL	dCache 1.9.5-23 (PNFS) Scalla xrootd 2.9.1/1.4.2-4	None	Putting Lustre into Production Service for Merging Pools
IN2P3	dCache 1.9.5-22 (Chimera)
KIT	dCache 1.9.5-15 (admin nodes) (Chimera) dCache 1.9.5-5 - 1.9.5-15 (pool nodes)
NDGF	dCache 1.9.7 (head nodes) (Chimera) dCache 1.9.5, 1.9.6 (pool nodes)
NL-T1	dCache 1.9.5-23 (Chimera) (SARA), DPM 1.7.3 (NIKHEF)
PIC	dCache 1.9.5-23 (PNFS)
RAL	CASTOR 2.1.9-6 (stagers) 2.1.9-1 (tape servers) SRM 2.8-6	Upgraded ATLAS disk servers to SL5 64bit and enabled checksum support	Will next upgrade CMS disk servers and enable checksum support. Will upgrade Oracle to 10.2.0.5 on 1 Feb 2011
TRIUMF	dCache 1.9.5-21 with Chimera namespace	None	None

CASTOR news

CERN operations

Development

xrootd news

dCache news

StoRM news

• StoRM 1.6.0 released for Early Adopters for SL5 X86_64 on Jan 13. Changelog is here.

FTS news

DPM news

• DPM 1.8.0 released for gLite 3.2 / SL5 on Jan 18, including these highlights:
  • new VOMS library fixing memory leaks in SRM daemons
  • facility to ban users and groups (VOMS attributes)
• DPM 1.8.0 for gLite 3.1 / SL4 still in Staged Rollout:
  • http://glite.cern.ch/staged_rollout
  • Edinburgh already upgraded
  • should be released to production soon

LFC news

• LFC 1.8.0 for gLite 3.1 / SL4 and 3.2 / SL5 still in Staged Rollout
  • new VOMS library fixing memory leaks in LFC daemon
  • facility to ban users and groups (VOMS attributes)
  • http://glite.cern.ch/staged_rollout
  • BNL already upgraded
  • should be released to production soon

LFC deployment

Site	Version	OS, n-bit	Backend	Upgrade plans
TRIUMF	1.7.3-1	SL5 64-bit	MySQL
FNAL	N/A			Not deployed at Fermilab
NDGF
ASGC	1.7.4-7	SLC5 64-bit	Oracle	None (upgrade done on 4/1)
BNL	1.8.0-1	SL5, 64-bit	Oracle
CERN	1.7.3 64-bit	SLC4	Oracle	Will upgrade to SLC5 64-bit by the end of Jan or begin of Feb.
CNAF	1.7.4-7	SL5 64-bit	Oracle
IN2P3	1.8.0-1	SL5 64-bit	Oracle	Upgraded to LFC 1.8.0 on January 4th
KIT	1.7.4	SL5 64-bit	Oracle
NL-T1	1.7.4-7	CentOS5 64-bit	Oracle
PIC	1.7.4-7	SL5 64-bit	Oracle
RAL	1.7.4-7	SL5 64-bit	Oracle

Experiment issues

Status of open GGUS tickets

GGUS - Service Now interface

Review of open SIRs

Conditions Data Access and related services

Database services

• Experiment reports:
  • Generic:
    • LHCb and ATLAS downstream capture databases have been patched to 10.2.0.5 on Wednesday 5th of Jan.
  • ALICE:
    • Online DB upgraded to 10.2.0.5 on 12 of Jan
    • Online DB not available between 19th of Jan 16h and 20th of Jan 19h due to power tests in the pit.
  • ATLAS:
    • Atlas offline database for ADC applications (PANDA, DQ2 and prodsys accounts) have been moved to a dedicated database on Monday 17th of Jan 2011. The operation has involved a scheduled downtime for the affected accounts from 9am till 6pm and has included a full backup to tape of the new DB accounts. The rest of Atlas offline DB, in particular conditions and PVSS have been untouched.
    • Atlas online DB has been upgraded to 10.2.0.5 on Wednesday afternoon (19th of Jan).
    • After the CC power cut on 18th of December, the ATLARC database did not want to reboot. The investigation showed, that one of online log files was corrupted. This file has been already archived by the database but it turned out that it was archived corrupted. That looks like a database bug, since a corrupted online log file should not be archived successfully. In the consequence, the restore of the database from the backup was needed. The recovery operation was possible only to the point in time few hours before the power cut (7.30 a.m.), because TSM service lost some backup data after power cut and the existing archivelog was corrupted. The users agreed to restore the DB to the point in time several hours before, without waiting for TSM to be able to recover more data.
  • CMS:
    • CMS online DB was not available between 8:25 and 19:10 on the 4th of January, due to power cut in P5. Manual interventions were require to start the DB after power was back (~17h).
    • Maintenance activities over the weekend on schemas replicated by CMS PVSS streaming introduced aborts and high latency of the replication. To avoid further problems other changes has been done manually on the target database - 17th of Jan
    • CMS online production database was stopped twice during last week on Monday and Tuesday morning (10-11 Jan). First downtime was necessary due to power tests at P5. At the same time several disks critical for database operation were replaced with new ones. Second downtime was related to upgrade to 10.2.0.5. The upgrade was completed successfully.
    • CMS offline database has been upgraded to version 10.2.0.5 on Wednesday afternoon (19th of Jan).
  • LHCb:
    • LHCb online DB was not available between 0:10 and 19h on the 20th of Dec due to scheduled power tests in LHCb pit.

• Site reports:

Action List

AOB