WLCG Tier1 Service Coordination Minutes - 16 June 2011

Attendance

Action list review

Release update

Data Management & Other Tier1 Service Issues

CASTOR news

• CASTOR 2.1.11-0 Release: release notes
• CASTOR SRM 2.11-0 Release: release notes
• CASTOR xrootd 2.1.11 Release: release notes

CERN operations

Development

EOS news

xrootd news

dCache news

StoRM news

FTS news

• FTS 2.2.5 in gLite Staged Rollout: http://glite.cern.ch/staged_rollout
• FTS 2.2.6 in development: FTS 2.2.6 patch (see list of bugs at the end)

DPM news

• DPM 1.8.1 in EGI UMD Staged Rollout: changes

LFC news

• LFC 1.8.1 in EGI UMD Verification

LFC deployment

Experiment issues

WLCG Baseline Versions

• Release report: deployment status wiki page
• WLCG Baseline versions: table

Status of open GGUS tickets

Review of recent / open SIRs and other open service issues

Conditions data access and related services

Database services

• Experiment reports (long list covering issues since 5th of May):
  • ALICE:
  • ATLAS:
    • First instance of Atlas offline database (ADCR) has crashed on Sunday (08.05). Issue has been caused by internal database error and is now being under investigation. Services were available on the surviving nodes while the instance restarted and were relocated back to instance one after it came back into operation.
    • We had three hangs of ATLAS offline DB (ADCR) during which the service was not available: Monday 16th between 16:25 and 17:10, Monday 16th between 21:50 and 23:30 and Tuesday 17th between 1:50 a.m. and 2:40 a.m. No data loss occurred except for all uncommitted data. All incidents were caused by unusual reaction of ASM on a broken disk (itstor737 disk 3). ASM did not properly initiated a rebalance operation during the first incident and was affected by some problems during second and third. After the incidents a normal rebalance has finished and we were trying to forcefully evict the problematic disk. SR has been opened on this issue.
    • On 18th of May around 11:20 ADCR DB experienced another disk failure during rebalancing operation which did not finish. Decision was taken to switchover the DB to the standby cluster. Switchover completed successfully after several minor issues and the DB was back operational at 13:05. IN2P3 reported that AMI applications are not able to reach the DB. It turned out that DB was not visible outside of CERN. We requested the port on the firewall to be opened and it was done the next day (19th of May in the morning).
    • The INT8R Atlas integration database has been migrated to Oracle 11g (version 11.2.0.2) on Monday (30.05).
    • Atlas offline (ADCR) database has been switched back to orginal hardware on Tuesday (31.05). Switchover operation required 1h database downtime.
    • ADCR database, hosting Atlas DQ2, PANDA and PRODSYS services has suffered full downtime on Monday (06.06) from 13:20 till 14:00. The cause of downtime has been a DB hang caused by the ASM layer following a double disk failure occurred in the morning. A full restart of the DB and ASM unblocked the hang. We are following up with Oracle support on the causes of the hang. We are also investigating on the rate of disk failures observed in this system recently (2 disks have failed over the long WE in addition to the 2 failed on Monday)
    • There has been a failure of one of redundatant Fibre Channel Switches on 09.06 to which storages of ATLAS offline DB and ATLAS online DB are connected. Issue is now being investigated by FC-sunsupport. We are also discussing with ATLAS a schedule for replacing the swich.
  • CMS:
    • 3rd node of the CMS offline production database (CMSR) rebooted on 14th of June at 14:45, most likely due to an issue with HBA's firmware or driver. Due to the reboot some sessions of several CMS applications failed. Fortunately none of the 3 main applications running on the DB were affected. The machine re-joined the cluster around 15:00.
  • LHCb:
    • Streams issues - see below
  • Streams:
    • Due to SARA’s Advance Queuing configuration problems which were caused by January migration of the database back to RAC (consequence of the August 2010 storage failure), replication of LHCB conditions data was frozen from Saturday (22.05) midnight until Monday 11 o’clock. Problem was temporarily fixed by migration of buffered queue to different instance. Permanent solution is being discussed with Oracle Support.
    • After Tuesday (24.05 10:00) SARA’s database upgrade to 10.2.0.5 both LHCb replication (conditions data and LFC) were hit by streams related Oracle bug. Solution: Patches fixing the bug has been applied.
    • After extended intervention on Tuesday (24.05 7:00) on database storage at IN2P3, LFC replica became partially inconsistent - at least one transaction was missing. After skipping some other transactions, streams managed to continue replication without further errors. Full LFC data consistency at IN2P3 has been restored on Wednesday afternoon (25.05).
    • Replication to SARA was down from Sunday 29.05 (around 9:45 am) to Monday 30.05 (around 10:00 am) duew to a spanning tree problem in a part of SARA's network. The replica at SARA was out of sync during the period of the issue which could potentially cause some problems for the jobs using this replica.
    • The replication of ATLAS conditions data to BNL hanged twice: Monday night and Tuesday night for 6 hours. Root cause of the problem is gathering statistics job which due to cross cluster contention blocks processing of big transactions by streams components. The temporal workaround is to kill manually the locking job. This problem is being investigated by Oracle Support without any results so far.

• Site reports:

AOB

-- AndreaSciaba - 15-Jun-2011