Real time downstream was not set for LFC replication

Description

After SARA database recovery LFC replication was restored without real time downstream turned on.

Impact

• LFC replication was suffering from latency problems between 10.09 and 14.09

Time line of the incident

• Wednesday 8th of September, 14:00 - SARA replication was restored (for both ATLAS and LHCB)
• Friday 10th of September, 16:30 - SARA replication added back to the main setup
• Tuesday 14th of September, 17:00 - Real time downstream parameter re-enabled for LFC replication

Analysis

LFC replication in opposite to LHCB replication has special 'real time downstream' optimization turned on in order to minimize replication latency through downstream database. It looks that during recovery of SARA it has disabled itself because of two capture processes working for the same set of data on LHCBDSC database.

-- MarcinBlaszczyk - 27-Sep-2010