Drop of MC jobs at T1 sites on the week-end, solved this morning and restarting now. DataReconstruction at T1s.
T0
T1
RAL: Problem with pilot submission to ARC-CEs, working on a fix
CNAF OUTAGE downtime declared Friday evening for LHCb storage
27th August (Thursday)
Data Processing:
p-Ne collisions reconstruction started
T0
Nothing to report
T1
RAL 'at-risk' yesterday and today due to castor update.
24th August (Monday)
Data Processing:
validation productions of 25ns data will be redone this week (ReCo + Stripping), with proper refitting of PV
current validation (Reco15b and Stripping 23a) quasi finished (>99% data processed, 75% merged)
T0
Nothing to report
T1
Nothing to report
20th August (Thursday)
Data Processing: second run of validation for 25ns data started with new calibration, ~90% already processed. New stripping production will start later today.
T0
Nothing to report
T1
Nothing to report
CVMFS problem at NIKHEF-ELPROD (GGUS:115767) due to an upgrade of their WNs from CentOS 6.6 to CentOS 6.7. Promptly solved.
PBS issues at PIC (GGUS:115748) due to a high load in few WNs, pbs server had problems to refresh information, and denied some connections. Promptly solved.
17th August (Monday)
Data Processing: validation of 25ns data started. ~70% already processed.
T0
Nothing to report
T1
Nothing to report
13th August (Thursday)
Data Processing: data "stripping" productions ongoing almost complete. Ready to process new data. User and MC jobs
T0
Nothing to report
T1
GRIDKA: Still we have problems getting TURLs from SURLs.
10th August (Monday)
Data Processing: data "stripping" productions ongoing almost complete. Ready to process new data. User and MC jobs
T0
Nothing to report
T1
RAL: Network issues over the weekend
GRIDKA: Problems over the weekend getting TURLs from SURLs. Overloaded SRM? Difficult to track down as the jobs retry/go to failover and complete.
6th August (Thursday)
Data Processing: data "stripping" productions ongoing. User and MC jobs
T0
Discussion with LSF team about wall and cpu time queue lengths, ongoing (GGUS:115027). Some progress and bugs found on both sides. Deployment of fixes happening in progress.
T1
RAL: All network problems appear to be solved.
3rd August (Monday)
Data Processing: data "stripping" productions ongoing. User and MC jobs
T0
Discussion with LSF team about wall and cpu time queue lengths, ongoing (GGUS:115027). Some progress and bugs found on both sides. Deployment of fixes happening in progress.
T1
RAL: LAN network instabilities continue, CVMFS and data access problems over the weekend.