Q3 2011

Site Service Area Date Duration Service Impact Report
IN2P3 Infrastructure Aug 26 7.5h CE 7500 job failures https://twiki.cern.ch/twiki/pub/LCG/WLCGTempArea/SIR-IN2P3-CC-PowerIncident-2011-08-26-v2.pdf
IN2P3 Infrastructure Aug 15 19h CEs CEs at 100%, others at 85% degradation https://twiki.cern.ch/twiki/pub/LCG/WLCGTempArea/SIR_CCIN2P3_15aug2011.pdf
CERN DB Aug 09 17h CASTOR CASTOR nameserver database overload https://twiki.cern.ch/twiki/bin/view/CASTORService/IncidentsnameserverDBoverload09Aug2011
CERN DB July 29 Scheduled+2h CASTOR Upgrade-related problems with stager DB (ATLAS and CMS) https://twiki.cern.ch/twiki/bin/view/CASTORService/IncidentsStagerDBUpgradeIssues04Jul2011
KIT infrastructure July 22 5d GGUS GGUS alarm emails not working 20110727GGUS_Service_Incident_Report.pdf
IN2P3 Databases July 19 3d LFC, FTS, VOMS, 3D, AMI services unavailable, some data loss https://twiki.cern.ch/twiki/pub/LCG/WLCGServiceIncidents/SIR_CCIN2P3_19july2011.pdf
KIT Storage July 12 15d ATLAS dCache 11k files lost https://twiki.cern.ch/twiki/pub/LCG/WLCGServiceIncidents/sir-kit-atlas-dcache-20110728.pdf
CERN CASTOR July 7th 4 h CASTOR Garbage Collector taking too long https://twiki.cern.ch/twiki/bin/view/CASTORService/Incidentst1transferfull07Jul2011
CERN DB July 05 Scheduled+7h CASTOR Upgrade-related problems with stager DB (ATLAS and CMS) https://twiki.cern.ch/twiki/bin/view/CASTORService/IncidentsStagerDBUpgradeIssues04Jul2011
CERN DB July 04 Scheduled+1h CASTOR Upgrade-related problems with stager DB (ATLAS and CMS) https://twiki.cern.ch/twiki/bin/view/CASTORService/IncidentsStagerDBUpgradeIssues04Jul2011

Q2 2011

Site Service Area Date Duration Service Impact Report
CERN CASTOR June 26 8 h CASTOR CMS was unable to stage files back from tape https://twiki.cern.ch/twiki/bin/view/CASTORService/IncidentsCMSnostageout26Jun2011
PIC Computing/Storage June 10 5h dCache PNFS dCache namespace overload https://twiki.cern.ch/twiki/pub/LCG/WLCGServiceIncidents/Post_Mortem_Tier-1_Service_Incident_dCache_PNFS_overload_10-June-2011_f.pdf
KIT Storage Jun 5 14 d ALICE xrootd managed storage 3% of the files unreadable https://twiki.cern.ch/twiki/pub/LCG/WLCGServiceIncidents/GridKa_SIR_lost_files_alice_20110526.pdf
CERN Infrastructure May 26 6 wks KDC high KDC load https://twiki.cern.ch/twiki/pub/LCG/WLCGTempArea/KDC-SIR.pdf
CERN CASTOR May 24 6 h DB overload on the CASTOR CMS instance Progressive degradation gradually affecting 80% of the servie https://twiki.cern.ch/twiki/bin/view/CASTORService/IncidentsCMSDbOverload24May2011
CERN VObox / Lxplus / SVN,CVS / Batch May 24 3 h XLDAP overload and nscd problem Logins blocked, access to software version control blocked, batch jobs failed https://twiki.cern.ch/twiki/bin/view/PESgroup/IncidentLdapNscd24052011
PIC Computing May 25-26 12h Batch System BS instabilities, ~600 jobs lost https://twiki.cern.ch/twiki/pub/LCG/WLCGServiceIncidents/Post_Mortem_PIC_Tier-1_SIR_Computing_SSC5_20110525.pdf
ASGC Infrastructure May 21 to May 23 36h Whole Site DC Power Cut https://twiki.cern.ch/twiki/pub/LCG/WLCGServiceIncidents/20110521_SIR_ASGC_DCPOWERCUT.pdf
CERN Batch / Lxplus / Vobox / Lxadm / Castor May 10 8 h Kerberos KDC Logins blocked, batch jobs failed, some file access blocked https://twiki.cern.ch/twiki/bin/view/PESgroup/IncidentKerberos10052011
RAL DB May 10 1h LFC Outage After Database Update >80% https://www.gridpp.ac.uk/wiki/RAL_Tier1_Incident_20110510_LFC_Outage_After_DB_Update
ASGC Network May 01 to May 08 8 days Storage service and CMS Squid Slow transfer from/to ASGC Taiwan https://twiki.cern.ch/twiki/pub/LCG/WLCGServiceIncidents/20110501_SIR_ASGC_10GbLINKDOWN.pdf
CERN DB Apr 28 1.5h CMS offline DB cluster service was down for 1.5h https://twiki.cern.ch/twiki/bin/view/DB/PostMortem28Apr11
IN2P3 Infrastructure Apr 8 5h various, incl. batch system, LFC, VOBOX job failures, various services unavailable https://twiki.cern.ch/twiki/pub/LCG/WLCGServiceIncidents/SIR-IN2P3-CC-PowerIncident-2011-04-08-v0.pdf

-- JamieShiers - 16-Sep-2011

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r1 - 2011-09-16 - JamieShiers
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback