Week from 15062009 to 21062009

Job Statistics

  • Summary:
    • Almost 205 K jobs run last week
    • Over 4.7% failed
    • Daily peak of over 1.2 K jobs
    • 150 K Production jobs run to end
    • 14 K User jobs run to the end
    • 7 K Production Jobs Failed
    • 1.5 K User Jobs Failed

  • Total number of Jobs by Final Major Status
Total_Number_of_Jobs_by_FinalMajorStatus.png

  • Daily number of Jobs by Final Mayor Status
Daily_Number_of_Jobs_by_FinalMajorStatus.png

  • Done|Completed Jobs by User Group
Done+Complete_Jobs_by_UserGroup.png

  • Done|Completed Production Jobs by Job Type
Done+Complete_Production_Jobs_by_JobType.png

  • Failed Jobs by User Group
Failed_Jobs_by_UserGroup.png

  • Failed Production Jobs by Minor Status
Failed_Production_Jobs_by_MinorStatus.png

*Comment*(Copied from previous week report): The BK-LFC mismatch are failures in the WMS because the FEST data of this week was registered incorrectly in the BK/LFC by the RunDB

  • Failed User Jobs by Minor Status
Failed_User_Jobs_by_MinorStatus.png

Running at Tier1's

  • Summary:
    • 52 K Production Jobs at Tier1s
      • shares as previous week
    • 11 K User Jobs at Tier1s
      • 22 % CERN Share
      • shares as previous week
comment CNAF in Downtime since Friday Chosen site is not eligible Error

  • Done|Completed Production Jobs by Site
Done+Complete_Production_Jobs_at_Tier1_by_Site.png

  • Done|Completed User Jobs by Site
Done+Complete_User_Jobs_at_Tier1_by_Site.png

Job Failure Analysis

  • Summary:
    • Production Jobs Failed mostly due to:
      • Application error everywhere
      • Watchdog killed job as Stalled at Sheffield, PDC, Bristol-HPC
    • User Jobs Failed mosty due to:
      • Input Data Resolution at CERN, CNAF, GRIDKA, et al

  • Failed Production Jobs (Application Finished With Error) by Site
Failed_Production_Jobs_Application_Finished_With_Errors_by_Site.png

  • Failed User Jobs (Input Data Resolution) by Site
Failed_Users_Jobs_Input_Data_Resolution_by_Site.png

  • Failed Jobs at CERN by Minor Status
Failed_Jobs_at_CERN_by_MinorStatus.png

  • Failed Jobs at CNAF by Minor Status
Failed_Jobs_at_CNAF_by_MinorStatus.png

  • Failed Jobs at GRIDKA by Minor Status
Failed_Jobs_at_GRIDKA_by_MinorStatus.png

  • Failed Jobs at IN2P3 by Minor Status
Failed_Jobs_at_IN2P3_by_MinorStatus.png

  • Failed Jobs at NIKHEF by Minor Status
Failed_Jobs_at_NIKHEF_by_MinorStatus.png

  • Failed Jobs at PIC by Minor Status
Failed_Jobs_at_PIC_by_MinorStatus.png

  • Failed Jobs at RAL by Minor Status
Failed_Jobs_at_RAL_by_MinorStatus.png

  • Failed Production Watchdog Jobs by Site
Failed_Production_Jobs_WatchDog.png
Comment Half Watchdog killed jobs have Application status Executing finalization, Sheffield and PDC were banned

Hardware Status

  • WMS volhcb09:
    • IO Wait peaks 16, 17 and 18 June, last two coincident with disk IO activity

volhcb09_1_-86400_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb09_1_86400_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb09_1_-86400_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb09_1_-86400_SWAP_SPACE_USED_STACKEDS_1.gif.png

  • DMS volhcb10:
    • Swap Used: May be It was reloaded 17 June?

volhcb10_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb10_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb10_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb10_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png

  • LogSE volhcb06:
    • Partition Used: Save ROOT files to LogSE is bad idea

volhcb06_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb06_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb06_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb06_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png

  • Various volhcb01:
    • Swap Used: Reloaded?

volhcb01_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb01_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb01_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb01_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png

-- VladimirRomanovskiy - 22 Jun 2009

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng Daily_Number_of_Jobs_by_FinalMajorStatus.png r1 manage 43.9 K 2009-06-22 - 12:17 VladimirRomanovskiy  
PNGpng Done+Complete_Jobs_by_UserGroup.png r1 manage 34.2 K 2009-06-22 - 12:19 VladimirRomanovskiy  
PNGpng Done+Complete_Production_Jobs_at_Tier1_by_Site.png r1 manage 65.0 K 2009-06-22 - 13:03 VladimirRomanovskiy  
PNGpng Done+Complete_Production_Jobs_by_JobType.png r1 manage 35.6 K 2009-06-22 - 12:21 VladimirRomanovskiy  
PNGpng Done+Complete_User_Jobs_at_Tier1_by_Site.png r1 manage 64.2 K 2009-06-22 - 13:04 VladimirRomanovskiy  
PNGpng Failed_Jobs_at_CERN_by_MinorStatus.png r1 manage 50.0 K 2009-06-22 - 13:53 VladimirRomanovskiy  
PNGpng Failed_Jobs_at_CNAF_by_MinorStatus.png r1 manage 55.6 K 2009-06-22 - 13:53 VladimirRomanovskiy  
PNGpng Failed_Jobs_at_GRIDKA_by_MinorStatus.png r1 manage 55.2 K 2009-06-22 - 13:54 VladimirRomanovskiy  
PNGpng Failed_Jobs_at_IN2P3_by_MinorStatus.png r1 manage 56.3 K 2009-06-22 - 13:54 VladimirRomanovskiy  
PNGpng Failed_Jobs_at_NIKHEF_by_MinorStatus.png r1 manage 51.0 K 2009-06-22 - 13:54 VladimirRomanovskiy  
PNGpng Failed_Jobs_at_PIC_by_MinorStatus.png r1 manage 61.3 K 2009-06-22 - 13:55 VladimirRomanovskiy  
PNGpng Failed_Jobs_at_RAL_by_MinorStatus.png r1 manage 57.9 K 2009-06-22 - 13:55 VladimirRomanovskiy  
PNGpng Failed_Jobs_by_UserGroup.png r1 manage 35.1 K 2009-06-22 - 12:23 VladimirRomanovskiy  
PNGpng Failed_Production_Jobs_Application_Finished_With_Errors_by_Site.png r1 manage 120.6 K 2009-06-22 - 13:40 VladimirRomanovskiy  
PNGpng Failed_Production_Jobs_WatchDog.png r1 manage 61.9 K 2009-06-22 - 12:52 VladimirRomanovskiy  
PNGpng Failed_Production_Jobs_by_MinorStatus.png r1 manage 64.6 K 2009-06-22 - 12:25 VladimirRomanovskiy  
PNGpng Failed_User_Jobs_by_MinorStatus.png r1 manage 67.8 K 2009-06-22 - 12:28 VladimirRomanovskiy  
PNGpng Failed_Users_Jobs_Input_Data_Resolution_by_Site.png r1 manage 54.1 K 2009-06-22 - 13:46 VladimirRomanovskiy  
PNGpng Total_Number_of_Jobs_by_FinalMajorStatus.png r1 manage 33.2 K 2009-06-22 - 12:14 VladimirRomanovskiy  
PNGpng volhcb01_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 24.5 K 2009-06-22 - 15:34 VladimirRomanovskiy  
PNGpng volhcb01_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 15.5 K 2009-06-22 - 15:34 VladimirRomanovskiy  
PNGpng volhcb01_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 12.7 K 2009-06-22 - 15:34 VladimirRomanovskiy  
PNGpng volhcb01_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png r1 manage 11.4 K 2009-06-22 - 15:35 VladimirRomanovskiy  
GIFgif volhcb06_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif r1 manage 27.2 K 2009-06-22 - 15:15 VladimirRomanovskiy  
PNGpng volhcb06_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 27.2 K 2009-06-22 - 15:28 VladimirRomanovskiy  
GIFgif volhcb06_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif r1 manage 16.8 K 2009-06-22 - 15:15 VladimirRomanovskiy  
PNGpng volhcb06_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 16.8 K 2009-06-22 - 15:28 VladimirRomanovskiy  
GIFgif volhcb06_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif r1 manage 13.6 K 2009-06-22 - 15:16 VladimirRomanovskiy  
PNGpng volhcb06_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 13.6 K 2009-06-22 - 15:29 VladimirRomanovskiy  
GIFgif volhcb06_1_0_SWAP_SPACE_USED_STACKEDS_1.gif r1 manage 12.8 K 2009-06-22 - 15:17 VladimirRomanovskiy  
PNGpng volhcb06_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png r1 manage 12.8 K 2009-06-22 - 15:29 VladimirRomanovskiy  
PNGpng volhcb09_1_-86400_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 28.8 K 2009-06-22 - 14:15 VladimirRomanovskiy  
PNGpng volhcb09_1_-86400_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 23.6 K 2009-06-22 - 14:19 VladimirRomanovskiy  
PNGpng volhcb09_1_-86400_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 13.7 K 2009-06-22 - 14:19 VladimirRomanovskiy  
PNGpng volhcb09_1_-86400_SWAP_SPACE_USED_STACKEDS_1.gif.png r1 manage 14.3 K 2009-06-22 - 14:20 VladimirRomanovskiy  
PNGpng volhcb10_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 27.6 K 2009-06-22 - 15:08 VladimirRomanovskiy  
PNGpng volhcb10_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 21.0 K 2009-06-22 - 15:08 VladimirRomanovskiy  
PNGpng volhcb10_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 12.3 K 2009-06-22 - 15:09 VladimirRomanovskiy  
PNGpng volhcb10_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png r1 manage 12.0 K 2009-06-22 - 15:09 VladimirRomanovskiy  
Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2009-06-22 - VladimirRomanovskiy
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback