Week from 01062009 to 07062009

Job Statistics

  • Summary:
    • Almost 82K jobs run last week
    • Over 13% failed
    • Daily peak of over 21K jobs
    • 53K Production jobs run to end
    • 16.5K User jobs run to the end
    • 6.2K Production Jobs Failed
    • 4.7K User Jobs Failed

  • Total number of Jobs by Final Major Status
Total_Number_of_Jobs_by_FinalMajorStatus.png

  • Daily number of Jobs by Final Mayor Status
Daily_Number_of_Jobs_by_FinalMajorStatus.png

  • Done|Completed Jobs by User Group
Done+Complete_Jobs_by_UserGroup.png

  • Done|Completed Production Jobs by Job Type
Done+Complete_Production_Jobs_by_JobType.png

  • Failed Jobs by User Group
Failed_Jobs_by_UserGroup.png

  • Failed Production Jobs by Minor Status
Failed_Production_Jobs_by_MinorStatus.png

  • Failed User Jobs by Minor Status
Failed_User_Jobs_by_MinorStatus.png

Running at Tier1's

  • Summary:
    • 14K Production Jobs at Tier1s
      • Shares?
    • 15.4K User Jobs at Tier1s
      • 50 % CERN Share
      • comment the shares is appropriate

  • Done|Completed Production Jobs by Site
Done+Complete_Production_Jobs_at_Tier1_by_Site.png

  • Done|Completed User Jobs by Site
Done+Complete_User_Jobs_at_Tier1_by_Site.png

Job Failure Analysis

  • Summary:
    • Production Jobs Failed mostly due to "Application Finished With Errors":
      • 3371 everywhere
      • 378 mostly at CERN
    • User Jobs Failed mosty due to "Input Data Resolution":
      • 5038 everywhere
      • 4079 mostly at CERN
    • Comment about any site is appropriated

  • Failed Production Jobs (Application Finished With Error) by Site
Failed_Production_Jobs_Application_Finished_With_Errors_by_Site.png

  • Failed User Jobs (Input Data Resolution) by Site
Failed_Users_Jobs_Input_Data_Resolution_by_Site.png

* Failed Production and User Jobs (Watchdog identified this job as stalled) by Site Failed_Jobs_Watchdog_Identified_This_Job_As_stalled_by_Site.png

  • Failed Jobs at CERN by Minor Status
Failed_Jobs_at_CERN_by_MinorStatus.png

Hardware Status

  • WMS volhcb09:
    • CPU utilization: Idle < 50%?, IO Wait peaks?,
    • Network utilization: 700K peak?
    • Swap Used: do we get close to the limit 1.9GB reached?.
    • Partition Used: stable

volhcb09_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb09_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb09_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb09_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png

  • DMS volhcb10:
    • CPU utilization: Idle < 50%?, IO Wait peaks?,
    • Network utilization: above 1 M?
    • Swap Used: do we get close to the limit (2 GB)?.
    • Partition Used: Is stable?

volhcb10_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb10_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb10_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb10_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png

  • LogSE volhcb06:
    • CPU utilization: Idle < 50%?, IO Wait peaks?,
    • Network utilization: above 1 M?
    • Swap Used: do we get close to the limit (2 GB)?.
    • Partition Used: Is stable?

volhcb06_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb06_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb06_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb06_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png

  • Various volhcb01:
    • CPU utilization: Idle < 50%?, IO Wait peaks?,
    • Network utilization: above 1 M?
    • Swap Used: do we get close to the limit (2 GB)?.
    • Partition Used: Is stable?

volhcb01_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb01_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb01_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb01_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png

-- JoelClosier - 08 Jun 2009

  • Failed_Jobs_Watchdog_Identified_This_Job_As_stalled_by_Site.png:
    Failed_Jobs_Watchdog_Identified_This_Job_As_stalled_by_Site.png
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng Daily_Number_of_Jobs_by_FinalMajorStatus.png r1 manage 39.8 K 2009-06-08 - 11:44 MarcoAdinolfi  
PNGpng Done+Complete_Jobs_by_UserGroup.png r1 manage 34.2 K 2009-06-08 - 12:06 MarcoAdinolfi  
PNGpng Done+Complete_Production_Jobs_at_Tier1_by_Site.png r1 manage 61.9 K 2009-06-08 - 12:14 MarcoAdinolfi  
PNGpng Done+Complete_Production_Jobs_by_JobType.png r2 r1 manage 35.7 K 2009-06-08 - 12:08 MarcoAdinolfi  
PNGpng Done+Complete_User_Jobs_at_Tier1_by_Site.png r1 manage 53.8 K 2009-06-08 - 12:15 MarcoAdinolfi  
PNGpng Failed_Jobs_Watchdog_Identified_This_Job_As_stalled_by_Site.png r1 manage 67.9 K 2009-06-10 - 14:18 MarcoAdinolfi  
PNGpng Failed_Jobs_at_CERN_by_MinorStatus.png r2 r1 manage 47.2 K 2009-06-10 - 12:16 MarcoAdinolfi  
PNGpng Failed_Jobs_by_UserGroup.png r1 manage 35.6 K 2009-06-08 - 12:09 MarcoAdinolfi  
PNGpng Failed_Production_Jobs_Application_Finished_With_Errors_by_Site.png r1 manage 122.0 K 2009-06-08 - 12:19 MarcoAdinolfi  
PNGpng Failed_Production_Jobs_by_MinorStatus.png r2 r1 manage 59.8 K 2009-06-10 - 11:55 MarcoAdinolfi  
PNGpng Failed_User_Jobs_by_MinorStatus.png r2 r1 manage 56.9 K 2009-06-10 - 11:57 MarcoAdinolfi  
PNGpng Failed_Users_Jobs_Input_Data_Resolution_by_Site.png r1 manage 33.4 K 2009-06-08 - 12:22 MarcoAdinolfi  
PNGpng Total_Number_of_Jobs_by_FinalMajorStatus.png r1 manage 39.0 K 2009-06-08 - 11:29 JoelClosier  
PNGpng volhcb01_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 23.0 K 2009-06-08 - 12:59 MarcoAdinolfi  
PNGpng volhcb01_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 20.8 K 2009-06-08 - 13:00 MarcoAdinolfi  
PNGpng volhcb01_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 12.9 K 2009-06-08 - 13:00 MarcoAdinolfi  
PNGpng volhcb01_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png r1 manage 11.2 K 2009-06-08 - 13:00 MarcoAdinolfi  
PNGpng volhcb06_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 23.1 K 2009-06-08 - 12:57 MarcoAdinolfi  
PNGpng volhcb06_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 20.2 K 2009-06-08 - 12:57 MarcoAdinolfi  
PNGpng volhcb06_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 13.1 K 2009-06-08 - 12:58 MarcoAdinolfi  
PNGpng volhcb06_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png r1 manage 11.8 K 2009-06-08 - 12:58 MarcoAdinolfi  
PNGpng volhcb09_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 28.4 K 2009-06-08 - 12:37 MarcoAdinolfi  
PNGpng volhcb09_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.png r1 manage 28.4 K 2009-06-08 - 12:37 MarcoAdinolfi  
PNGpng volhcb09_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 22.9 K 2009-06-08 - 12:39 MarcoAdinolfi  
PNGpng volhcb09_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 13.5 K 2009-06-08 - 12:42 MarcoAdinolfi  
PNGpng volhcb09_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png r1 manage 13.8 K 2009-06-08 - 12:43 MarcoAdinolfi  
PNGpng volhcb10_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 28.5 K 2009-06-08 - 12:52 MarcoAdinolfi  
PNGpng volhcb10_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 21.3 K 2009-06-08 - 12:53 MarcoAdinolfi  
PNGpng volhcb10_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 12.3 K 2009-06-08 - 12:54 MarcoAdinolfi  
PNGpng volhcb10_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png r1 manage 11.7 K 2009-06-08 - 12:54 MarcoAdinolfi  
Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r4 - 2009-06-10 - MarcoAdinolfi
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback