Week from 28062009 to 05072009

Job Statistics

  • Summary:
    • Almost 170 K jobs run last week
    • Over 24% failed
    • Daily peak of over 35 K jobs
    • 103 K Production jobs run to end
    • 27 K User jobs run to the end
    • 22 K Production Jobs Failed
    • 8 K User Jobs Failed

  • Total number of Jobs by Final Major Status
Total_Number_of_Jobs_by_FinalMajorStatus.png

  • Daily number of Jobs by Final Mayor Status
Daily_Number_of_Jobs_by_FinalMajorStatus.png

  • Done|Completed Jobs by User Group
Done+Complete_Jobs_by_UserGroup.png

  • Done|Completed Production Jobs by Job Type
Done+Complete_Production_Jobs_by_JobType.png

  • Failed Jobs by User Group
Failed_Jobs_by_UserGroup.png

  • Failed Production Jobs by Minor Status
Failed_Production_Jobs_by_MinorStatus.png

  • Failed User Jobs by Minor Status
Failed_User_Jobs_by_MinorStatus.png

Running at Tier1's

  • Summary:
    • 19 K Production Jobs at Tier1s
      • 36 % CERN
      • 24 % GRIDKA
      • 13 % NIKHEF
      • 12 % PIC
      • 8 % CNAF
      • 7 % IN2P3
    • 26 K User Jobs at Tier1s
      • 54 % CERN
      • 20 % NIKHEF
      • 13 % IN2P3
      • 9 % GRIDKA
      • 2 % PIC
      • 2 % CNAF

  • Done|Completed Production Jobs by Site
Done+Complete_Production_Jobs_at_Tier1_by_Site.png

  • Done|Completed User Jobs by Site
Done+Complete_User_Jobs_at_Tier1_by_Site.png

Job Failure Analysis

  • Summary:
    • Production Jobs Failed mostly due to:
      • Application finished with errors
      • Pending requests
      • Input Sandbox Download
    • User Jobs Failed mosty due to:
      • Input Data resolution
      • Application finished with errors

  • Failed Production Jobs (Application Finished With Error) by Site
Failed_Production_Jobs_Application_Finished_With_Errors_by_Site.png

  • Failed User Jobs (Input Data Resolution) by Site
Failed_Users_Jobs_Input_Data_Resolution_by_Site.png

  • Failed Jobs at GRIDKA by Minor Status
Failed_Jobs_at_GRIDKA_by_MinorStatus.png

  • Failed Jobs at CERN by Minor Status
Failed_Jobs_at_CERN_by_MinorStatus.png

  • Failed Jobs at CNAF by Minor Status
Failed_Jobs_at_CNAF_by_MinorStatus.png

Hardware Status

  • WMS volhcb09:

volhcb09_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb09_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb09_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb09_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png

  • DMS volhcb10:

volhcb10_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb10_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb10_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb10_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png

  • LogSE volhcb06:

volhcb06_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb06_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb06_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb06_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png

  • Various volhcb01:

volhcb01_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb01_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb01_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb01_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png

-- AlexeyZhelezov - 6 Jul 2009

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng Daily_Number_of_Jobs_by_FinalMajorStatus.png r1 manage 45.7 K 2009-07-06 - 12:53 AlexeyZhelezov  
PNGpng Done+Complete_Jobs_by_UserGroup.png r1 manage 34.4 K 2009-07-06 - 12:53 AlexeyZhelezov  
PNGpng Done+Complete_Production_Jobs_at_Tier1_by_Site.png r1 manage 61.3 K 2009-07-06 - 12:53 AlexeyZhelezov  
PNGpng Done+Complete_Production_Jobs_by_JobType.png r1 manage 31.5 K 2009-07-06 - 12:53 AlexeyZhelezov  
PNGpng Done+Complete_User_Jobs_at_Tier1_by_Site.png r1 manage 53.0 K 2009-07-06 - 12:54 AlexeyZhelezov  
PNGpng Failed_Jobs_at_CERN_by_MinorStatus.png r1 manage 59.4 K 2009-07-06 - 12:54 AlexeyZhelezov  
PNGpng Failed_Jobs_at_CNAF_by_MinorStatus.png r1 manage 51.3 K 2009-07-06 - 12:54 AlexeyZhelezov  
PNGpng Failed_Jobs_at_GRIDKA_by_MinorStatus.png r1 manage 42.5 K 2009-07-06 - 12:55 AlexeyZhelezov  
PNGpng Failed_Jobs_by_UserGroup.png r1 manage 34.7 K 2009-07-06 - 12:55 AlexeyZhelezov  
PNGpng Failed_Production_Jobs_Application_Finished_With_Errors_by_Site.png r1 manage 119.3 K 2009-07-06 - 12:55 AlexeyZhelezov  
PNGpng Failed_Production_Jobs_by_MinorStatus.png r1 manage 59.2 K 2009-07-06 - 12:55 AlexeyZhelezov  
PNGpng Failed_User_Jobs_by_MinorStatus.png r1 manage 59.2 K 2009-07-06 - 12:56 AlexeyZhelezov  
PNGpng Failed_Users_Jobs_Input_Data_Resolution_by_Site.png r1 manage 43.0 K 2009-07-06 - 12:56 AlexeyZhelezov  
PNGpng Total_Number_of_Jobs_by_FinalMajorStatus.png r1 manage 35.6 K 2009-07-06 - 12:56 AlexeyZhelezov  
PNGpng volhcb01_1_-86400_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 20.7 K 2009-07-06 - 12:57 AlexeyZhelezov  
PNGpng volhcb01_1_-86400_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 13.3 K 2009-07-06 - 12:57 AlexeyZhelezov  
PNGpng volhcb01_1_-86400_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 11.8 K 2009-07-06 - 12:57 AlexeyZhelezov  
PNGpng volhcb01_1_-86400_SWAP_SPACE_USED_STACKEDS_1.gif.png r1 manage 11.8 K 2009-07-06 - 12:58 AlexeyZhelezov  
PNGpng volhcb06_1_-86400_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 27.6 K 2009-07-06 - 12:58 AlexeyZhelezov  
PNGpng volhcb06_1_-86400_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 16.1 K 2009-07-06 - 12:58 AlexeyZhelezov  
PNGpng volhcb06_1_-86400_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 13.8 K 2009-07-06 - 12:59 AlexeyZhelezov  
PNGpng volhcb06_1_-86400_SWAP_SPACE_USED_STACKEDS_1.gif.png r1 manage 12.1 K 2009-07-06 - 12:59 AlexeyZhelezov  
PNGpng volhcb09_1_-86400_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 25.6 K 2009-07-06 - 12:59 AlexeyZhelezov  
PNGpng volhcb09_1_-86400_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 14.9 K 2009-07-06 - 13:00 AlexeyZhelezov  
PNGpng volhcb09_1_-86400_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 13.1 K 2009-07-06 - 13:00 AlexeyZhelezov  
PNGpng volhcb09_1_-86400_SWAP_SPACE_USED_STACKEDS_1.gif.png r1 manage 11.6 K 2009-07-06 - 13:01 AlexeyZhelezov  
PNGpng volhcb10_1_-86400_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 27.7 K 2009-07-06 - 13:01 AlexeyZhelezov  
PNGpng volhcb10_1_-86400_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 22.2 K 2009-07-06 - 13:01 AlexeyZhelezov  
PNGpng volhcb10_1_-86400_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 13.5 K 2009-07-06 - 13:01 AlexeyZhelezov  
PNGpng volhcb10_1_-86400_SWAP_SPACE_USED_STACKEDS_1.gif.png r1 manage 13.7 K 2009-07-06 - 13:02 AlexeyZhelezov  
Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r1 - 2009-07-06 - AlexeyZhelezov
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback