Week from 18052009 to 24052009

Job Statistics

  • Summary:
    • Almost 56 K jobs run last week
    • Over 11.67% failed
    • Daily peak of over 14 K jobs
    • 32 K Production jobs run to end
    • 16 K User jobs run to the end
    • 3 K Production Jobs Failed
    • 2 K User Jobs Failed

  • Total number of Jobs by Final Major Status
Total_Number_of_Jobs_by_FinalMajorStatus.png

  • Daily number of Jobs by Final Mayor Status
Daily_Number_of_Jobs_by_FinalMajorStatus.png

  • Done|Completed Jobs by User Group
Done+Complete_Jobs_by_UserGroup.png

  • Done|Completed Production Jobs by Job Type
Done+Complete_Production_Jobs_by_JobType.png

  • Failed Jobs by User Group
Failed_Jobs_by_UserGroup.png

  • Failed Production Jobs by Minor Status
Failed_Production_Jobs_by_MinorStatus.png

  • Failed User Jobs by Minor Status
Failed_User_Jobs_by_MinorStatus.png

Running at Tier1's

  • Summary:
    • 9 K Production Jobs at Tier1s
      • Share
    • 11 K User Jobs at Tier1s
      • 30.92 % CERN Share
      • comment the shares is appropriate

  • Done|Completed Production Jobs by Site
Done+Complete_Production_Jobs_at_Tier1_by_Site.png

  • Done|Completed User Jobs by Site
Done+Complete_User_Jobs_at_Tier1_by_Site.png

Job Failure Analysis

  • Summary:
    • Production Jobs Failed mostly due to:
      • Application finished with Everywhere (2878)
      • Application finished Mostly at LCG.CERN.ch (356)

  • Failed Production Jobs (Application Finished With Error) by Site
Failed_Production_Jobs_Application_Finished_With_Errors_by_Site.png

  • Failed User Jobs (Input Data Resolution) by Site
Failed_Users_Jobs_Input_Data_Resolution_by_Site.png

  • Failed Jobs at GRIDKA by Minor Status
Failed_Jobs_at_GRIDKA_by_MinorStatus.png

Hardware Status

  • WMS volhcb09:
    • CPU utilization: Idle < 50%?, IO Wait peaks?,
    • Network utilization: 1.3 M to 208 k
    • Swap Used: it starts from 36.5 to 1.7GB.
    • Partition Used: For short term

volhcb09_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb09_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb09_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb09_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png

  • DMS volhcb10:
    • CPU utilization: In NICE and System
    • Network utilization: Bellow 1 M
    • Partition Used: Bellow 70G

volhcb10_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb10_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb10_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb10_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png

  • LogSE volhcb06:
    • CPU utilization: bellow 50% , and Idle
    • Network utilization: bellow 50%
    • Swap Used: yes maximum 2M.
    • Partition Used: Yes most of the time

volhcb06_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb06_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb06_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb06_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png

  • Various volhcb01:
    • CPU utilization: Idle 86.4%
    • Network utilization: above 1 M
    • Swap Used: 11.7%

-- MamunurRashid - 24 May 2009

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng Daily_Number_of_Jobs_by_FinalMajorStatus.png r1 manage 38.8 K 2009-05-25 - 14:24 MdmamunurRashid  
PNGpng Done+Complete_Jobs_by_UserGroup.png r1 manage 37.4 K 2009-05-25 - 14:30 MdmamunurRashid  
PNGpng Done+Complete_Production_Jobs_at_Tier1_by_Site.png r1 manage 64.7 K 2009-05-25 - 15:16 MdmamunurRashid  
PNGpng Done+Complete_Production_Jobs_by_JobType.png r1 manage 36.0 K 2009-05-25 - 14:56 MdmamunurRashid  
PNGpng Done+Complete_User_Jobs_at_Tier1_by_Site.png r1 manage 52.9 K 2009-05-25 - 15:20 MdmamunurRashid  
PNGpng Failed_Jobs_at_GRIDKA_by_MinorStatus.png r1 manage 46.7 K 2009-05-25 - 15:38 MdmamunurRashid  
PNGpng Failed_Jobs_by_UserGroup.png r1 manage 33.7 K 2009-05-25 - 14:35 MdmamunurRashid  
PNGpng Failed_Production_Jobs_Application_Finished_With_Errors_by_Site.png r1 manage 116.3 K 2009-05-25 - 15:23 MdmamunurRashid  
PNGpng Failed_Production_Jobs_by_MinorStatus.png r1 manage 52.6 K 2009-05-25 - 14:45 MdmamunurRashid  
PNGpng Failed_User_Jobs_by_MinorStatus.png r1 manage 47.6 K 2009-05-25 - 14:48 MdmamunurRashid  
PNGpng Failed_Users_Jobs_Input_Data_Resolution_by_Site.png r1 manage 67.1 K 2009-05-25 - 15:25 MdmamunurRashid  
PNGpng Total_Number_of_Jobs_by_FinalMajorStatus.png r1 manage 37.2 K 2009-05-25 - 14:09 MdmamunurRashid  
PNGpng img_alt=Failed_Jobs_at_GRIDKA_by_MinorStatus.png r1 manage 46.7 K 2009-05-25 - 15:30 MdmamunurRashid  
PNGpng volhcb01_1_-86400_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 22.4 K 2009-05-25 - 19:31 MdmamunurRashid  
PNGpng volhcb01_1_-86400_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 21.9 K 2009-05-25 - 19:31 MdmamunurRashid  
PNGpng volhcb01_1_-86400_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 12.7 K 2009-05-25 - 19:30 MdmamunurRashid  
PNGpng volhcb01_1_-86400_XROOT_USE_STACKEDX_1.gif.png r1 manage 10.5 K 2009-05-25 - 19:32 MdmamunurRashid  
PNGpng volhcb01_1_-86400_XVAR_USE_STACKEDX_1.gif.png r1 manage 10.9 K 2009-05-25 - 19:32 MdmamunurRashid  
PNGpng volhcb06_1_-86400_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 23.0 K 2009-05-25 - 19:43 MdmamunurRashid  
PNGpng volhcb06_1_-86400_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 21.3 K 2009-05-25 - 19:43 MdmamunurRashid  
PNGpng volhcb06_1_-86400_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 13.5 K 2009-05-25 - 19:42 MdmamunurRashid  
PNGpng volhcb06_1_-86400_SWAP_SPACE_USED_STACKEDS_1.gif.png r1 manage 12.4 K 2009-05-25 - 19:42 MdmamunurRashid  
PNGpng volhcb09_1_-86400_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 29.2 K 2009-05-25 - 15:53 MdmamunurRashid  
PNGpng volhcb09_1_-86400_CPUUTILPERCUSER_STACKEDC_1.gif.png r1 manage 15.9 K 2009-05-25 - 15:54 MdmamunurRashid  
PNGpng volhcb09_1_-86400_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 18.5 K 2009-05-25 - 15:54 MdmamunurRashid  
PNGpng volhcb09_1_-86400_NUMKBREADAVG_STACKEDN_1.gif.png r1 manage 15.4 K 2009-05-25 - 15:54 MdmamunurRashid  
PNGpng volhcb09_1_-86400_NUMKBWRITEAVG_STACKEDN_1.gif.png r1 manage 13.3 K 2009-05-25 - 15:55 MdmamunurRashid  
PNGpng volhcb09_1_-86400_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 13.9 K 2009-05-25 - 15:55 MdmamunurRashid  
PNGpng volhcb10_1_-86400_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 25.7 K 2009-05-25 - 16:00 MdmamunurRashid  
PNGpng volhcb10_1_-86400_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 20.8 K 2009-05-25 - 16:08 MdmamunurRashid  
PNGpng volhcb10_1_-86400_XROOT_USE_STACKEDX_1.gif.png r1 manage 10.4 K 2009-05-25 - 16:08 MdmamunurRashid  
PNGpng volhcb10_1_-86400_XTMP_USE_STACKEDX_1.gif.png r1 manage 9.5 K 2009-05-25 - 16:07 MdmamunurRashid  
PNGpng volhcb10_1_-86400_XVAR_USE_STACKEDX_1.gif.png r1 manage 10.8 K 2009-05-25 - 16:07 MdmamunurRashid  
Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r3 - 2009-05-25 - MdmamunurRashid
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback