Week from 22062009 to 28062009

Job Statistics

  • Summary:
    • Almost 122 K jobs run last week
    • Over 7.7% failed
    • Daily peak of over 1.5 K jobs
    • 97 K Production jobs run to end
    • 15 K User jobs run to the end
    • 4 K Production Jobs Failed
    • 5 K User Jobs Failed

  • Total number of Jobs by Final Major Status
Total_Number_of_Jobs_by_FinalMajorStatus.png

  • Daily number of Jobs by Final Mayor Status
Daily_Number_of_Jobs_by_FinalMajorStatus.png

  • Done|Completed Jobs by User Group
Done+Complete_Jobs_by_UserGroup.png

  • Done|Completed Production Jobs by Job Type
Done+Complete_Production_Jobs_by_JobType.png

  • Failed Jobs by User Group
Failed_Jobs_by_UserGroup.png

  • Failed Production Jobs by Minor Status
Failed_Production_Jobs_by_MinorStatus.png

  • Failed User Jobs by Minor Status
Failed_User_Jobs_by_MinorStatus.png

Running at Tier1's

  • Summary:
    • 20 K Production Jobs at Tier1s
      • 35% CERN
      • 22% GRIDKA
      • 14% CNAF
    • 14 K User Jobs at Tier1s
      • 40% CERN
      • 18% GRIDKA
      • 17% IN2P3
      • 15% NIKHEF
      • comment: RAL in DownTime

  • Done|Completed Production Jobs by Site
Done+Complete_Production_Jobs_at_Tier1_by_Site.png

  • Done|Completed User Jobs by Site
Done+Complete_User_Jobs_at_Tier1_by_Site.png

Job Failure Analysis

  • Summary:
    • Production Jobs Failed mostly due to:
      • Application finished with error everywhere
      • Uploading Job Outputs mostly at
      • Watchdog identified this job as stalled Sheffield, PDC, Bristol-HPC, RAL-HEP
    • User Jobs Failed mosty due to:
      • Application finished with error everywhere
      • Input Data Resolution mostly at CNAF, CERN
    • Comment: RAL in DownTime

  • Failed Production Jobs (Uploading_Job_Outputs_by_Site) by Site
Failed_Production_Jobs_Uploading_Job_Outputs_by_Site.png
Failed_Production_Jobs_Uploading_Job_Outputs_by_Site_Time.png

* Failed Production Jobs (Watchdog_Stalled_by_Site) by Site Failed_Production_Jobs_Watchdog_Stalled_by_Site.png
Failed_Production_Jobs_Watchdog_Stalled_by_Site_Time.png

  • Failed User Jobs (Input Data Resolution) by Site
Failed_Users_Jobs_Input_Data_Resolution_by_Site.png

  • Failed Jobs at CERN by Minor Status
Failed_Jobs_at_CERN_by_MinorStatus.png
  • Failed Jobs at CNAF by Minor Status
Failed_Jobs_at_CNAF_by_MinorStatus.png
  • Failed Jobs at GRIDKA by Minor Status
Failed_Jobs_at_GRIDKA_by_MinorStatus.png
  • Failed Jobs at IN2P3 by Minor Status
Failed_Jobs_at_IN2P3_by_MinorStatus.png
  • Failed Jobs at NIKHEF by Minor Status
Failed_Jobs_at_NIKHEF_by_MinorStatus.png
  • Failed Jobs at PIC by Minor Status
Failed_Jobs_at_PIC_by_MinorStatus.png

Hardware Status

  • WMS volhcb09:

volhcb09_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb09_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb09_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb09_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png

  • DMS volhcb10:

volhcb10_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb10_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb10_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb10_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png

  • LogSE volhcb06:
    • CPU utilization: Multiple IO Wait peaks
    • Network utilization: above 1 M (27-06-09 - backup logs?)
    • Partition Used: Ii was stable and close to limit

volhcb06_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb06_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb06_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb06_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png

  • Various volhcb01:

volhcb01_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png volhcb01_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png
volhcb01_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png volhcb01_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng Daily_Number_of_Jobs_by_FinalMajorStatus.png r1 manage 45.9 K 2009-06-29 - 10:28 VladimirRomanovskiy  
PNGpng Done+Complete_Jobs_by_UserGroup.png r1 manage 35.8 K 2009-06-29 - 10:28 VladimirRomanovskiy  
PNGpng Done+Complete_Production_Jobs_at_Tier1_by_Site.png r1 manage 57.5 K 2009-06-29 - 10:50 VladimirRomanovskiy  
PNGpng Done+Complete_Production_Jobs_by_JobType.png r1 manage 32.5 K 2009-06-29 - 10:29 VladimirRomanovskiy  
PNGpng Done+Complete_User_Jobs_at_Tier1_by_Site.png r1 manage 58.0 K 2009-06-29 - 10:51 VladimirRomanovskiy  
PNGpng Failed_Jobs_at_CERN_by_MinorStatus.png r1 manage 54.5 K 2009-06-29 - 11:51 VladimirRomanovskiy  
PNGpng Failed_Jobs_at_CNAF_by_MinorStatus.png r1 manage 41.2 K 2009-06-29 - 11:51 VladimirRomanovskiy  
PNGpng Failed_Jobs_at_GRIDKA_by_MinorStatus.png r1 manage 52.3 K 2009-06-29 - 11:53 VladimirRomanovskiy  
PNGpng Failed_Jobs_at_IN2P3_by_MinorStatus.png r1 manage 55.1 K 2009-06-29 - 11:53 VladimirRomanovskiy  
PNGpng Failed_Jobs_at_NIKHEF_by_MinorStatus.png r1 manage 51.9 K 2009-06-29 - 11:54 VladimirRomanovskiy  
PNGpng Failed_Jobs_at_PIC_by_MinorStatus.png r1 manage 61.5 K 2009-06-29 - 11:55 VladimirRomanovskiy  
PNGpng Failed_Jobs_by_UserGroup.png r1 manage 35.5 K 2009-06-29 - 10:29 VladimirRomanovskiy  
PNGpng Failed_Production_Jobs_Uploading_Job_Outputs_by_Site.png r1 manage 80.4 K 2009-06-29 - 11:22 VladimirRomanovskiy  
PNGpng Failed_Production_Jobs_Uploading_Job_Outputs_by_Site_Time.png r1 manage 60.2 K 2009-06-29 - 11:26 VladimirRomanovskiy  
PNGpng Failed_Production_Jobs_Uploading_Job_Uotputs_by_Site_Time.png r1 manage 60.2 K 2009-06-29 - 11:23 VladimirRomanovskiy  
PNGpng Failed_Production_Jobs_Watchdog_Stalled_by_Site.png r1 manage 91.6 K 2009-06-29 - 11:28 VladimirRomanovskiy  
PNGpng Failed_Production_Jobs_Watchdog_Stalled_by_Site_Time.png r1 manage 65.7 K 2009-06-29 - 11:29 VladimirRomanovskiy  
PNGpng Failed_Production_Jobs_by_MinorStatus.png r1 manage 59.9 K 2009-06-29 - 10:30 VladimirRomanovskiy  
PNGpng Failed_User_Jobs_by_MinorStatus.png r1 manage 62.4 K 2009-06-29 - 10:30 VladimirRomanovskiy  
PNGpng Failed_Users_Jobs_Input_Data_Resolution_by_Site.png r1 manage 47.4 K 2009-06-29 - 11:39 VladimirRomanovskiy  
PNGpng Total_Number_of_Jobs_by_FinalMajorStatus.png r1 manage 29.9 K 2009-06-29 - 10:31 VladimirRomanovskiy  
PNGpng volhcb01_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 23.6 K 2009-06-29 - 12:20 VladimirRomanovskiy  
PNGpng volhcb01_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 16.1 K 2009-06-29 - 12:22 VladimirRomanovskiy  
PNGpng volhcb01_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 12.3 K 2009-06-29 - 12:22 VladimirRomanovskiy  
PNGpng volhcb01_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png r1 manage 12.3 K 2009-06-29 - 12:23 VladimirRomanovskiy  
PNGpng volhcb06_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 26.0 K 2009-06-29 - 12:12 VladimirRomanovskiy  
PNGpng volhcb06_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 19.1 K 2009-06-29 - 12:12 VladimirRomanovskiy  
PNGpng volhcb06_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 14.2 K 2009-06-29 - 12:14 VladimirRomanovskiy  
PNGpng volhcb06_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png r1 manage 12.7 K 2009-06-29 - 12:15 VladimirRomanovskiy  
PNGpng volhcb09_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 28.5 K 2009-06-29 - 12:02 VladimirRomanovskiy  
PNGpng volhcb09_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 22.5 K 2009-06-29 - 12:02 VladimirRomanovskiy  
PNGpng volhcb09_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 13.5 K 2009-06-29 - 12:03 VladimirRomanovskiy  
PNGpng volhcb09_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png r1 manage 14.7 K 2009-06-29 - 12:04 VladimirRomanovskiy  
PNGpng volhcb10_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png r1 manage 25.9 K 2009-06-29 - 12:09 VladimirRomanovskiy  
PNGpng volhcb10_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png r1 manage 20.5 K 2009-06-29 - 12:21 VladimirRomanovskiy  
PNGpng volhcb10_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif.png r1 manage 12.2 K 2009-06-29 - 12:10 VladimirRomanovskiy  
PNGpng volhcb10_1_0_SWAP_SPACE_USED_STACKEDS_1.gif.png r1 manage 12.4 K 2009-06-29 - 12:10 VladimirRomanovskiy  
Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r1 - 2009-06-29 - VladimirRomanovskiy
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback