Week from 01062009 to 07062009

Job Statistics

• Summary:
  • Almost 82K jobs run last week
  • Over 13% failed
  • Daily peak of over 21K jobs
  • 53K Production jobs run to end
  • 16.5K User jobs run to the end
  • 6.2K Production Jobs Failed
  • 4.7K User Jobs Failed

• Total number of Jobs by Final Major Status

• Daily number of Jobs by Final Mayor Status

• Done|Completed Jobs by User Group

• Done|Completed Production Jobs by Job Type

• Failed Jobs by User Group

• Failed Production Jobs by Minor Status

• Failed User Jobs by Minor Status

Running at Tier1's

• Summary:
  • 14K Production Jobs at Tier1s
    • Shares?
  • 15.4K User Jobs at Tier1s
    • 50 % CERN Share
    • comment the shares is appropriate

• Done|Completed Production Jobs by Site

• Done|Completed User Jobs by Site

Job Failure Analysis

• Summary:
  • Production Jobs Failed mostly due to "Application Finished With Errors":
    • 3371 everywhere
    • 378 mostly at CERN
  • User Jobs Failed mosty due to "Input Data Resolution":
    • 5038 everywhere
    • 4079 mostly at CERN
  • Comment about any site is appropriated

• Failed Production Jobs (Application Finished With Error) by Site

• Failed User Jobs (Input Data Resolution) by Site

* Failed Production and User Jobs (Watchdog identified this job as stalled) by Site

• Failed Jobs at CERN by Minor Status

Hardware Status

• WMS volhcb09:
  • CPU utilization: Idle < 50%?, IO Wait peaks?,
  • Network utilization: 700K peak?
  • Swap Used: do we get close to the limit 1.9GB reached?.
  • Partition Used: stable

• DMS volhcb10:
  • CPU utilization: Idle < 50%?, IO Wait peaks?,
  • Network utilization: above 1 M?
  • Swap Used: do we get close to the limit (2 GB)?.
  • Partition Used: Is stable?

• LogSE volhcb06:
  • CPU utilization: Idle < 50%?, IO Wait peaks?,
  • Network utilization: above 1 M?
  • Swap Used: do we get close to the limit (2 GB)?.
  • Partition Used: Is stable?

• Various volhcb01:
  • CPU utilization: Idle < 50%?, IO Wait peaks?,
  • Network utilization: above 1 M?
  • Swap Used: do we get close to the limit (2 GB)?.
  • Partition Used: Is stable?

-- JoelClosier - 08 Jun 2009

• Failed_Jobs_Watchdog_Identified_This_Job_As_stalled_by_Site.png: