DIRACWeeklyReport20090720 < LHCb

LHCb Web>LHCbComputing>DIRACWeeklyReport>DIRACWeeklyReport20090720 (2009-08-10, FedericoStagni) (raw view)
---+++ Week from _13072009_ to _20072009_ <br />

---++++ Job Statistics<br />

   * Summary:
      * Almost 351 K jobs run last week
      * Over 7% failed
      * Daily peak of over 60 K jobs
      * 278 K Production jobs run to end
      * 26 K User jobs run to the end
      * 19 K Production Jobs Failed
      * About 5 K User Jobs Failed


   * *Total* number of Jobs by Final Major Status
<img alt="Total_Number_of_Jobs_by_FinalMajorStatus.png" src="%ATTACHURL%/Total_Number_of_Jobs_by_FinalMajorStatus.png" /><br />

   * *Daily* number of Jobs by Final Mayor Status
<img alt="Daily_Number_of_Jobs_by_FinalMajorStatus.png" src="%ATTACHURL%/Daily_Number_of_Jobs_by_FinalMajorStatus.png" /><br />

   * *Done|Completed* Jobs by User Group
<img alt="Done+Complete_Jobs_by_UserGroup.png" src="%ATTACHURL%/Done+Complete_Jobs_by_UserGroup.png" /><br />


   * *Done|Completed* *Production* Jobs by Job Type
<img alt="Done+Complete_Production_Jobs_by_JobType.png" src="%ATTACHURL%/Done+Complete_Production_Jobs_by_JobType.png" /><br />

   * *Failed* Jobs by User Group
<img alt="Failed_Jobs_by_UserGroup.png" src="%ATTACHURL%/Failed_Jobs_by_UserGroup.png" /><br />

   * *Failed* *Production* Jobs by Minor Status 
<img alt="Failed_Production_Jobs_by_MinorStatus.png" src="%ATTACHURL%/Failed_Production_Jobs_by_MinorStatus.png" /><br />


   * *Failed* *User* Jobs by Minor Status
<img alt="Failed_User_Jobs_by_MinorStatus.png" src="%ATTACHURL%/Failed_User_Jobs_by_MinorStatus.png" /><br />


---++++ Running at Tier1's<br />

   * Summary:
      * 87 K Production Jobs at Tier1s
         * 41% at GRIDKA
         * 13% at CNAF
         * 12% at CERN
         * 11% at IN2P3
         * 10% at RAL
         * 8% at PIC
         * 5% at NIKHEF
      * 20 K User Jobs at Tier1s
         * 19 % CERN Share

   * *Done|Completed* *Production* Jobs by Site
<img alt="Done+Complete_Production_Jobs_at_Tier1_by_Site.png" src="%ATTACHURL%/Done+Complete_Production_Jobs_at_Tier1_by_Site.png" /><br />

   * *Done|Completed* *User* Jobs by Site
<img alt="Done+Complete_User_Jobs_at_Tier1_by_Site.png" src="%ATTACHURL%/Done+Complete_User_Jobs_at_Tier1_by_Site.png" /><br />


---++++ Job Failure Analysis<br />

   * Summary:
      * Production Jobs Failed mostly due to:
         * Application finished  with errors ( ~7200)
         * Watchdog identified this job as stalled (~6400)
      * User Jobs Failed mosty due to:
         * Watchdog identified this job as stalled (~1400)
         * Application finished  with errors (~1200)

   * *Failed* *Production* Jobs (*Application Finished With Error*) by Site
<img alt="Failed_Production_Jobs_Application_Finished_With_Errors_by_Site.png" src="%ATTACHURL%/Failed_Production_Jobs_Application_Finished_With_Errors_by_Site.png" /><br />

   * *Failed* *User* Jobs (*Input Data Resolution*) by Site
<img alt="Failed_Users_Jobs_Input_Data_Resolution_by_Site.png" src="%ATTACHURL%/Failed_Users_Jobs_Input_Data_Resolution_by_Site.png" /><br />


   * *Failed* Jobs at *GRIDKA* by Minor Status
<img alt="Failed_Jobs_at_GRIDKA_by_MinorStatus.png" src="%ATTACHURL%/Failed_Jobs_at_GRIDKA_by_MinorStatus.png" /><br />

   * *Failed* Jobs at *CERN* by Minor Status
<img alt="Failed_Jobs_at_CERN_by_MinorStatus.png" src="%ATTACHURL%/Failed_Jobs_at_CERN_by_MinorStatus.png" /><br />

   * *Failed* Jobs at *PIC* by Minor Status
<img alt="Failed_Jobs_at_PIC_by_MinorStatus.png" src="%ATTACHURL%/Failed_Jobs_at_PIC_by_MinorStatus.png" /><br />


---++++ Hardware Status<br />

   * _WMS_ *volhcb09*: 
      * CPU utilization: Idle > 50%, IO Wait peaks?,
      * Network utilization: < 700K
      * Swap Used: less than 1.2 G, under the limit (2 GB).
      * Partition Used: Stable at 101G


<img alt="volhcb09_1_-86400_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png" src="%ATTACHURL%/volhcb09_1_-86400_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png" />
<img alt="volhcb09_1_-86400_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png" src="%ATTACHURL%/volhcb09_1_-86400_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png" /><br />
<img alt="volhcb09_1_-86400_PARTITIONUSEDPERC_STACKEDP_1.gif.png" src="%ATTACHURL%/volhcb09_1_-86400_PARTITIONUSEDPERC_STACKEDP_1.gif.png" />
<img alt="volhcb09_1_-86400_SWAP_SPACE_USED_STACKEDS_1.gif.png" src="%ATTACHURL%/volhcb09_1_-86400_SWAP_SPACE_USED_STACKEDS_1.gif.png" /><br />

   * _DMS_ *volhcb10*: 
      * CPU utilization: Idle < 17%
      * Network utilization: between 90K to 250K 
      * Swap Used: < 700M, under the limit (2 GB).
      * Partition Used: Stable at 84G


<img alt="volhcb10_1_-86400_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png" src="%ATTACHURL%/volhcb10_1_-86400_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png" />
<img alt="volhcb10_1_-86400_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png" src="%ATTACHURL%/volhcb10_1_-86400_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png" /><br />
<img alt="volhcb10_1_-86400_PARTITIONUSEDPERC_STACKEDP_1.gif.png" src="%ATTACHURL%/volhcb10_1_-86400_PARTITIONUSEDPERC_STACKEDP_1.gif.png" />
<img alt="volhcb10_1_-86400_SWAP_SPACE_USED_STACKEDS_1.gif.png" src="%ATTACHURL%/volhcb10_1_-86400_SWAP_SPACE_USED_STACKEDS_1.gif.png" /><br />

   * _LogSE_ *volhcb06*: 
      * CPU utilization: Idle > 50%
      * Network utilization: < 200K
      * Swap Used: < 230K, under the limit (2 GB).
      * Partition Used: Stable

=(https://lemonweb.cern.ch/lemon-web/info.php?time=1&offset=1d&entity=volhcb06&detailed=yes )= <br />


<img alt="volhcb06_1_-86400_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png" src="%ATTACHURL%/volhcb06_1_-86400_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png" />
<img alt="volhcb06_1_-86400_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png" src="%ATTACHURL%/volhcb06_1_-86400_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png" /><br />
<img alt="volhcb06_1_-86400_PARTITIONUSEDPERC_STACKEDP_1.gif.png" src="%ATTACHURL%/volhcb06_1_-86400_PARTITIONUSEDPERC_STACKEDP_1.gif.png" />
<img alt="volhcb06_1_-86400_SWAP_SPACE_USED_STACKEDS_1.gif.png" src="%ATTACHURL%/volhcb06_1_-86400_SWAP_SPACE_USED_STACKEDS_1.gif.png" /><br />

   * Various *volhcb01*: 
      * CPU utilization: Idle > 80%
      * Network utilization: <6K
      * Swap Used: about 135K, under the limit (2 GB).
      * Partition Used: Stable at 740G

=(https://lemonweb.cern.ch/lemon-web/info.php?time=1&offset=1d&entity=volhcb01&detailed=yes )= <br />


<img alt="volhcb01_1_-86400_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png" src="%ATTACHURL%/volhcb01_1_-86400_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif.png" />
<img alt="volhcb01_1_-86400_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png" src="%ATTACHURL%/volhcb01_1_-86400_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif.png" /><br />
<img alt="volhcb01_1_-86400_PARTITIONUSEDPERC_STACKEDP_1.gif.png" src="%ATTACHURL%/volhcb01_1_-86400_PARTITIONUSEDPERC_STACKEDP_1.gif.png" />
<img alt="volhcb01_1_-86400_SWAP_SPACE_USED_STACKEDS_1.gif.png" src="%ATTACHURL%/volhcb01_1_-86400_SWAP_SPACE_USED_STACKEDS_1.gif.png" /><br />


-- Main.JiboHE - 20 Jul 2009
Topic revision: r2 - 2009-08-10 - FedericoStagni
LHCb
Public webs
Welcome Guest
- Cern Search
- TWiki Search
- Google Search
LHCb All webs
Copyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback