-- PaulSzczypka - 03 May 2009

DIRAC Weekly Report 20090503

Week Ending 2009-05-03

Job Statistics

(Follow the URLs, change the dates, save the new plot and add as attachement, then update the images)

  • Total number of Jobs by Final Major Status
Total_Number_of_Jobs_by_FinalMajorStatus.png
(https://lhcbweb.pic.es/DIRAC/LHCb-Production/lhcb_prod/systems/accountingPlots/job#ds9:_plotNames17:TotalNumberOfJobss13:_timeSelectors2:-1s10:_startTimes10:2009-04-25s8:_endTimes10:2009-05-02s9:_typeNames3:Jobs9:_groupings16:FinalMajorStatuse)
  • Daily number of Jobs by Final Mayor Status
Daily_Number_of_Jobs_by_FinalMajorStatus.png

(https://lhcbweb.pic.es/DIRAC/LHCb-Production/lhcb_prod/systems/accountingPlots/job#ds9:_plotNames12:NumberOfJobss13:_timeSelectors2:-1s10:_startTimes10:2009-04-25s8:_endTimes10:2009-05-02s9:_typeNames3:Jobs9:_groupings16:FinalMajorStatuse)

  • Done|Completed Jobs by User Group
Done_and_Complete_Jobs_by_UserGroup.png
(https://lhcbweb.pic.es/DIRAC/LHCb-Production/lhcb_prod/systems/accountingPlots/job#ds9:_plotNames17:TotalNumberOfJobss13:_timeSelectors2:-1s10:_startTimes10:2009-04-25s8:_endTimes10:2009-05-02s17:_FinalMajorStatuss14:Completed,Dones9:_typeNames3:Jobs9:_groupings9:UserGroupe)

  • Done|Completed Production Jobs by JobType
Done_and_Complete_Production_Jobs_by_JobType.png
(https://lhcbweb.pic.es/DIRAC/LHCb-Production/lhcb_prod/systems/accountingPlots/job#ds9:_plotNames17:TotalNumberOfJobss13:_timeSelectors2:-1s10:_startTimes10:2009-04-25s8:_endTimes10:2009-05-02s10:_UserGroups9:lhcb_prods17:_FinalMajorStatuss14:Completed,Dones9:_typeNames3:Jobs9:_groupings7:JobTypee)

  • Failed Jobs by User Group
Failed_Jobs_by_UserGroup.png
(https://lhcbweb.pic.es/DIRAC/LHCb-Production/lhcb_prod/systems/accountingPlots/job#ds9:_plotNames17:TotalNumberOfJobss13:_timeSelectors2:-1s10:_startTimes10:2009-04-25s8:_endTimes10:2009-05-02s17:_FinalMajorStatuss6:Faileds9:_typeNames3:Jobs9:_groupings9:UserGroupe)

  • Failed Production Jobs by Minor Status
Failed_Production_Jobs_by_MinorStatus.png
(https://lhcbweb.pic.es/DIRAC/LHCb-Production/lhcb_prod/systems/accountingPlots/job#ds9:_plotNames17:TotalNumberOfJobss13:_timeSelectors2:-1s10:_startTimes10:2009-04-25s8:_endTimes10:2009-05-02s10:_UserGroups9:lhcb_prods17:_FinalMajorStatuss6:Faileds9:_typeNames3:Jobs9:_groupings16:FinalMinorStatuse)

  • Failed User Jobs by Minor Status
Failed_User_Jobs_by_MinorStatus.png
(https://lhcbweb.pic.es/DIRAC/LHCb-Production/lhcb_prod/systems/accountingPlots/job#ds9:_plotNames17:TotalNumberOfJobss13:_timeSelectors2:-1s10:_startTimes10:2009-04-25s8:_endTimes10:2009-05-02s10:_UserGroups9:lhcb_users17:_FinalMajorStatuss6:Faileds9:_typeNames3:Jobs9:_groupings16:FinalMinorStatuse)

  • Summary:
    • Almost 100 K jobs run last week
    • Approximately 11 K failed
    • Daily peak of over 16 K jobs
    • 55 K Production jobs run to end
    • 27 K User jobs run to the end
    • ~ 11 K Failed Jobs

Running at Tier1's

  • Done|Completed Production Jobs by Site
Done_and_Complete_Production_Jobs_at_Tier1_by_Site.png
(https://lhcbweb.pic.es/DIRAC/LHCb-Production/lhcb_prod/systems/accountingPlots/job#ds9:_plotNames17:TotalNumberOfJobss13:_timeSelectors2:-1s10:_startTimes10:2009-04-25s8:_endTimes10:2009-05-02s10:_UserGroups9:lhcb_prods5:_Sites86:LCG.CERN.ch,LCG.CNAF.it,LCG.GRIDKA.de,LCG.IN2P3.fr,LCG.NIKHEF.nl,LCG.PIC.es,LCG.RAL.uks17:_FinalMajorStatuss14:Done,Completeds9:_typeNames3:Jobs9:_groupings4:Sitee)

  • Done|Completed User Jobs by Site
Done_and_Complete_User_Jobs_at_Tier1_by_Site.png
(https://lhcbweb.pic.es/DIRAC/LHCb-Production/lhcb_prod/systems/accountingPlots/job#ds9:_plotNames17:TotalNumberOfJobss13:_timeSelectors2:-1s10:_startTimes10:2009-04-25s8:_endTimes10:2009-05-02s10:_UserGroups9:lhcb_users5:_Sites86:LCG.CERN.ch,LCG.CNAF.it,LCG.GRIDKA.de,LCG.IN2P3.fr,LCG.NIKHEF.nl,LCG.PIC.es,LCG.RAL.uks17:_FinalMajorStatuss14:Done,Completeds9:_typeNames3:Jobs9:_groupings4:Sitee)

  • Summary:
    • 20 K Production Jobs at Tier1s
    • 16 K User Jobs at Tier1s
      • > 50 % CERN Share

Job Failure Analysis

(Change Error and User Group as Appropriated)

  • Failed Production Jobs by FinalMinorStatus
Failed_Production_Jobs_by_Final_Minor_Status.png
(https://lhcbweb.pic.es/DIRAC/LHCb-Production/lhcb_prod/systems/accountingPlots/job#ds9:_plotNames17:TotalNumberOfJobss13:_timeSelectors2:-1s10:_startTimes10:2009-04-25s8:_endTimes10:2009-05-02s10:_UserGroups9:lhcb_prods17:_FinalMajorStatuss6:Faileds9:_typeNames3:Jobs9:_groupings16:FinalMinorStatuse)

  • Failed Production Jobs (Application Finished With Error) by Site
Failed_Production_Jobs_Application_Finished_With_Errors_by_Site.png
(https://lhcbweb.pic.es/DIRAC/LHCb-Production/lhcb_prod/systems/accountingPlots/job#ds9:_plotNames17:TotalNumberOfJobss13:_timeSelectors2:-1s10:_startTimes10:2009-04-25s8:_endTimes10:2009-05-02s10:_UserGroups9:lhcb_prods17:_FinalMajorStatuss6:Faileds17:_FinalMinorStatuss32:Application%20Finished%20With%20Errorss9:_typeNames3:Jobs9:_groupings4:Sitee)

  • Failed Production Jobs (Input Sandbox Download) by Site
Failed_Production_Jobs_Input_Sandbox_Download_by_Site.png
(https://lhcbweb.pic.es/DIRAC/LHCb-Production/lhcb_prod/systems/accountingPlots/job#ds9:_plotNames17:TotalNumberOfJobss13:_timeSelectors2:-1s10:_startTimes10:2009-04-25s8:_endTimes10:2009-05-02s10:_UserGroups9:lhcb_prods17:_FinalMajorStatuss6:Faileds17:_FinalMinorStatuss22:Input%20Sandbox%20Downloads9:_typeNames3:Jobs9:_groupings4:Sitee)

  • Failed Production Jobs (Input Data Resolution) by Site
Failed_Production_Jobs_Input_Data_Resolution_by_Site.png
(https://lhcbweb.pic.es/DIRAC/LHCb-Production/lhcb_prod/systems/accountingPlots/job#ds9:_plotNames17:TotalNumberOfJobss13:_timeSelectors2:-1s10:_startTimes10:2009-04-25s8:_endTimes10:2009-05-02s10:_UserGroups9:lhcb_prods17:_FinalMajorStatuss6:Faileds17:_FinalMinorStatuss21:Input%20Data%20Resolutions9:_typeNames3:Jobs9:_groupings4:Sitee)

  • Failed User Jobs by FinalMinorStatus
Failed_User_Jobs_by_Final_Minor_Status.png
(https://lhcbweb.pic.es/DIRAC/LHCb-Production/lhcb_prod/systems/accountingPlots/job#ds9:_plotNames17:TotalNumberOfJobss13:_timeSelectors2:-1s10:_startTimes10:2009-04-25s8:_endTimes10:2009-05-02s10:_UserGroups9:lhcb_users17:_FinalMajorStatuss6:Faileds9:_typeNames3:Jobs9:_groupings16:FinalMinorStatuse)

  • Failed User Jobs (Input Data Resolution) by Site
Failed_Users_Jobs_Input_Data_Resolution_by_Site.png
(https://lhcbweb.pic.es/DIRAC/LHCb-Production/lhcb_prod/systems/accountingPlots/job#ds9:_plotNames17:TotalNumberOfJobss13:_timeSelectors2:-1s10:_startTimes10:2009-04-25s8:_endTimes10:2009-05-02s10:_UserGroups9:lhcb_users17:_FinalMajorStatuss6:Faileds17:_FinalMinorStatuss21:Input%20Data%20Resolutions9:_typeNames3:Jobs9:_groupings4:Sitee)

  • Failed User Jobs (Application Finished With Error) by Site
Failed_Users_Jobs_Application_Finished_With_Errors_by_Site.png
(https://lhcbweb.pic.es/DIRAC/LHCb-Production/lhcb_prod/systems/accountingPlots/job#ds9:_plotNames17:TotalNumberOfJobss13:_timeSelectors2:-1s10:_startTimes10:2009-04-25s8:_endTimes10:2009-05-02s10:_UserGroups9:lhcb_users17:_FinalMajorStatuss6:Faileds17:_FinalMinorStatuss32:Application%20Finished%20With%20Errorss9:_typeNames3:Jobs9:_groupings4:Sitee)

  • Summary:
    • Production Jobs Failed mostly due to:
      • Application Finished with Errors (4 K total, 500 at CNAF)
      • Input Sandbox Download mostly at GRIDKA (2.5 K total, 1.9 K at GRIDKA)
      • Input Data Resolution mostly at GRIDKA (1.2 K total, 630 at GRIDKA)
    • User Jobs Failed mostly due to:
      • Input Data Resolution mostly at RAL (4.7 K total, 3.3K at RAL)
      • Application Finished with Error (1K, 500 at CERN)

Hardware Status

Status at ~15:00 20090503:

  • WMS volhcb09:
    • swap usage ~ .6Gb
(https://lemonweb.cern.ch/lemon-web/info.php?time=1&offset=0&entity=volhcb09&detailed=yes)
volhcb09_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif volhcb09_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif
volhcb09_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif volhcb09_1_0_SWAP_SPACE_USED_STACKEDS_1.gif
[root@volhcb09 ~]# df -h /opt/dirac
NO ACCESS!

  • DMS volhcb10:
    • No swap data available
(https://lemonweb.cern.ch/lemon-web/info.php?time=1&offset=0&entity=volhcb10&detailed=yes)
volhcb10_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif volhcb10_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif
volhcb10_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif volhcb10_1_0_SWAP_SPACE_USED_STACKEDS_1.gif
[root@volhcb10 ~]# df -h /opt/dirac
NO ACCESS!

  • LogSE volhcb06:
    • Must watch disk usage more closely, probably need additional disk.
    • Daily IOwait peaks, probably due to full disk scan by updatedb cron (to be removed).
(https://lemonweb.cern.ch/lemon-web/info.php?time=1&offset=0&entity=volhcb06&detailed=yes)
volhcb06_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif volhcb06_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif
volhcb06_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif volhcb06_1_0_SWAP_SPACE_USED_STACKEDS_1.gif
[root@volhcb06 ~]# df -h /opt/dirac/ /storage/
NO ACCESS!
  • Various volhcb01:
    • What are the peaks in the CPU usage? An agent?. (Unresolved)

(https://lemonweb.cern.ch/lemon-web/info.php?time=1&offset=0&entity=volhcb01&detailed=yes)
volhcb01_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif volhcb01_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif
volhcb01_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif volhcb01_1_0_SWAP_SPACE_USED_STACKEDS_1.gif
[root@volhcb01 ~]# df -h /opt/dirac /storage
NO ACCESS!
-- RicardoGraciani - 27 Apr 2009
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng Daily_Number_of_Jobs_by_FinalMajorStatus.png r1 manage 39.9 K 2009-05-03 - 20:51 PaulSzczypka  
PNGpng Done+Complete_Jobs_by_UserGroup.png r1 manage 39.8 K 2009-05-03 - 11:37 PaulSzczypka  
PNGpng Done+Complete_Production_Jobs_by_JobType.png r1 manage 29.2 K 2009-05-03 - 11:34 PaulSzczypka  
PNGpng Done_and_Complete_Jobs_by_UserGroup.png r1 manage 34.9 K 2009-05-03 - 20:49 PaulSzczypka  
PNGpng Done_and_Complete_Production_Jobs_at_Tier1_by_Site.png r1 manage 56.1 K 2009-05-03 - 20:50 PaulSzczypka  
PNGpng Done_and_Complete_Production_Jobs_by_JobType.png r1 manage 38.0 K 2009-05-03 - 20:49 PaulSzczypka  
PNGpng Done_and_Complete_User_Jobs_at_Tier1_by_Site.png r1 manage 57.2 K 2009-05-03 - 20:50 PaulSzczypka  
PNGpng Failed_Jobs_by_UserGroup.png r2 r1 manage 34.4 K 2009-05-03 - 20:51 PaulSzczypka  
PNGpng Failed_Production_Jobs_Application_Finished_With_Errors_by_Site.png r1 manage 118.8 K 2009-05-03 - 20:50 PaulSzczypka  
PNGpng Failed_Production_Jobs_Input_Data_Resolution_by_Site.png r1 manage 41.5 K 2009-05-03 - 22:14 PaulSzczypka  
PNGpng Failed_Production_Jobs_Input_Sandbox_Download_by_Site.png r1 manage 65.7 K 2009-05-03 - 20:50 PaulSzczypka  
PNGpng Failed_Production_Jobs_by_Final_Minor_Status.png r1 manage 54.6 K 2009-05-03 - 20:53 PaulSzczypka  
PNGpng Failed_Production_Jobs_by_MinorStatus.png r2 r1 manage 54.6 K 2009-05-03 - 20:51 PaulSzczypka  
PNGpng Failed_User_Jobs_by_Final_Minor_Status.png r1 manage 49.4 K 2009-05-03 - 20:54 PaulSzczypka  
PNGpng Failed_User_Jobs_by_MinorStatus.png r2 r1 manage 49.4 K 2009-05-03 - 20:51 PaulSzczypka  
PNGpng Failed_Users_Jobs_Application_Finished_With_Errors_by_Site.png r1 manage 72.9 K 2009-05-03 - 20:49 PaulSzczypka  
PNGpng Failed_Users_Jobs_Input_Data_Resolution_by_Site.png r1 manage 44.2 K 2009-05-03 - 20:50 PaulSzczypka  
PNGpng Total_Number_of_Jobs_by_FinalMajorStatus.png r1 manage 39.9 K 2009-05-03 - 20:51 PaulSzczypka  
GIFgif volhcb01_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif r1 manage 23.7 K 2009-05-03 - 15:49 PaulSzczypka  
GIFgif volhcb01_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif r1 manage 16.1 K 2009-05-03 - 15:49 PaulSzczypka  
GIFgif volhcb01_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif r1 manage 13.4 K 2009-05-04 - 11:34 PaulSzczypka  
GIFgif volhcb01_1_0_SWAP_SPACE_USED_STACKEDS_1.gif r1 manage 11.3 K 2009-05-04 - 11:34 PaulSzczypka  
GIFgif volhcb06_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif r1 manage 25.2 K 2009-05-03 - 15:49 PaulSzczypka  
GIFgif volhcb06_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif r1 manage 16.6 K 2009-05-03 - 15:49 PaulSzczypka  
GIFgif volhcb06_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif r1 manage 14.1 K 2009-05-03 - 15:49 PaulSzczypka  
GIFgif volhcb06_1_0_SWAP_SPACE_USED_STACKEDS_1.gif r1 manage 12.0 K 2009-05-03 - 15:52 PaulSzczypka  
GIFgif volhcb09_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif r1 manage 26.5 K 2009-05-03 - 15:35 PaulSzczypka  
GIFgif volhcb09_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif r1 manage 21.8 K 2009-05-03 - 15:36 PaulSzczypka  
GIFgif volhcb09_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif r1 manage 12.8 K 2009-05-03 - 15:35 PaulSzczypka  
GIFgif volhcb09_1_0_SWAP_SPACE_USED_STACKEDS_1.gif r1 manage 12.7 K 2009-05-03 - 15:36 PaulSzczypka  
GIFgif volhcb10_1_0_CPUUTILPERCUSER_CPUUTILPERCSYSTEM_CPUUTILPERCNICE_CPUUTILPERCIDLE_CPUUTILPERCIOWAIT_CPUUTILPERCIRQ_CPUUTILPERCSOFTIRQSTACKEDC_1.gif r1 manage 27.5 K 2009-05-03 - 15:48 PaulSzczypka  
GIFgif volhcb10_1_0_NUMKBREADAVG_NUMKBWRITEAVGOVERLAYN_1.gif r1 manage 21.5 K 2009-05-03 - 15:48 PaulSzczypka  
GIFgif volhcb10_1_0_PARTITIONUSEDPERC_STACKEDP_1.gif r1 manage 12.1 K 2009-05-04 - 11:34 PaulSzczypka  
GIFgif volhcb10_1_0_SWAP_SPACE_USED_STACKEDS_1.gif r1 manage 12.5 K 2009-05-04 - 11:34 PaulSzczypka  
Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r5 - 2009-05-04 - PaulSzczypka
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback