Issues Pages and CompOps Meeting Plots

These scripts generate:

  • A summary overview of the workflows in the system, in web page.
  • A summary plot for the requests in the system and how old are they.
  • A running/pending jobs plot for the agents in production, for the last week.

Before Starting

Make a Single-Run shell script

  • Create a shell file in your AFS home folder run_issues.sh:
#! /bin/sh
# Set Python PATH
PYTHONPATH=$PYTHONPATH:YOUR_AFS_PATH/WmAgentScripts:
source /data/srv/wmagent/current/apps/wmagent/etc/profile.d/init.sh

# Create a proxy
voms-proxy-init -pwstdin < password
# Set the PROXY environment, replace XXXX with the proxy filename is
export X509_USER_PROXY=/tmp/x509up_u5XXXX

# get an Overview file (list of workflows)
python WmAgentScripts/issues/getoverview.py
# get data for each workflow
python WmAgentScripts/issues/getdata.py
# generate web pages
python WmAgentScripts/issues/makeSummaryIssues.py
  • Keep in mind that voms-proxy-init asks for password, so you can either:
    • Comment this line and create a 24hour proxy yourself before running the script Creating a Proxy
    • Write your password in a text file and pass it as standard input (as shown in the example)
  • Tune file paths according to your folders:
    • Set afs_base and cachedoverview in issues/getdata.py and issues/makeSummaryIssues.py
    • Set cachedoverview and cachedoverviewzipped in issues/getoverview.py
    • you may need to create ~/www/mc , ~/www/reproc and ~/www/tc and check permissions according to what you set up.
  • Run the script (this may take a while):
source run_issues.sh
  • Check that the files/pages are produced
    • overview.cache: A list of workflows
    • data.json: A list of workflows and some details about them
    • mc/issues.html, reproc/issues.html and tc/issues.html: Pages with issues details for each type of request.

Set up an Acronjob

  • From an lxplus machine, open the acrontab
acrontab -e
  • Add an entry for the script you run
# everyday at 6:05am and 6:05pm
5 6,18 * * * vocms049 source YOUR_AFS_PATH/run_issues.sh >> temp.txt
  • Make sure you choose one of the machines in which you can properly run the script.
  • The syntax is very similar to that of standard Unix crontab, check man acrontab to know details.

Make Plots for the Monday Meeting

  • Create a plot_compops_report.sh file in your AFS home folder with:
#! /bin/sh
# Generate age distribution for status, type and campaign (make sure to use the data.json resulting from the script)
python WmAgentScripts/issues/plotAgeDistribution.py ~/www/data.json -j

# Production agents, running jobs
curl "http://dashb-cms-prod.cern.ch/dashboard/request.py/condorjobnumbers_individual?sites=vocms0308.cern.ch&sites=vocms0309.cern.ch&sites=vocms0310.cern.ch&sites=vocms0311.cern.ch&sites=cmsgwms-submit1.fnal.gov&sites=cmsgwms-submit2.fnal.gov&sites=cmssrv217.fnal.gov&sites=cmssrv218.fnal.gov&sites=cmssrv219.fnal.gov&sitesSort=11&jobTypes=&start=null&end=null&timeRange=lastWeek&granularity=Hourly&cores=&sortBy=4&series=All&type=r" > ~/www/plots/production_running.png

# Production agents, pending jobs
curl "http://dashb-cms-prod.cern.ch/dashboard/request.py/condorjobnumbers_individual?sites=vocms0308.cern.ch&sites=vocms0309.cern.ch&sites=vocms0310.cern.ch&sites=vocms0311.cern.ch&sites=cmsgwms-submit1.fnal.gov&sites=cmsgwms-submit2.fnal.gov&sites=cmssrv217.fnal.gov&sites=cmssrv218.fnal.gov&sites=cmssrv219.fnal.gov&sitesSort=11&jobTypes=&start=null&end=null&timeRange=lastWeek&granularity=Hourly&cores=&sortBy=4&series=All&type=p" > ~/www/plots/production_pending.png
  • Make sure all the WMAgent servers are included in the list.
  • Check file paths and tune them according to your folders: /www/plots/by_type.png and /www/plots/by_status.png in issues/plotAgeDistribution.py
  • Run the script: remember this uses data.json as input.
source plot_compops_report.sh
  • Check the generated files.
  • Add the following entry to your acrontab:
# Running CompOps meeting plots at Mondays at 1:20pm
20 13 * * 1 lxplus source YOUR_AFS_PATH/plot_compops_report.sh > temp.txt
  • If the generated files are in a public url, you can make a link to them from the meeting report (instead of manually inserting the file).
  • The file temp.txt will show how old is every workflow, you can use that information to make a tail report.

-- JulianBadillo - 2015-12-03

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2015-12-03 - JulianBadillo
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback