Issues Pages and CompOps Meeting Plots

These scripts generate:

  • A summary overview of the workflows in the system, in web page.
  • A summary plot for the requests in the system and how old are they.
  • A running/pending jobs plot for the agents in production, for the last week.

Before Starting

Make a Single-Run shell script

  • Create a shell file in your AFS home folder
#! /bin/sh
# Set Python PATH
source /data/srv/wmagent/current/apps/wmagent/etc/profile.d/

# Create a proxy
voms-proxy-init -pwstdin < password
# Set the PROXY environment, replace XXXX with the proxy filename is
export X509_USER_PROXY=/tmp/x509up_u5XXXX

# get an Overview file (list of workflows)
python WmAgentScripts/issues/
# get data for each workflow
python WmAgentScripts/issues/
# generate web pages
python WmAgentScripts/issues/
  • Keep in mind that voms-proxy-init asks for password, so you can either:
    • Comment this line and create a 24hour proxy yourself before running the script Creating a Proxy
    • Write your password in a text file and pass it as standard input (as shown in the example)
  • Tune file paths according to your folders:
    • Set afs_base and cachedoverview in issues/ and issues/
    • Set cachedoverview and cachedoverviewzipped in issues/
    • you may need to create ~/www/mc , ~/www/reproc and ~/www/tc and check permissions according to what you set up.
  • Run the script (this may take a while):
  • Check that the files/pages are produced
    • overview.cache: A list of workflows
    • data.json: A list of workflows and some details about them
    • mc/issues.html, reproc/issues.html and tc/issues.html: Pages with issues details for each type of request.

Set up an Acronjob

  • From an lxplus machine, open the acrontab
acrontab -e
  • Add an entry for the script you run
# everyday at 6:05am and 6:05pm
5 6,18 * * * vocms049 source YOUR_AFS_PATH/ >> temp.txt
  • Make sure you choose one of the machines in which you can properly run the script.
  • The syntax is very similar to that of standard Unix crontab, check man acrontab to know details.

Make Plots for the Monday Meeting

  • Create a file in your AFS home folder with:
#! /bin/sh
# Generate age distribution for status, type and campaign (make sure to use the data.json resulting from the script)
python WmAgentScripts/issues/ ~/www/data.json -j

# Production agents, running jobs
curl "" > ~/www/plots/production_running.png

# Production agents, pending jobs
curl "" > ~/www/plots/production_pending.png
  • Make sure all the WMAgent servers are included in the list.
  • Check file paths and tune them according to your folders: /www/plots/by_type.png and /www/plots/by_status.png in issues/
  • Run the script: remember this uses data.json as input.
  • Check the generated files.
  • Add the following entry to your acrontab:
# Running CompOps meeting plots at Mondays at 1:20pm
20 13 * * 1 lxplus source YOUR_AFS_PATH/ > temp.txt
  • If the generated files are in a public url, you can make a link to them from the meeting report (instead of manually inserting the file).
  • The file temp.txt will show how old is every workflow, you can use that information to make a tail report.

-- JulianBadillo - 2015-12-03

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2015-12-03 - JulianBadillo
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback