Manual override tool

The main function of manual override tool is to make temporary change of the site status for certain metric. Manual override tool is available for lifeStatus, analysis and production metrics.

Workflow:

  1. User changes status of site in manual override tool
  2. Manual changes metric gets a new value
  3. Original metric takes value from manual changes metric and applies new state.
  4. The person who sets manual override should pursue resolving the issue requiring the manual
override and clear it promptly, so the primary metric should be able to set the real status of the site.

LifeStatus

  • MonIT HDFS: path "/project/monitoring/archive/cmssst/raw/ssbmetric/", metadata.path "sts15min", data.status
  • MonIT Grafana: index "monit_prod_cmssst_*", metadata.path "sts15min, data.status
  • Manual Override tool
  • Manual Override JSON
  • Manual Override log
  • States: Ok, Waiting room, Morgue, No override
  • Rights: SiteDB operators (SST team) - can manage all sites, site admin - can manage their site
  • Delay: primal LifeStatus updates values every 24 hours, LifeStatus updates every 15 minutes

ProdStatus

  • MonIT HDFS: path "/project/monitoring/archive/cmssst/raw/ssbmetric/", metadata.path "sts15min", data.prod_status
  • MonIT Grafana: index "monit_prod_cmssst_*", metadata.path "sts15min, data.prod_status
  • Manual Override tool
  • Manual Override JSON
  • Manual Override log
  • States: Enabled, Disabled, Drain, Test, No override
  • Rights: cms-comp-ops-site-support-team, cms-comp-ops-workflow-team - can manage all the sites, cms-tier0-operations - can manage T0_CH_CERN, T2_CH_CERN, T2_CH_CERN_HLT, site admin - can manage their site,
  • Delay: prodStatus is updated every 15 minutes

CrabStatus

  • MonIT HDFS: path "/project/monitoring/archive/cmssst/raw/ssbmetric/", metadata.path "sts15min", data.crab_status
  • MonIT Grafana: index "monit_prod_cmssst_*", metadata.path "sts15min, data.crab_status
  • Manual Override tool
  • Manual Override JSON
  • Manual Override log
  • States: Enabled, Disabled, No override
  • Rights: cms-comp-ops-site-support-team - can manage all the sites, site admin - can manage their site,
  • Delay: crabStatus is updated every 15 minutes

Edit | Attach | Watch | Print version | History: r12 < r11 < r10 < r9 < r8 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r12 - 2020-12-16 - StephanLammel
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback