Manual override tool

The main function of manual override tool is to make temporary change of the site status for certain metric. Manual override tool is available for lifeStatus, analysis and production metrics.

Workflow:

1. User changes status of site in manual override tool
2. Manual changes metric gets a new value
3. Original metric takes value from manual changes metric and applies new state.
4. The person who sets manual override should pursue resolving the issue requiring the manual

LifeStatus

• MonIT HDFS: path "/project/monitoring/archive/cmssst/raw/ssbmetric/", metadata.path "sts15min", data.status
• MonIT Grafana: index "monit_prod_cmssst_*", metadata.path "sts15min, data.status
• Manual Override tool
• Manual Override JSON
• Manual Override log
• States: Ok, Waiting room, Morgue, No override
• Rights: SiteDB operators (SST team) - can manage all sites, site admin - can manage their site
• Delay: primal LifeStatus updates values every 24 hours, LifeStatus updates every 15 minutes

ProdStatus

• MonIT HDFS: path "/project/monitoring/archive/cmssst/raw/ssbmetric/", metadata.path "sts15min", data.prod_status
• MonIT Grafana: index "monit_prod_cmssst_*", metadata.path "sts15min, data.prod_status
• Manual Override tool
• Manual Override JSON
• Manual Override log
• States: Enabled, Disabled, Drain, Test, No override
• Rights: cms-comp-ops-site-support-team, cms-comp-ops-workflow-team - can manage all the sites, cms-tier0-operations - can manage T0_CH_CERN, T2_CH_CERN, T2_CH_CERN_HLT, site admin - can manage their site,
  
• Delay: prodStatus is updated every 15 minutes

CrabStatus

• MonIT HDFS: path "/project/monitoring/archive/cmssst/raw/ssbmetric/", metadata.path "sts15min", data.crab_status
• MonIT Grafana: index "monit_prod_cmssst_*", metadata.path "sts15min, data.crab_status
• Manual Override tool
• Manual Override JSON
• Manual Override log
• States: Enabled, Disabled, No override
• Rights: cms-comp-ops-site-support-team - can manage all the sites, site admin - can manage their site,
  
• Delay: crabStatus is updated every 15 minutes