In/Out of Waiting Room

What is the Waiting Room?

• Is a status set for a site in Site Status Board (SSB).

• The list of sites in the waiting room can be seen here:
  • WR LIST - SSB Default view
  • WR HISTORY PLOT (Important to use it to check for latest changes in WR metric).

What constitutes being in the Waiting Room?

• Being in the Waiting Room means that:
  • Site is not working correctly for a longer period of time despite support efforts by CompOps.
    • Please refer to the criteria in: How do you get IN the Waiting Room?
  • Tickets are not being handled with highest urgency like other sites.
    • Please reffer to our Ticketing policy.

Consequences

• Site will be excluded from MC production and analysis.
• In the future:
  • ESP credit will be reduced according to the time in the waiting room.
    • Also according to our Ticketing policy.

How do you get IN the Waiting Room?

• If Site Readiness (SR) < 80% lifeStatus = ERROR. If site has ERROR status for 5 days in last 15 days and last day. Note that weekends are not counted to get in waiting room.
• Manaul override tool for lifeStatus metric - https://cmssst.web.cern.ch/cmssst/man_override/cgi/manualOverride.py/lifestatus

• CMS Site Readiness Report
  • http://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html
  • Site Readiness is based on the critical metrics from SSB SR View - Expanded table.

How do you get OUT of the Waiting Room?

• If the site's SR > 80% lifeStatus = OK. If site had OK status for the last 3 days.
  • This gives the chance to move out of the Waiting room quickly, even if your past months were bad.

• IF a site keeps falling into the Waiting Room too often (50% or more in a month)
  • The site will be kept in the Waiting Room for a longer period of time until it shows that issues have been properly solved and are not failing again.

• IF a site has been in the Waiting Room for more than 5 weeks, we will have to test it for stability and keep it under observance for a little longer than a week.

Morgue

• A site is moved to the Morgue if it has been IN the Waiting Room for 8 weeks or more in the last 2 months.

Actions taken by the Site Support Team

EVERY WEEK (Monday/Tuesday)

• Site Readiness Plots:

COMBINED	Plot	SSB
3 MONTHS	Plot	Table w/ Stats
LAST WEEK	Plot	Table w/ Stats

• Check last week's readiness for sites in waiting room
  • Update the tickets, close if above threshold.
• Check last week and trimonth's readiness for all sites not in waiting room
  • follow procedure below for sites going into it.

• Make summary table and place it in CompOps Meeting and Site Support Twikis.
  • Script to create this summary based on dashboard stats: Script
  • USE metrics shown in SSB: site_readiness view
    • Site Readiness last Week_3 months (red if <80%)
    • Waiting Room - YES/NO (YES=site do NOT have entry in the latest values of metric 39 file)

When a site goes IN to the waiting room

• Open GGUS ticket to the site explaining the action, with reasons, and how to solve them.
• Communicate with Workflow team so they stop sending jobs to sites in waiting room.
  • email: cms-comp-ops-workflow-team@cernNOSPAMPLEASE.ch
  • Put the sites in drain (workflow team)

When a site comes OUT of the waiting room

• Communicate with Workflow team so they re start sending jobs to sites.
  • email: cms-comp-ops-workflow-team@cernNOSPAMPLEASE.ch
  • Test site if valid for re commissioning workflows
  • Remove the sites in drain (workflow team)

Instructions to move sites in SSB

https://cmsdoc.cern.ch/cms/LCG/SiteComm/T2WaitingList/WasCommissionedT2ForSiteMonitor.txt

Longterm tasks

• Automatic procedures - the idea of site readiness is that sites could be automatically included and excluded in the systems (WMAgent, CRAB, etc.)
• Steps towards an automated system:
  • DONE - develop procedures moving sites in and out of waiting room
  • DONE - move sites in and out manually without them having an effect on the usage of the sites
  • DONE - report on statistics and tell the sites what we found and what we are going to do
  • DONE - allow WMAgents to exclude sites because of readiness (put them in drain), still move sites manually in and out of the waiting room
  • DONE: automate moving sites in and out - requires creating and linking new metrics in SSB site readiness and production views.