Site Support Team - Documentation

In/Out of Waiting Room

What is the Waiting Room?

  • Is a status set for a site in Site Status Board (SSB).

What constitutes being in the Waiting Room?

  • Being in the Waiting Room means that:
    • Site is not working correctly for a longer period of time despite support efforts by CompOps.
      • Please refer to the criteria in: How do you get IN the Waiting Room?
    • Tickets are not being handled with highest urgency like other sites.

Consequences

  • Site will be excluded from MC production and analysis.
  • In the future:
    • ESP credit will be reduced according to the time in the waiting room.

How do you get IN the Waiting Room?

How do you get OUT of the Waiting Room?

  • If the site's SR > 80% lifeStatus = OK. If site had OK status for the last 3 days.
    • This gives the chance to move out of the Waiting room quickly, even if your past months were bad.

  • IF a site keeps falling into the Waiting Room too often (50% or more in a month)
    • The site will be kept in the Waiting Room for a longer period of time until it shows that issues have been properly solved and are not failing again.

  • IF a site has been in the Waiting Room for more than 5 weeks, we will have to test it for stability and keep it under observance for a little longer than a week.

Morgue

  • A site is moved to the Morgue if it has been IN the Waiting Room for 8 weeks or more in the last 2 months.
  • How to modify morgue list? Please read: Morgue.

Actions taken by the Site Support Team

EVERY WEEK (Monday/Tuesday)

  • Site Readiness Plots:
COMBINED Plot SSB
3 MONTHS Plot Table w/ Stats
LAST WEEK Plot Table w/ Stats

  • Check last week's readiness for sites in waiting room
    • Update the tickets, close if above threshold.
  • Check last week and trimonth's readiness for all sites not in waiting room
    • follow procedure below for sites going into it.

  • Make summary table and place it in CompOps Meeting and Site Support Twikis.
    • Script to create this summary based on dashboard stats: Script
    • USE metrics shown in SSB: site_readiness view
      • Site Readiness last Week_3 months (red if <80%)
      • Waiting Room - YES/NO (YES=site do NOT have entry in the latest values of metric 39 file)

When a site goes IN to the waiting room

When a site comes OUT of the waiting room

Instructions to move sites in SSB

https://cmsdoc.cern.ch/cms/LCG/SiteComm/T2WaitingList/WasCommissionedT2ForSiteMonitor.txt

Longterm tasks

  • Automatic procedures - the idea of site readiness is that sites could be automatically included and excluded in the systems (WMAgent, CRAB, etc.)
  • Steps towards an automated system:
    • DONE - develop procedures moving sites in and out of waiting room
    • DONE - move sites in and out manually without them having an effect on the usage of the sites
    • DONE - report on statistics and tell the sites what we found and what we are going to do
    • DONE - allow WMAgents to exclude sites because of readiness (put them in drain), still move sites manually in and out of the waiting room
    • DONE: automate moving sites in and out - requires creating and linking new metrics in SSB site readiness and production views.
Topic attachments
I Attachment History Action Size Date Who Comment
Texttxt Sites_to_Waiting_Room.py.txt r1 manage 1.1 K 2013-06-13 - 18:09 JohnArtieda Script_TableSitesWR
Edit | Attach | Watch | Print version | History: r27 < r26 < r25 < r24 < r23 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r27 - 2016-04-11 - RokasMaciulaitis
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback