ILCDIRAC monitoring pages

JIRA

https://its.cern.ch/jira/browse/ILCDIRAC

Shifter pages to check

• SiteSummary: Look for sites or CEs that show many failed jobs or do not have any jobs.
• Job and Pilot status: Check number of failed jobs for certain users, sites, JobGroups. Aborted Pilots. Sites suddenly dropping out.
• Load from Services/Agents: Look for sudden increase or decrease of load. Change to week overview
• CastorStatus: Check free space on disk pool
• SystemOverview: Check error logs, check systemload
• Pilot Summary: Check if sites has many aborted pilots, or no pilots at all
• JobMonitor: Detailed view of jobs
• ILCDirac Server Dashboard: Machine status. check for high load, swap space, anything out of the ordinary...

• IT Status Board: Incidents, announcements, changes of CERN IT infrastructure

Grid Resource Sites

Websites offering overview or portals for the grid:

• GGus: Ticketing for LCG (and some OSG) sites
• GStat: Information about Sites, their queues and resources
• GOCDB: Downtime and Resource overview

Overview sites

The following links point to useful pages to monitor the status of the system:

• CERN Storage: https://sls.cern.ch/sls/service.php?id=CASTORPUBLIC_ILCDATA
• GSTAT status overview: http://gstat-prod.cern.ch/gstat/summary/VO/ilc/
• EGI accounting: https://accounting.egi.eu/egi.php
• OSG Information: http://myosg.grid.iu.edu/

Debugging CEs and SEs

How to directly access the underlying CEs for more direct debugging of issues: Cream, ARC, Globus and SEs/FileCatalogs

• ILCDiracDebuggingSites

Checking the Status of Machines, Agents, and Services

• Check Machine Status
• Things that normally go wrong:
  • Pilots aborted at Site/CE

Castor Diskpool, Unstaging Files

When the castor diskpool is getting full, one needs to unstage files from it.
Clone the ILCDiracOps git repository and run the script
StagerScripts/UnstageProdFiles -P'prodID' -F'REC|SIM'
prodID is the first production to check, REC|SIM the file types to unstage.

Production Output Data Checking