Monitoring FAX

• FSB FAX Status Board

TWiki pages

• Current coverage
• Names, addresses and states of all FAX access points
• Remotely Obtained XRootD information
• Queues enabling FAX failover
• Queues FAX overflow settings

Site Status Board (SSB)

• Endpoints tests Direct, upstream, downstream, X509
• Redirectors test Upstream, downstream
• FAX cost matrix Transfer rates by xrdcp
• FAX specialists Delays, tests of the monitoring chain

DDM Dashboard

• Transfers dashboard
  • This dashboard situated at CERN, presents information sent to it via ActiveMQ from two collectors (UCSD and CERN). Point of contact is Julia Andreeva <andreeva@NO.mail.SPAM.cern.ch>.
• Who collects what...
• Detailed - Summary comparator

Kibana Dashboards

• overflow job efficiencies.
• overflow resource efficiency.

More Monitoring links

• Failover monitoring
• *Overflow Monitor * - http://dashb-atlas-job.cern.ch/dashboard/request.py/dailysummary#button=successfailures&resourcetype=All&sitesSort=0&timerange=last48&granularity=Hourly&generic=0&sortby=0&series=30&transfertypes[]=fax&start=null&end=null&pandares[]=ANALY_AGLT2_SL6&pandares[]=ANALY_BNL_LONG&pandares[]=ANALY_BNL_SHORT&pandares[]=ANALY_BU_ATLAS_Tier2_SL6&pandares[]=ANALY_CONNECT&pandares[]=ANALY_CONNECT_SHORT&pandares[]=ANALY_HU_ATLAS_Tier2&pandares[]=ANALY_MWT2_SL6&pandares[]=ANALY_OU_OCHEP_SWT2&pandares[]=ANALY_SWT2_CPB&pandares[]=ANALY_SFU&pandares[]=ANALY_SLAC&pandares[]=ANALY_TRIUMF&pandares[]=ANALY_VICTORIA&pandares[]=ANALY_MCGILL&pandares[]=ANALY_SCINET&pandares[]=ANALY_AUSTRALIA&sites[]=All+Sites&sitesCat[]=All+Countries&sitesCatSort=0

• FDR monitor FAX Full Dress Rehearsal job submissions
• Monitors of redirector services hosted by CERN
  • Lemon based
  • Kibana/ES based

• http://atl-prod07.slac.stanford.edu:8080/display Site level Xrootd traffic monitoring.
  • Again, the xrd.report line above will enable your site. Site naming schema is derived from this script: http://ivukotic.web.cern.ch/ivukotic/dropbox/host_to_site.txt in case you want it changed please send a mail to the author of this page (Ilija Vukotic).
• http://atl-prod05.slac.stanford.edu:4242/ Real time event monitoring
  • In case of XRootD site the xroort.monitor line above will enable your site. This will be migrated to use monitoring f-stream.
• WAN transfer viewer Latency, copy and direct read performance
  • Results fed by automated submission of jobs between pairs of endpoints
  • Choose server and client combinations
  • Showing excerpts from OracleDB collecting HC measurement results
• AGIS FAX topology Redirector hierarchy in AGIS
  • The FAX network as represented in the ATLAS information system (JSON format)

Expert material

Proposed FAX monitoring schema:

Current FAX monitoring schema

xRootD monitoring Protocol

The latest version of the protocol can be found here: http://xrootd.slac.stanford.edu/doc/prod/xrd_monitoring.htm

Geting information from AGIS

A short python script retrieves and prints information on sites and their first redirectors from AGIS. You may find it here: FAXlist.py.txt. You should remove '.txt' appended by twiki. Except modestly recent python version, there are no other dependencies.

Dashboard monitoring for ADC operations

Information collected each half an hour for FAX sites that are ACTIVE in AGIS:

• Gives its site-unique file directly from the site's redirector
• Redirects one level up on request for a non-existing file at the site
• Request to its parent redirector gives the site-unique file
• If it gives a site-unique file without having any kind of voms proxy
• Checks in what time xrd existfile will return while looking for an unexisting file.

The tests are run as a cron job on the lxplus cluster. The source code can be found here: https://github.com/ivukotic/FAXtools/tree/master/FAXconfiguration. Results are sent by ActiveMQ to a dedicated machine (atlas-faxmon) at CERN. There a cron job running from an account of adcmusr3 user, accepts the messages and puts them in json files. Log files of each of the tests can be obtained by simply clicking on it.

FAX JIRA pages

CERN JIRA

USATLAS JIRA

---

Major updates:
-- IlijaVukotic - 26-Dec-2012

Responsible: IlijaVukotic
Last reviewed by: Never reviewed