WLCG Transfers Dashboard Latency Monitoring

This page documents the development of latency monitoring, which is a component of WLCGTransfersDashboard.

Requirements

TBD

See also WLCGTransfersDashboard#Email_30_05_2012_from_Alessandro

Development

Environment:

• Development branch of code in SVN
  • NAME: dashboard-transfers-0-3-0-branch
  • WEB_URL: http://svnweb.cern.ch/world/wsvn/dashboard/branches/arda.dashboard.transfers/dashboard-transfers-0-3-0-branch/arda.dashboard.transfers/
  • SVN_URL: https://svn.cern.ch/reps/dashboard/branches/arda.dashboard.transfers/dashboard-transfers-0-3-0-branch/arda.dashboard.transfers
  • NOTES:
    • Use "python setup.py release -c" to release new 0.3.1_rcX RPM.
    • There is a skeleton view object for the Latency tab with comments: view_latency.js
• Development database account.
  • Use developer db account initially.
• Development VM.
  • Use developer machine initially.
• NOTES:
  • See also: Dashboard documentation, Xbrowse documentation.

• Develop on dev SVN branch.
• Deploy to development environment.
• On delivery of a functional milestone:
  • review functionality with stakeholders
  • review code
  • merge to SVN trunk
  • deploy/test in integration
  • deploy/test in production

Milestones

1. Application setup in the testbed : https://dashb-wlcg-transfers-dev-latency.cern.ch

Delivered

1. Application presented to the ADC monitoring meeting, 22/08/2012
   • Link to the presentation: ADC-latency-monitoring.pdf
   • Waiting for Feedback
2. First requirements reached. PRESENTED 10/07/2012
   • Link to the presentation: TransfersLatencyUpdate.pdf
   • Include a large set of chart breakdown (vo, server, channel, source and destination country, site)
   • Time bins can be customize (from 5 minutes to 1 hour) as well their number (from 1 to 10)
   • Include a prototype for channel filtering (working but subject to further discussion)
3. First proposal presentation. PRESENTED 20/06/2012
   • Link to the presentation: TransfersLatency-FirstProposal.pdf
4. Create development branch of code in SVN. DELIVERED 06/06/2012
   • Create development branch of code in SVN.
   • Create skeleton view object for latency tab. * Deliverable: * Development environment section of twiki updated with SVN links.

Completed

• DB and collector:
  • Idea of a single table containing only active transfers have been dropped (We want to keep history)
  • 3 new tables have been added:
    • t_transfers: snapshot of the active transfers
    • t_transfers_stats: 10 minutes aggregation data from t_transfers
    • t_transfers_stats_a: daily aggregation data from t_transfers
• User Interface:
  • Latency, latency history and ranking plots tabs have been added
  • Filter by Channel
  • Detailled page with running transfers

In progress

• Xbrowse plugin for plots and table

Future (draft)

• User Interface:
  • Investigation on the plot wanted by CMS (channel occupancy, stream / throughput, SRM overhead)
• Add new metrics: * major requests: statistics per channel/host * config: N files in parallel -> drilldown per file * change the number of streams by the number of files

Links

• User Interface: http://dashb-wlcg-transfers-dev-latency.cern.ch
• SVN: trunk, branch
• Bugs: Savannah
• Twikis: WLCGTransferMonitoring, WLCGTransfersDashboard.

• TransfersLatency-FirstProposal.pdf: TransfersLatency-FirstProposal.pdf