Grid Publisher Development

Introduction

The goal of this wiki page is to plan but also track the work on Grid Publisher foreseen by the Monitoring Working Group.

Plan

Phase I

Goal: Set up a simple ActiveMQ based environment and test it using GridFTP log data:
  • take historical logs from CERN production services to replay or generate simulated data transfer events
  • adapt log parser (Python) to publish to ActiveMQ broker
  • set up a single instance of broker under Tomcat application server
  • develop a simple message summarizer (Java) that consumes GridFTP log messsages in ActiveMQ, does simple summarisation (for example number of events per minute) and publishes the results back to ActiveMQ
  • put an additional consumer that simply reads the messages
  • test the capabilities of the system in following dimentions:
    • number of messages per second
    • number of parallel consumers/producers
    • number of topics
    • complexity of JMS selector

Details:

  • machines for test installation provided by James (lxb6117, lxb6118)
  • an existing Python implementation of STOMP protocol pointed out by James, to be evaluated/extended by Wojtek

Phase II

Goal: Evaluate possible broker network configurations, set up and test the most promising one(s), test durable subscriptions
  • enable persistent messaging on broker running at lxb6117 (MySQL database)
  • provide high availability and fault tolerance by Master-Slave broker configurations (Shared Filesystem based, JDBC based)
  • test networks of brokers and brokers' discovery methods
  • test client-side dynamic failover

Phase III

Goal: Develop security mechanisms by using digital signatures and encryption on the message level with Grid certificates (proxies).

Details to come

Progress tracking

2007-07-19 - planning meeting

  • Participants: James Casey, Ian Neilson, Wojciech Czech, Piotr Nyczyk
  • Goals achieved: Initial planning

2007-07-30 - phase I tests report

2007-08-09 - phase II tests report

2007-08-09 - evaluation of tests results

  • Participants: James Casey, Ian Neilson, Wojciech Czech, Piotr Nyczyk
  • Goals achieved: Planning to test prototype in SAME configuration

2007-09-19 - Grid Messaging System Presentation

2007-10-25 - ActiveMQ extended Tests Presentation and Report

  • Participants: Max Boehn, James Casey, David Collados, Gavin Mccance, PiotrNyczyk, Daniel Rodrigues
  • Summary: Presentation of the 1st month of work on the MSG ( Messaging Service For Grids )
  • Final Remarks:
    • Double Check inconsistent results on the ActiveMQ broker capabilites
    • Further test with more fine-grained measurements, namely consumers throughput and latencies.
    • Stick to ActiveMQ 5.0, unless unusable
    • Time visualization on impact of aditional producers or subscribers. ( graphs of messages/second per each producer/ consumer )



2008-01-29 - Openlab Presentation ( MSG: An Overview on a Messaging System for Grid )


  • Summary: Presentation on test results and further directions for the MSG in Openlab Quarterly Meeting
  • Further Steps:
    • Test with different topics, queues, + master slave configurations;
    • Implement SAM messaging (based on Piotr Prototype
  • MSGOpenlab_v1.2.pps: MSG Openlab Presentation

2007-01-30 - Test Plan


Further test plans:

clients on different machines, improvedClient 1 Consumer, 1 Producer 3Consumers, 3 Producers 5 Consumers, 5 Producers
Queue Performance 1 Consumer 1 Producer 3 Consumers 3 Producers, 1 queue 3 Consumers, 3 Producers, different queues
Master/Slave 1 Consumer 1 Producer Queue 3 Consumers 3 Producers, 1 topic 100 Producers, 1 Consumer, Queue
Master/Slave,SAN      
Master/Slave,JDBC      
*Network Of Brokers      



Topic attachments
I Attachment History Action Size Date Who Comment
PowerPointpps AnOverviewOnAMessagingSystemForTheGrid_v1.2.pps r1 manage 1453.4 K 2008-01-30 - 15:14 DanielRodrigues MSGOpenlabPresentation
PowerPointppt Evaluation_of_ActiveMQ_broker_for_the_Grid_Messaging.ppt r1 manage 50.5 K 2007-10-25 - 16:26 DanielRodrigues First Month Work Presentation
PDFpdf gms-presentation.pdf r1 manage 398.6 K 2007-09-19 - 17:38 DanielRodrigues Grid Messaging System Presentation
PDFpdf report-A-18Oct2007.pdf r1 manage 714.0 K 2007-10-25 - 16:24 DanielRodrigues First Month Work Report
PDFpdf report1.pdf r1 manage 58.5 K 2007-07-30 - 16:24 PiotrNyczyk Report on phase I tests
PDFpdf report2.pdf r1 manage 72.9 K 2007-08-13 - 16:48 UnknownUser Report on phase II tests
Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r8 - 2008-01-30 - DanielRodrigues
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback