Grid Publisher Development


The goal of this wiki page is to plan but also track the work on Grid Publisher foreseen by the Monitoring Working Group.


Phase I

Goal: Set up a simple ActiveMQ based environment and test it using GridFTP log data:
  • take historical logs from CERN production services to replay or generate simulated data transfer events
  • adapt log parser (Python) to publish to ActiveMQ broker
  • set up a single instance of broker under Tomcat application server
  • develop a simple message summarizer (Java) that consumes GridFTP log messsages in ActiveMQ, does simple summarisation (for example number of events per minute) and publishes the results back to ActiveMQ
  • put an additional consumer that simply reads the messages
  • test the capabilities of the system in following dimentions:
    • number of messages per second
    • number of parallel consumers/producers
    • number of topics
    • complexity of JMS selector


  • machines for test installation provided by James (lxb6117, lxb6118)
  • an existing Python implementation of STOMP protocol pointed out by James, to be evaluated/extended by Wojtek

Phase II

Goal: Evaluate possible broker network configurations, set up and test the most promising one(s), test durable subscriptions
  • enable persistent messaging on broker running at lxb6117 (MySQL database)
  • provide high availability and fault tolerance by Master-Slave broker configurations (Shared Filesystem based, JDBC based)
  • test networks of brokers and brokers' discovery methods
  • test client-side dynamic failover

Phase III

Goal: Develop security mechanisms by using digital signatures and encryption on the message level with Grid certificates (proxies).

Details to come

Progress tracking

2007-07-19 - planning meeting

  • Participants: James Casey, Ian Neilson, Wojciech Czech, Piotr Nyczyk
  • Goals achieved: Initial planning

2007-07-30 - phase I tests report

2007-08-09 - phase II tests report

2007-08-09 - evaluation of tests results

  • Participants: James Casey, Ian Neilson, Wojciech Czech, Piotr Nyczyk
  • Goals achieved: Planning to test prototype in SAME configuration

2007-09-19 - Grid Messaging System Presentation

2007-10-25 - ActiveMQ extended Tests Presentation and Report

  • Participants: Max Boehn, James Casey, David Collados, Gavin Mccance, PiotrNyczyk, Daniel Rodrigues
  • Summary: Presentation of the 1st month of work on the MSG ( Messaging Service For Grids )
  • Final Remarks:
    • Double Check inconsistent results on the ActiveMQ broker capabilites
    • Further test with more fine-grained measurements, namely consumers throughput and latencies.
    • Stick to ActiveMQ 5.0, unless unusable
    • Time visualization on impact of aditional producers or subscribers. ( graphs of messages/second per each producer/ consumer )

Topic attachments
I Attachment History Action Size Date Who Comment
PowerPointppt Evaluation_of_ActiveMQ_broker_for_the_Grid_Messaging.ppt r1 manage 50.5 K 2007-10-25 - 16:26 DanielRodrigues First Month Work Presentation
PDFpdf gms-presentation.pdf r1 manage 398.6 K 2007-09-19 - 17:38 DanielRodrigues Grid Messaging System Presentation
PDFpdf report-A-18Oct2007.pdf r1 manage 714.0 K 2007-10-25 - 16:24 DanielRodrigues First Month Work Report
PDFpdf report1.pdf r1 manage 58.5 K 2007-07-30 - 16:24 PiotrNyczyk Report on phase I tests
PDFpdf report2.pdf r1 manage 72.9 K 2007-08-13 - 16:48 UnknownUser Report on phase II tests
Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r5 - 2007-10-25 - DanielRodrigues
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback