MSG Format Choices

Introduction

The message body is transparent to the message broker (with exceptions, but these are no recommended !) and it is an application choice to choose a message format which is appropriate.

Until now we've been using the WLCG Grid Monitoring Probe Format which initially came out the SAM project. This is used for several systems:

• Nagios MSG integration
• OSG RSV result publication
• OSG Downtime publication
• gridftp/jobstatus log publication
• Grid configuration management system

It has several limitations. The two main ones are:

• It only supports flat records of key-value pairs. This works well in a pseudo-R-GMA application where you consider each message is a database row, but doesn't work so well for applications with structured records
• It doesn't support multi-line records. It has hard-coded support for one multi-line records DetailsData, which must be the last record.

Also, the current specification is intermixed with details on how to run WCLG format monitoring probes, even through the specification to describe messages that has been used outside this context.

In this document we describe some alternative rendering methods which could be used, and show how a simple standard example (of a nagios probe record ) could be represented.

Formats

WLCG Format

• Pros:
  • The tooling (e.g. msg-publish-simple and msg-consume2oracle) already support it.
  • It works for the applications that use it now.
• Cons:
  • It's a bespoke standard. We always have problems when writing new tools using it, and implicit assumptions in the tooling that aren't specified fully.
  • Doesn't support structured data

Example

Note in this example, we can't show any performance data easily.

serviceURI: wms202.cern.ch:2170
serviceType: BDII
siteName: CERN-PROD
metricStatus: OK
metricName: org.bdii.Freshness
summaryData: OK: createTimestamp=Thu Mar 12 10:29:42 2009 UTC, diff=1 min
gatheredAt: lxbra1905.cern.ch
timestamp: 2009-03-12T10:30:06Z
nagiosName: org.bdii.Freshness
performanceData0Name: diff
performanceData0Value: 1 min
EOF

YAML

YAML is a human-friendly markup language which has good support already in many languages. The WLCG format is (by accident, not design !) nearly a subset of YAML - the only incompatibility is the record-separator.

• Pros:
  • It's a standard
  • It supports hierarchical records, and references
  • Standard parsers available for most major languages
  • It's nearly forward compatible with WLCG Format
• Cons:
  • Still quite chatty text-based protocol
  • More difficult to generate and parse than simpler formats like JSON

Example

---
serviceURI: wms202.cern.ch:2170
serviceType: BDII
siteName: CERN-PROD
metricStatus: OK
metricName: org.bdii.Freshness
summaryData: OK: createTimestamp=Thu Mar 12 10:29:42 2009 UTC, diff=1 min
gatheredAt: lxbra1905.cern.ch
timestamp: 2009-03-12T10:30:06Z
nagiosName: org.bdii.Freshness
performanceData: 
  - diff 
    1 min
...

JSON

• Pros:
  • xxx
• Cons:
  • xxx

Example

XML

• Pros:
  • xxx
• Cons:
  • xxx

Example

GData

• Pros:
  • xxx
• Cons :

Example

---

JamesCasey - 13 Mar 2009