Ganglia Hierarchical Deployment

Introduction

This page aims to provide a generic understanding of how the distributed monitoring system Ganglia works and how can one deploy it, considering the working environment.This information can be faced as a deployment guide where many deployment possibilities will be considered, analyzed and explained.

To be more objective, this document will address mostly a working environment based on cloud infrastructures (private cloud, public clouds, hybrid clouds...).

NOTE: please bare in mind that this page is not trying to replace any of the official documentation, which can be found in Ganglia's Website. Therefore it is assumed that the user already has a basic understanding of what Ganglia is and what is it for.

NOTE 2: all the technical references made in this document are meant to be understood within a Linux based environment.

Before starting

NOTE: Legend of colors for the examples:

GRAY background for the commands to execute  (cut&paste)

GREEN background for the output sample of the executed commands

BLUE background for the configuration files (cut&paste)

Setting up Ganglia

In a few words, Ganglia is a monitoring system based on a headnode-node relationship. In technical terms, Ganglia uses the monitoring daemon gmond in the nodes to report their metrics back to the Ganglia meta daemon gmetad in the headnode. Then, on the headnode, all the data is pulled from the gmonds at regular intervals, saved in a Round-Robin Database (RRD) and displayed in real-time through Ganglia's web front-end (which is available at hostname.domain/ganglia).

Installation

Independently of the chosen deployment, the Ganglia installation will always be the same both in the headnode and workernodes. Only its configuration will change afterwards.

Headnode Installation

For a generic installation of the collector node, here's what's needed:

#!/bin/bash

yum -y install ganglia ganglia-gmond httpd php ganglia-gmetad ganglia-web

# Modify Apache configuration to define incoming accesses to the web-frontend
sed -i "s/<\/Location>/ Allow from cern.ch\n <\/Location>/g" /etc/httpd/conf.d/ganglia.conf

# Configuring the firewall might be needed, mostly for Apache. This depends on the system's local configurations
# If the headnode is within a private network and the nodes are outside, public firewall authorizations are needed as well

setsebool httpd_can_network_connect 1

chkconfig gmond on

service httpd restart
service gmond restart
service gmetad restart

Node Installation

For the nodes the installation is simpler:

#!/bin/bash

yum -y install ganglia ganglia-gmond

chkconfig gmond on

service gmond restart

Configuration

When it comes to configuring gmond and gmetad it will all depend on how is our system distributed. This system distribution and the way Ganglia is configured will then define how the final system is layered and therefore its hierarchy. Let's evaluate and enumerate the main characteristics of the most common Ganglia distribution layouts.

Standard (multicast)

Red —

= UDP connections

Blue —

= TCP connections

This is Ganglia's default layout. The standard gmond installation (as stated above) will provide this multicast broadcasting mechanism between VM's by default. While gmond uses multicast channels in a peer-to-peer way, gmetad pulls the XML description from Ganglia data sources (normally these data sources are one or more VM's within the same pool) via XML over unicast routes. In this case there's only the need to change gmetad configuration in the headnode. This is done through the file /etc/ganglia/gmetad.conf. Open this file

#!/bin/bash

vim /etc/ganglia/gmetad.conf

and add the data sources

# comment the default data_source
# data_source "another source" 1.3.4.7:8655  1.3.4.8

data_source "Cloud A" a1.domain:port1  ax.domain:portx   # Redundancy is allowed
# (same for Cloud B and Cloud C...etc)

# other configurations can be added but their are not essential for a good Ganglia behavior in multicast

NOTE: it is good practice to change the cluster name in gmond as well, to be the same as the one referred in gmetad configuration.

Here's some points worth considering when using multicast:

Advantages

• Standard setup
• Groups addresses
• Provides redundancy

Disadvantages

• Sometimes hard to setup (when nodes are in different subnets and/or VLAN's for example) and some environments don't even support it (i.e. Amazon's AWS)
• Too much chatter within the network
• Setup changes require restart of all gmonds
• In large scale, having all nodes communicating and saving information about the other nodes is a burden and not very useful
• With opportunistic resources and considering the short lifetime of the VM's, either the data sources in gmetad would have to change or the VM's would need some extra configurations

Unicast

Red —

= UDP connections

Blue —

= TCP connections

Another way of setting up Ganglia is through unicast instead of multicast.

Unicast means that every gmond will report its metric directly with an endpoint, instead of broadcasting this information to every gmond on its network. The official documentation suggests that to do this, we make use of a (one or more) receiving gmond to where all the other nodes report to and from where the gmetad get the data from.

These receiving gmonds act like intermediate collectors, and the machine where they are running are the ones which should be mentioned as data sources in gmetad.

For the above scheme here's how we would configure Ganglia:

In the headnode

#!/bin/bash

vim /etc/ganglia/gmetad.conf

# comment the default data_source
# data_source "another source" 1.3.4.7:8655  1.3.4.8

data_source "Cloud A" a1.domain:port1 
# (same for Cloud B and Cloud C...etc)

Now, the difference is in the nodes,

#!/bin/bash

vim /etc/ganglia/gmond.conf

udp_send_channel {
  #mcast_join = 239.2.11.72
  host = a1.domain
  port = port1
  ttl = 1
}

/* You can specify as many udp_recv_channels as you like as well. */
udp_recv_channel {
  #mcast_join = 239.2.11.72
  port = port1
  bind = 239.2.11.71
}

A good question is: is there a difference between the reporting gmonds and the receiving ones? Yes, BUT, it is a non-crucial difference for the system behavior. Basically, in the reporting nodes, the channels "udp_recv_channel" and "tcp_accept_channel" can be left empty.

NOTE: it is good practice to change the cluster name in gmond as well, to be the same as the one referred in gmetad configuration.

A quick evaluation in this configuration can be made by looking at the following points:

Advantages

• Reduces the "jitter" between nodes inside the network
• Setup changes are easier to implement
• Works in all environments

Disadvantages

• Metada information will be lost in the receiving gmond if this one is restarted (which will probably cause blank graphs in the headnode front-end)
• Doesn't solve the problem of having to re-configure gmetad every time there is a new receiving gmond (due to the nodes' short lifetime)
• Initial setup is more complex

Unicast (with remote middle gmond)

Red —

= UDP connections

Blue —

= TCP connections

A recurring disadvantage of the past two configurations is the lifetime of the gmond nodes within a cloud environment, which leads to constant changes in the headnode and other nodes.

For this reasons there is one small modification which can be made to the common unicast deployment which is moving the receiving gmond node outside the working environment and make it a dedicated collector&sender server.

The configuration for gmond in this dedicated machine is exactly the same as shown in the common unicast example above.

Advantages

• The middle gmond machine can be placed anywhere
• There's no problem on keeping this machine alive for an unlimited amount of time (unlike VM's)

Disadvantages

• Need to maintain a machine just for collecting and sending monitoring data
• Creating redundancy means creating more dedicated gmond servers

Unicast (without middle gmond)

Red —

= UDP connections

Blue —

= TCP connections

This layout is a custom version of the above, where some of the disadvantages are hereby eliminated. Considering the disadvantages of using multicast and baring in mind that this document mainly focus on Cloud environments, unicast is the way to go.

In the previous unicast scheme it was noticed that having an "middleman" gmond would basically harm as much as using multicast, since gmetad would still be collecting the data from a dedicated node. Basically, keeping a dedicated node inside a Cloud (or even outside) is a waste of resources and money.

This approach without the receiving gmond makes use of the headnode to collect all the data from all clusters. It is basically a moving of the "middleman" gmond from the previous layout to the headnode.

What happens here is that, the gmond nodes will still behave as before, but now there will be a gmond configuration in the headnode as well.

Technically speaking, in the headnode:

#!/bin/bash

vim /etc/ganglia/gmetad.conf

# comment the default data_source
# data_source "another source" 1.3.4.7:8655  1.3.4.8

data_source "Cloud A" localhost:port1 
data_source "Cloud B" localhost:port2
# etc

and now, by adding new gmond configuration files in the same machine we're saying that this is where all the data should coming directly to.

#!/bin/bash

vim /etc/ganglia/gmond_cloudA.conf

cluster {
  name = "Cloud A"
  owner = "unspecified"
  latlong = "unspecified"
  url = "unspecified"
}

/* The host section describes attributes of the host, like the location */
host {
  location = "unspecified"
}

udp_send_channel {
  host = localhost
  port = port1
  ttl = 1
}
udp_recv_channel {
  port = port1
}

/* You can specify as many tcp_accept_channels as you like to share
   an xml description of the state of the cluster */
tcp_accept_channel {
  port = port1
}

#!/bin/bash

vim /etc/ganglia/gmond_cloudB.conf

cluster {
  name = "Cloud B"
  owner = "unspecified"
  latlong = "unspecified"
  url = "unspecified"
}

/* The host section describes attributes of the host, like the location */
host {
  location = "unspecified"
}

udp_send_channel {
  host = localhost
  port = port2
  ttl = 1
}
udp_recv_channel {
  port = port2
}

/* You can specify as many tcp_accept_channels as you like to share
   an xml description of the state of the cluster */
tcp_accept_channel {
  port = port2
}

Then, finally, there's just the need to daemonize these new gmond configurations and restart gmetad:

#!/bin/bash

. /etc/rc.d/init.d/functions
daemon gmond -c gmond_cloudA.conf 
daemon gmond -c gmond_cloudB.conf
service gmetad restart

In the Cloud nodes, the gmond configuration will be exactly the same as in the regular unicast example, being the host endpoint parameter the only thing in need of changing.

Advantages

• No dependencies what so ever in remote nodes
• gmond configurations never change unless the headnode changes
• All the data and metadata is saved directly in the headnode
• There is a CERN module for Cloud-init which enables gmond contextualization, providing easy setup in the gmond nodes (Cloud-init@CERN)

Disadvantages

• Clearly creates a single point of failure
• Initial headnode setup requires some work
• Might create network congestion and/or packet loss (it is a UDP direct connection), in cases where there are many monitored clusters/cloud in very distant locations (latency)

This is the deployment we are currently using. For more information there is an attached PDF file in the end of this Twiki, addressing only this kind of approach.

See an example of this deployment at ATLAS Ganglia Monitor

Layering Ganglia

After considering the above configurations we can now address different layers that can be placed on top of them, either to simplify Ganglia's distribution or to attenuate some of the now known disadvantages.

Grid-of-Grids

Black —

= TCP connections

One very practical example is to create a layer on top of other gmetads. Usually one headnode monitors multiple clusters. This kind of setup can be called a Grid. Once one has multiple Grids, it is possible to deploy another gmetad on top of the existing ones, creating a "Grid-of-Grids", where each gmetad will answer requests for XML through a specific port (8651 by default).

To configure this layout one should:

• In gmetad1 and gmetad2

       Perform exactly the same configurations as it was done before, but add the following in /etc/ganglia/gmetad_conf

# List of machines this gmetad will share XML with. Localhost
# is always trusted. 
# default: There is no default value
trusted_hosts 127.0.0.1 top-level-gmetad.domain

• In top level *gmetad*

#!/bin/bash

vim /etc/ganglia/gmetad.conf

#...
# data_source "another source" 1.3.4.7:8655  1.3.4.8

data_source "gmetad1 grid" gmetad1.domain:8651
data_source "gmetad2 grid" gmetad2.domain:8651

#
# Round-Robin Archives
# You can specify custom Round-Robin archives here (defaults are listed below)
#...
# The name of this Grid. All the data sources above will be wrapped in a GRID
# tag with this name.
# default: unspecified
gridname "MyGridofGrids"
#...
#-------------------------------------------------------------------------------
# If you want any host which connects to the gmetad XML to receive
# data, then set this value to "on"
# default: off
all_trusted on
#...
#...

Please note what is said in gmetad configuration

# The data_source tag specifies either a cluster or a grid to
# monitor. If we detect the source is a cluster, we will maintain a complete
# set of RRD databases for it, which can be used to create historical
# graphs of the metrics. If the source is a grid (it comes from another gmetad),
# we will only maintain summary RRDs for it.

Advantages

• Creates a clean and intuitive organization of the Ganglia distribution structure
• Doesn't affect the monitored gmetad neither adds a big amount of data transfer coming from them (since it only maintains the summary RRDs)

Adding Redundancy

As it was said before, most of these configuration suffer from the Single Point of Failure disadvantage. This is something which can be avoided by adding redundancy to the system.

Redundancy can be achieved through several means. One can do it through non-Ganglia related approaches, like external servers copying the RRD database from the gmetad servers, creating backups. We can see a simple example of this layout in the following image.

Advantages

• Data is backed up in a totally independent and Ganglia-unrelated machine

Disadvantages

• Need of an extra server just to keep a backup database. Not worth the maintenance unless it is doing something else besides just backing up gmetad data

Another possible way is to make use of Ganglia's gmond and gmetad to create duplicates of the already running services, like it is shown in the next image.

In this last layout, extra Ganglia elements are added to achieve some redundancy. Basically, all the workernodes would be reporting not to one single receiving gmond but to more (in this image, two), which will afterwards send their collected data to their own gmetad servers. Although this configuration provides both redundancy on a gmond and gmetad level, what we really care is about preserving the monitoring data, which is saved in gmetad machine.

This layout requires configuration in the three Ganglia layers:

• Workernodes

#!/bin/bash

vim /etc/ganglia/gmond.conf

udp_send_channel {
  host = g1.domain
  port = port1
  ttl = 1
}
udp_send_channel {
  host = g2.domain
  port = port2
  ttl = 1
}

• Receiving gmond (where _x_= 1 or 2)

#!/bin/bash

vim /etc/ganglia/gmond.conf

udp_send_channel {
  host = gmetadx.domain
  port = portx
  ttl = 1
}

• Headnode (where _x_= 1 or 2)

#!/bin/bash

vim /etc/ganglia/gmetad.conf

#...

data_source "gx" gx.domain:portx

#

Advantages

• Redundancy both in gmond and gmetad
• Two different gmetad servers with the same information and the same functionality

Disadvantages

• Complex setup in the workernodes - need to send metric to two different endpoints
• Although the extra nodes are Ganglia machines, those also still need maintenance

Finally, if we trust enough in the receiving gmond, we can abbreviate the previous scheme to look like the following.

In this layout, the workernodes have a regular gmond configuration, being only necessary to configure the receiving gmond and both gmetad.

• Receiving gmond

#!/bin/bash

vim /etc/ganglia/gmond.conf

udp_send_channel {
  host = gmetad1.domain
  port = port1
  ttl = 1
}
udp_send_channel {
  host = gmetad2.domain
  port = port2
  ttl = 1
}

• Headnode (where _x_= 1 or 2)

#!/bin/bash

vim /etc/ganglia/gmetad.conf

#...

data_source "g1" g1.domain:portx

#

Advantages

• Redundancy for the metrics database
• Two different gmetad servers with the same information and the same functionality
• No need for additional gmond collector
• Simple setup

Disadvantages

• If the gmond collector fails there is no way around

-- CristovaoCordeiro - 21 Mar 2014