Msg Server (Broker) Details


MSG uses ActiveMQ as the broker implementation within EGEE. This document outlines the architecture and configuration details for a single clustered broker within the federated broker network.


We wish to run a high-availability broker pair, in order to protect clients against downtimes that require restart of a machine - e.g. kernel upgrades. To do this, we use the Master-Slave capabilities of ActiveMQ. In particular, we use the shared-filesystem master-slave configuration. This gives the best features in terms of recovery after failure and performance. It requires a shared filesystem to be shared between the nodes in the ActiveMQ cluster. This can be done using e.g. a SAN, or NFS.

This implies that two machines are required for the master-slave pair, and that either a SAN storage array or NAS storage box is required for the shared storage.

HTTP & STOMP connection architecture

We provide 2 access protocols to the server:

  • STOMP - ActiveMQ has native STOMP support built in. This normally runs on port 61613, but for our application we will run it on 6163 to avoid very high numbered ports.
  • HTTP REST - ActiveMQ provides a simple REST-ful interface. this is useful for sending data from worker nodes which can only proxy HTTP externally. We recommend this is only used for publishers, and in particular ones who send small amounts of informationat irregular intervals. The REST-ful interface resides at the URL

The STOMP interface is the recommended way for most clients to connect to the Broker. The physical layout, showing the load-balancing and failover between the components is shown below.


CERN specific implementation details

At CERN, we use 2 x86_64 servers as the front-end nodes along with a 3rd x86_64 machine exporting a shared NFS volume to the 2 servers.

SRCE specific implementation details

ActiveMQ is installed on RedHat Cluster Suite platform. Data directory is stored on SAN volume. Nodes are Sun Fire X4240 with 2 Quad Core AMD Opteron 2356 processors and 16 GB RAM.

Architecture of the whole HA cluster is shown on the picture below. Besides for ActiveMQ HA cluster currently hosts top level BDII and regional Nagios instance. RedHat Cluster resource manager handles virtual IP, mounting of data storage and individual services. In case when node or service failure is detected, resource manager automatically moves everything to one of the healthy node.


Getting the broker

Installation via YAIM

You need to configure the following YUM repositories

If you are on 32 bit architecture, you also need the architecture specific Jpackage repository (for tanukiwrapper)

If you are on 64 bit architecture, you will need to modify the glite-UI repo, replacing $basearch with i386.

Now to install the server, you just need tio install the EGEE-MSG-broker meta-package.

     > yum install egee-MSG-broker

Configuration details

The configuration requires you to set the following variables in the YAIM site-info.def file:
Variable Description Example
INSTALL_ROOT Location of grid middleware. /opt
SITE_NAME the site where the MSG server is deployed MY-SITE
VOS A list of VOs who can use the MSG server. "ops dteam alice"

The following variables can be used if you have a failover configuration as mentioned above. They are used by the information providers to publish correct connection URIs as well as in the configuration files for the broker.

Variable Description Example
MSG_FAILOVER_HOST the hostname of the other server, if this one is in a failover pair
MSG_SERVER_ALIAS A DNS alias used to connect to the HTTP frontends

The following variables have defaults, which you may want to change in a non-standard configuration

Variable Description Default
MSG_CLUSTER The cluster name for a given network of brokers PROD
MSG_MAX_MEM The maximum system memory the broker will use 2048 mb
MSG_SHAREDFS_DIR the directory on the shared filesystem used for the persistent store /mnt/sharedfs/activemq/amqstore

OS Configuration

ActiveMQ opens a file for each topic and queue available. In the current setup this breaches the default limit open file descriptors (1024). Before starting the activemq service it is recommended to increase this value in file /etc/security/limits.conf, e.g.:
activemq        soft    nofile          32768
activemq        hard    nofile          32768

ActiveMQ Configuration

The current configuration (/etc/activemq/activemq.xml) used on the production servers is :


  <!-- Allows us to use system properties as variables in this configuration file -->
  <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>
  <broker xmlns="" brokerName="${MSG_CLUSTER}-${HOSTNAME}" persistent="true" useShutdownHook="true" >

    <!-- The transport connectors ActiveMQ will listen to -->
       <transportConnector uri="tcp://${HOSTNAME}:6166" />
       <transportConnector name="stomp" uri="stomp://${HOSTNAME}:6163"/>

    <!-- Currently we're a stand-alone broker pair -->
       <!--<networkConnector uri="multicast://foo"/>-->

    <!-- we're part of a Master-slave pair, using a shared FS -->
        <amqPersistenceAdapter directory="${MSG_SHAREDFS_DIR}"  maxFileLength="32mb"/>

    <!-- turn on the logging interceptor so we can see what is happening -->
      <!-- lets enable detailed logging in the broker -->

    <!-- Configure how ActiveMQ is exposed in JMX -->
       <managementContext connectorPort="1099" jmxDomainName="org.apache.activemq"/>

    <!--  Use the following to set the broker memory limit -->
    <!--  The maximum about of space the broker will use before slowing down producers -->
            <memoryUsage limit="${MSG_MAX_MEM}"/>
            <tempUsage limit="${MSG_MAX_MEM}"/>


  <!-- lets create a command agent to respond to message based admin commands on the ActiveMQ.Agent topic -->
  <commandAgent xmlns=""/>

  <!-- An embedded servlet engine for serving up the Admin console -->
  <jetty xmlns="">
      <nioConnector port="8161" />
      <webAppContext contextPath="/admin" resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true" />
      <webAppContext contextPath="/logtail" resourceBase="${activemq.base}/webapps/logtail" logUrlOnStart="true" />


Monitoring and alarms

Discovery of a Broker via a BDII

Known issues

  • Advisory topics ActiveMQ.Advisory.NoConsumer.Queue and ActiveMQ.Advisory.NoConsumer.Topic don't work as they should. Fixed in SVN revision 745031. They'll be available in the next stable release (ActiveMQ 5.3.0). Bug report.

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng SrceClusterSetup.png r1 manage 61.8 K 2009-02-25 - 16:06 UnknownUser Architecture of multipurpose HA cluster at SRCE.
Edit | Attach | Watch | Print version | History: r13 | r11 < r10 < r9 < r8 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r9 - 2009-03-09 - JoseDana
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EGEE All webs login

This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright & by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Ask a support question or Send feedback