Msg Server (Broker) Details

Introduction

EGEE uses ActiveMQ as the broker implementation. This document outlines the architecture and configuration details for the brokers within the federated broker network. See here on how to install a broker using YAIM.

Architecture

Where possible we wish to run a high-availability broker pair, in order to protect clients against downtimes that require restart of a machine - e.g. kernel upgrades. To do this, we use the Master-Slave capabilities of ActiveMQ. In particular, we use the shared-filesystem master-slave configuration. This gives the best features in terms of recovery after failure and performance. It requires a shared filesystem to be shared between the nodes in the ActiveMQ cluster. This can be done using e.g. a SAN, or NFS.

This implies that two machines are required for the master-slave pair, and that either a SAN storage array or NAS storage box is required for the shared storage.

Site Specific Deployment details

CERN

At CERN, we use 2 x86_64 servers as the broker nodes (gridmag101 + grimsg102(in new arch). These are currently separate brokers in the network and not failover due to lack of availability of a production SAN @ CERN.

SRCE

ActiveMQ is installed on RedHat Cluster Suite platform. Data directory is stored on SAN volume. Nodes are Sun Fire X4240 with 2 Quad Core AMD Opteron 2356 processors and 16 GB RAM.

Architecture of the whole HA cluster is shown on the picture below. Besides for ActiveMQ HA cluster currently hosts top level BDII and regional Nagios instance. RedHat Cluster resource manager handles virtual IP, mounting of data storage and individual services. In case when node or service failure is detected, resource manager automatically moves everything to one of the healthy node.

SrceClusterSetup.png

--+++ AUTH

--+++ INFN

Getting the broker

OS Configuration

ActiveMQ opens a file for each topic and queue available. In the current setup this breaches the default limit open file descriptors (1024). Before starting the activemq service it is recommended to increase this value in file /etc/security/limits.conf, e.g.:
activemq        soft    nofile          32768
activemq        hard    nofile          32768

ActiveMQ Configuration

The current configuration (/etc/activemq/activemq.xml) used on the production servers is :

<beans
  xmlns="http://www.springframework.org/schema/beans"
  xmlns:amq="http://activemq.org/config/1.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
  http://activemq.org/config/1.0 http://activemq.apache.org/schema/activemq-core.xsd
  http://activemq.apache.org/camel/schema/spring http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">

  <!-- Allows us to use system properties as variables in this configuration file -->
  <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>
  <broker xmlns="http://activemq.org/config/1.0" brokerName="${MSG_CLUSTER}-${HOSTNAME}" persistent="true" useShutdownHook="true" >

    <!-- The transport connectors ActiveMQ will listen to -->
    <transportConnectors>
       <transportConnector uri="tcp://${HOSTNAME}:6166" />
       <transportConnector name="stomp" uri="stomp://${HOSTNAME}:6163"/>
    </transportConnectors>

    <!-- Currently we're a stand-alone broker pair -->
    <networkConnectors>
       <!--<networkConnector uri="multicast://foo"/>-->
    </networkConnectors>

    <!-- we're part of a Master-slave pair, using a shared FS -->
    <persistenceAdapter>
        <amqPersistenceAdapter directory="${MSG_SHAREDFS_DIR}"  maxFileLength="32mb"/>
    </persistenceAdapter>

    <!-- turn on the logging interceptor so we can see what is happening -->
   <plugins>
      <!-- lets enable detailed logging in the broker -->
      <loggingBrokerPlugin/>
    </plugins>

    <!-- Configure how ActiveMQ is exposed in JMX -->
    <managementContext>
       <managementContext connectorPort="1099" jmxDomainName="org.apache.activemq"/>
    </managementContext>

    <!--  Use the following to set the broker memory limit -->
    <!--  The maximum about of space the broker will use before slowing down producers -->
    <systemUsage>
       <systemUsage>
          <memoryUsage>
            <memoryUsage limit="${MSG_MAX_MEM}"/>
          </memoryUsage>
          <tempUsage>
            <tempUsage limit="${MSG_MAX_MEM}"/>
          </tempUsage>
       </systemUsage>
    </systemUsage>

  </broker>

  <!-- lets create a command agent to respond to message based admin commands on the ActiveMQ.Agent topic -->
  <commandAgent xmlns="http://activemq.org/config/1.0"/>

  <!-- An embedded servlet engine for serving up the Admin console -->
  <jetty xmlns="http://mortbay.com/schemas/jetty/1.0">
    <connectors>
      <nioConnector port="8161" />
    </connectors>
    <handlers>
      <webAppContext contextPath="/admin" resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true" />
      <webAppContext contextPath="/logtail" resourceBase="${activemq.base}/webapps/logtail" logUrlOnStart="true" />
    </handlers>
  </jetty>

</beans>

Monitoring and alarms

Discovery of a Broker via a BDII

Known issues

  • Advisory topics ActiveMQ.Advisory.NoConsumer.Queue and ActiveMQ.Advisory.NoConsumer.Topic don't work as they should. Fixed in SVN revision 745031. They'll be available in the next stable release (ActiveMQ 5.3.0). Bug report.

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng SrceClusterSetup.png r1 manage 61.8 K 2009-02-25 - 16:06 EmirImamagic Architecture of multipurpose HA cluster at SRCE.
Edit | Attach | Watch | Print version | History: r13 < r12 < r11 < r10 < r9 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r10 - 2009-11-30 - unknown
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EGEE All webs login

This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright & by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Ask a support question or Send feedback