%CERTIFY%

TestTopic303

Introduction

This is a study on the performance aspects of the ActiveMQ software, which is currently used as the Message-Oriented-Middleware (MOM) for Atlas DaqAssistant. The information listed hereunder was compiled as part of a summer student project, and involves real tests on the system, and although not exhaustive, attempts to provide the necessary framework/foundation for further studies to be done. The need for this document arose from the lack of certain documentation, and as such, is intended to complement rather than replace other sources of information on the subject.

Testing Frameworks

A number of Testing Frameworks are available. This section lists each Testing solution, their potential advantages and disadvantages, and, where applicable, the setup procedure. This was included so that anyone wishing to use any of the solutions can do so, since some advantages appear in one and not the others. Namely, the available choices are to use the perf-test module which comes bundled with the source distribution of ActiveMQ, the ActiveMQ-JMeter combined solution and finally the open source JMeter framework, which was finally used for the tests themselves. The final section goes into more details about the characteristics of JMeter itself, being the chosen solution.

Perf-Test Plugin

The ActiveMQ source comes bundled with a performance testing framework, designed specifically for the software. This is however only available with the source distribution. It consists of a series of maven goals for implementing producers, consumers and test brokers (instances of ActiveMQ). The ActiveMQ-Performance-Users-Manual lists the usage procedure.

Advantages

• Designed specifically for ActiveMQ. As such, it has a wide range of configuration parameters, and offers the best range of test variables. Configuration of the parameters for each test are possible through command line arguments.
• Being entirely in command line, it should be less resource-intensive than a GUI-based alternative (such as the JMeter application).
• Fully supported by the ActiveMQ community

Disadvantages

• Perhaps the single biggest disadvantage is interpreting the results after the test. The solution generates .xml files with test data. However, there is currently no documentation to interpret the results, which are in an abbreviated notation: the link given on the above website for the plugin is outdated and even still, it gives no explanation of what the result acronyms mean. As such, unless further research is carried out to locate documentation of the same .xml file, the tests are as good as useless.
• Another potential disadvantage is the fact that all configuration is carried out through command line arguments, which can be tedious to configure for a large number of tests and test parameters.
• The maven build file (.pom) for the project is incomplete and needs some tweaking

ActiveMQ-JMeter Testing Framework

The next potential solution was to use the JMeter testing framework. This section details the ActiveMQ-configured version of JMeter: for the general JMeter client refer to the section below this one.

As explained in the documentation for JMeter the software is an open-source application under the Apache License, designed to load test web applications: in addition it can also test the functional behaviour of dynamic resources. The system boasts accurate control on timing and synchronization parameters, test flow and a very intuitive and descriptive logging facility. For this purpose, the ActiveMQ community provided a customized version of JMeter specifically for the ActiveMQ environment, with in-house versions of the JMS Producer Sampler and the JMS Consumer Sampler, which implement the JMS API. For a more in-depth discussion of the capabilities of JMeter, refer to the section below which gives a summary of its features.

Advantages

• Also tailor-made for ActiveMQ, and allows a respectable amount of tweaking of several parameters for different tests such as:
  • Message Domain type (Topic vs Queue)
  • Message Persistence
  • Number of Concurrent producers/clients to run
  • Duration of the test
• Intuitive GUI environment for configuration, allows testing to proceed at a faster rate.
• Good integration with ActiveMQ - was able to easily publish and consume messages...

Disadvantages

• The primary disadvantage lies in the implementation architecture. Rather than using the JMeter framework itself, the samplers run outside of JMeter, and thus, the same software’s listeners (required in order to log results) are not usable: in fact, the JMeter samplers only serve to multiple threads of JMS producers/subscribers which then run in the background. In fact, during testing, I was having trouble graphing results etc...
• The version is slightly outdated and with limited support.
• Some of the parameters are very restricted: for example the duration is restricted to integer multiples of minutes, else if an incorrect value is entered, it loops for infinity.
• Again, since the actual threads run outside of JMeter, the JMeter GUI is unable to interact with them: cannot stop them etc...

Apache JMeter

This section deals with the off-the shelf version of JMeter, which was eventually chosen for the tests. For the ActiveMQ-tailor made version, refer to the section above.

Advantages

• Official JMeter client, hence up to date and with support solutions.
• Intuitive and Versatile GUI which make JMeter a very good testing solution
• Allows usage of all the bells and whistles of JMeter (refer below)
• Only solution which allowed easy logging of test results
• Can also be run on multiple machines through command line interfaces, with one GUI version controlling the others

Disadvantages

• Being a general version, it is not designed specifically for ActiveMQ and hence, tends to be limited in configuration parameters.
• Relatively resource intensive to run.
• Requires an implementation of the JMS specification in the form of .jar files.
• It was found hard to interpret the graphical results by the Graph Listeners. That being said, the raw data can be easily interpreted and plotted using some other graphical package.

Installation Procedure

Installation is quite straightforward, as indicated below. Although the JMeter website gives the installation procedure, hereunder is listed the exact steps needed for an installation on Scientific Linux in use at CERN and specifically for ActiveMQ.

1. Download the latest Binary version of JMeter from the website:
2. Extract the .tar file to the requested directory: ensure that the directory path contains no spaces.
3. Locate the activemq-all-x.y.z.jar file from the ActiveMQ install directory: this is usually in the home directory of ActiveMQ not the lib folder
4. Copy this file to the lib directory of the install location of JMeter

The JMeter application can now be initiated by running the jmeter binary file from the installation directory: be patient, it takes a few seconds for the JMeter GUI prompt to load.

Fundamentals of Apache JMeter

This section lists the fundamental components of JMeter: it is basically a very brief overview/summary of the information available at the Jakarta Project Website.

The Test Plan

The test procedure centres around the Test Plan, of which there can only be one instance in any JMeter client. This component encapsulates all the other elements of the test, listed below. An empty Test Plan is provided when JMeter run, and Test Plans can be saved and reloaded later on. An important feature of the JMeter Test Plan is its ability to log all data communication in Functional Mode. This can be used to ensure correct functional operation of the application under test, at the expense of logging overhead.

Thread Groups

Thread Groups constitute the beginning point of each test: a Test plan will have one or more thread groups. Each thread group defines the sequence of operations to be executed sequentially: multiple thread groups execute in parallel. In addition, the user can set the Number of Threads per group, which consist of multiple copies of the same test running independently in parallel. The Ramp-Up period defines (in seconds) the total time to instantiate all threads, while the Loop field defines the number of iterations of each thread. The built-in scheduler comes in handy to synchronise tests across multiple machines, since one can specify the start times and duration of each test.

Controllers

Controllers fall under a Thread Group, and are of two types: Sampler Controllers and Logic Controllers:

1. Samplers : Samplers send requests to the system under test and receive back replies. They are used to implement the load-testing itself.
2. Logic Controllers : These are used in conjunction with Samplers and are used to control the flow of the test procedure, activating certain samplers under certain conditions, or repeating the operation a defined number of times within the thread execution.

Listeners

The various Test Listeners log the results of the test, by providing access to the information gathered during each sample. The different types store the same type of data but interpret the data in different manners, and can also be used to log the same results to file for later retrieval. For purpose of logging data, CSV formatting is obviously the more efficient option (rather than XML), especially if the field names are only stored once in the first row as a header. That being said, most listeners use a lot of memory, so for simple logging to file for offline analysis using the 'Simple Data Writer' (which just writes to file) or the Summary Report (which gives totals and averages only) are the best way to go. Note that in the Graphs Listeners, the data are the times taken by each sample to terminate.

Other Elements

1. Timers : Cause delays before each sampler in their scope: if multiple exist, they are all executed in the order they appear and before any samplers execute.Of particular interest is the Synchronisation Timer which blocks each thread under a predefined number have been blocked and then releases them simultaneously for intensive load testing on the server application.
2. Assertions : Assert correctness of the received data against user predefined values for the samplers in the same scope: they are executed after the samplers terminate
3. Configuration Elements : These work in conjunction with Samplers, and allow dynamic modification of requests sent
4. Variables : Like environment variables, these allow the user to specify dynamic values which can change per thread

Configuration of Testing Procedure

It is recommended to use the JMS Publisher and JMS Subscriber Samplers rather than the JMS Point to Point Sampler (which works only with queues). The latter choice allows the use of both Topic and Queue domains. The tables below detail the field parameters for each configuration point:

JMS Publisher Sampler

JMS Subscriber Sampler

Most of the fields are similar to those for the JMS Publisher: in addition however, the following fields exist:

In addition, another important configuration procedure regards the Listener's output to file for later retrieval if needed. The following table details the various parameters, giving also an advised configuration for speed while retaining all the necessary information needed for the purpose at hand: i.e. simply to load test the server. Obviously, if other features need to be tested, it is at the discretion of the test administrator to select the optimal configuration. Additionally, for performance reason, it is recommended to use either the Summary Report Listener (which gives some average readings of sample times etc...) or the Simple Data Writer (which simply writes to file only), since some of the other listeners can be very processor-intensive.

ASK LUCA ABT SAVE FIELD NAMES FIELD

N.B. Note that if you wish to retain all the results of subsequent tests, you must either change the name of the FileName from within the Listener dialog box or copy the generated text file to a different location between each test, otherwise it will be overwritten.

* Additional Considerations *

1. Ensure that the Subscriber Sampler is started with or before the Publisher Sampler to avoid losing samples.

Potential Test Variables for Performance Tuning

The next important consideration was to determine what parameters to test. For this reason, a detailed analysis of ActiveMQ was carried out, using various sources from the ActiveMQ website and configuration files, in order to find the characteristics which can impact the broker’s performance. The following is a list of the potential performance-impacting features, organised by the source from which they were retrieved: in the end, only a limited subset of these features were tested due to time constraints.

ActiveMQ Configuration Files Comments (activemq.xml and activemq-throughput.xml within the /conf/ directory)

• For better performance, it is advised to use the vmCursor with a small memory limit
• Producer flow control can hang the producer, rather than losing packets: it is up to the application designer to identify which is the better alternative
• System Usage flag controls the maximum amount of memory the broker will use before slowing down producers (throttling them)
• The KahaDB cursor system, which does not use Jetty, seems to be the best alternative, and can be tuned for performance.

Destination Policies for ActiveMQ (website)

• These are basically the same parameters that can be specified in the activemq.xml: however, using this configuration, different polices can be used for different topics, using the wild card configuration ability of activemq.
• If Producer Flow Control is disabled, messages get automatically off-lined to disk until buffer space is available (by subscribers consuming the messages): otherwise, the broker will throttle the producer
• the useCache flag informs the broker to cache persistent messages for faster retrieval
• the maxPageSize flag sets the maximum number of persistent messages to page from the store
• the memoryLimit flag acts as a child to memory limit (i.e. it must be equal to or less) and is defined on a per-topic basis rather than globally
• the tipping point for producer throttling is controlled by watermarks (filled space percentage of the buffer)
• it might be worthwhile to reduce advisory messages sent to increase performance, however, this will impact on the administration capability of the broker...

Slow Consumer Handling (website)

• This topic was not well understood, and may warrant some more looking into:
• Slow consumers pose a problem namely for non-durable topics: Therefore, in this case, one must specify the number of messages the broker will keep around in addition to the prefetch buffer: once this limit is reached, older messages are discarded. If the situation cannot tolerate lost messages, this discarding can be disabled by using a value of -1 for the PendingMessageLimitStrategy.
• Alternatively, one can use MBeans and JMX to monitor the messages being discarded.

Slow Consumers (website)

• Verifies that slow consumers do not pose a problem for durable topics, since the messages are persisted to disk (by default) and can simply be dropped from RAM when the buffer fills up.
• It is also not an issue for queues, where subscribers compete for the messages, and thus, slower consumers will simply receive less messages.
• On the other hand, for non-durable topics, this cannot be done, since there is no persistent storage.

Subscription Recovery Policy (website)

• Not very well understood. It seems that this is some kind of timing mechanism to retrieve lost packets during down-time of servers. However, it appears to be different from message persistence.
• With regards to performance reason, if not needed, it may make sense to remove it to increase performance.

Total Ordering (website)

• Adds synchronisation for consumers receiving from multiple producers.
• If not needed, should be removed, since it has quite a high performance cost.

Producer Flow Control (website)

• This is quite an important topic, performance-wise, and may warrant some more looking into since some topics are not well understood.
• Producer flow control is done at the application layer, in order to control each producer independently.
• Two options are available: the broker can either block the producer thread (by holding back acknowledgments until resources are available) or by causing it to throw a JMSException (in which case the producer must be programmed to act on such an exception)
• Using Async Sends (asynchronous sending), no notifications/acknowledgments are expected by the producer and hence no flow control is present. This yields better performance, but packets may be lost
• For non-persistent systems using the fileCursor, messages are automatically transferred to file, and therefore, the memory limits should never be exceeded.

Asynchronous Communication (website)

• Asynchronous Communication, which is the default way for sending unless needed by the JMS specification, has a huge impact on latency. Therefore, unless otherwise needed for reliability, it should be disabled.
• It is only in persistent subscriptions outside transactions that the broker defaults to synchronous transmission.
• It might be worthwile to look into transaction communication...

KahaDB Tuning (website)

• KahaDB should be the best compromise between efficiency and reliability when it comes to message persistence
• Certain Features which might be finetuned for performance include: will need considerable testing
  • indexWriteBatchSize = Number of indices written at one go
  • indexCacheSize = Number of index pages cached in memory
  • maxAsyncJobs = Maximum number of asynchronous messages that will be queued awaiting storage
  • concurrentStoreAndDispatchTopics = Allow the dispatching of Topics to happen concurrently with message storage

ActiveMQ in Action (Book: Manning Publications, Bruce Snyder, Dejan Bosanac, and Rob Davies)

• In some situations, it might make sense to embed the broker within the application generating the messages (i.e. within the same JVM), to avoid serialisation of the messages: however, this does not make sense in the ATLAS Alert environment due to the requirement of having an application-independent MOM broker which is on an other machine for reliability purposes.
• Persistence
  • Persistence is enabled by default by the broker, but the message producer must specify it.
  • If reliability is not an issue, or the server running the broker is itself quite reliable, then non-persistent topics make much more sense efficiency-wise
  • In addition, ActiveMQ provides additional reliability, even in a non-persistence scenario: this reliability avoids duplicate messages and can allow for failover to a different server when the main one fails, however, it does not guarantee delivery of packets, which is the realm of persistence mechanism.
  • In a Non-Persistent scenario, additional consideration should be given to:
    • Asynchronous sends
    • Less overhead due to avoiding writing to disk
• Transactions provide a certain reliability and performance compromise, because only the transaction boundaries are acknowledged, rather than each message. However, unlike in asynchronous sending, this acknowledgement provides added reliability.
• Tuning the openWire protocol: this is the default protocol used by the broker to communicate with consumers and producers.
  • tcpNoDelayEnabled = if set may improve performance, since messages are sent as soon as they are ready rather than buffering for a fixed size chunk transfer.
  • cacheEnabled = reduce message size at the cost of CPU load: makes sense if the network rather than the processing is the bottleneck
  • cacheSize = the bigger, the more performance should be achieved, but memory overhead increases with the number of transport connections
  • tightEncodingEnabled = compacts messages, but again is very CPU intensive.
• Tuning TCP:
  • socketBufferSize = controls the size of the TCP window: generally the bigger the better, but may need some tweaking for the optimal size
• Optimizing Producers:
  • In most cases uses asynchronous sending: need to test in our environment if this will affect reliability: maybe the underlying TCP protocol is reliable enough?
  • Flow Control:
    • The aim is to slow down producers when resources are low (e.g. memory buffer full...)
    • Must be specifically enabled if using asynchronous sending (by explicitly setting the producerWindowSize variable): otherwise flow control will default to blocking the Transport connector.
    • If disabled, the broker will store messages to disk (temporary queue). The memory limit determines the point at which messages start being offloaded to disk. This must be lower than the destination memory limit in order it to kick in before flow control (effectively disabling it).
    • Otherwise, if enabled, the broker has two configurations: either sendFailIfNoSpace, i.e. raise an exception on the producer side if no space is available, (which the producer must be programmed to handle) or additionally, the sendFailIfNoSpaceAfterTimeout, which waits for timeout before failing, to potentially allow space to become available: if it does, the exception is not raised.
    • It might be worthwhile to look into disabling message copy...
• Optimizing Consumers:
  • Consumers are usually the bottleneck of the system, due to the need for acknowledgements in a reliable environment.
  • Will TCP connection also be a bottleneck, since we are using the same connection?
  • One can try to increase the prefetch size (for topics), i.e. the memory limit set for each connection
  • It is also advised to use the Listener method, rather than the Receive method, (since the latter is a blocking function)
  • With regards to acknowledgments:
    • One can try to optimize acknowledgments, by using auto-acknowledgments which rolls up a group of acknowledgments: while this can potentially result in duplicate messages being delivered, ActiveMQ has mechanisms to reduce the probability of this happening
    • Setting the DUPS_OK_ACKNOWLEDGE has the same effect: i.e. it ignores duplicates as being an error: if the system can handle duplicates, then use this option for better throughput.
  • Disabling the alwaysSessionAsync flag allows messages to be passed directly from the transport to the message consumer, eliminating the need for a temporary queue, and speeding up the process.

Load Testing the broker

Load testing was carried out in the final week of my studentship, and as such was somewhat limited due to time constraints. Some trends did however emerge, which can point towards the direction in which further testing should proceed.

Test Architecture

Testing Framework

As stated above, the off-the shelf version of JMeter was chosen for load testing, after weighing the advantages/disadvantages with the other option. The JMS Subscriber and JMS Publisher samplers were used with Summary Report Listeners for efficiency. All instances of JMeter running on different machines (explained below) were individually synchronised to start simultaneously using JMeter's scheduler.

Test Variables

Topics and queues were individually tested:

• For topics, each test was repeated for 1, 5 and 10 producers, and 1 subscriber. A total of 63 tests were performed, spanning the three types of Message Cursors, three types of Message Store solutions, producer flow control and use of acknowledgements on the producer side.
• With queues each test was also repeated for 1, 5 and 10 receiving clients, (since this can potentially impact performance due to there being more consumers) while leaving the Message Store solution fixed (KahaDB). Tests were done with different cursor strategies against Producer Flow control and acknowledgements on or off, leading to a total of 72 tests.
• Each test was run for five minutes.

Test Machines

In this case, four Preseries machines (XPU-007 through XPU-010) were booked to be used during the last two weeks of the studentship. Each machine runs an 8-Core XEON processor at 2.5 GHz and 16 GB of RAM. In order to make the tests as uniform as possible and increase the number of tests within the limited timeframe, XPU-007 was used constantly as the server machine, and XPU-010 as the consumer machine. The producers were divided equally between machines 8 and 9. Moreover, during the test, the load on each computer was continuously monitored, using the top command (with the -C argument for less overhead - refresh rate at 1.5s) to ensure that the results were not being contaminated by overloading the machines running JMeter.

Test Results

Below are the results for all the test carried out, with all detail:

1. For Topics

2. For Queues

(Up to test 112, the 1MB memory limit was being used: consider when interpreting the results)

Appendix

General Comments (in order of appearance on notes)

• Not sure about the persistence feature: it is unclear whether JMeter is using persistent destinations or not, since it is unable to specify it on the producer side: however, the broker uses persistence property by default....probably?
• Note that even with no subscriber listeners, publishers still work...in actual fact, they work faster since there are no clients and all messages are simply discarded as soon as they are received...
• Sometimes messages seemed to be lost: i.e. the publisher samplers would register more messages than the subscriber samplers.
• During the initial stages of testing with the different frameworks, unifying the two JMeter implementations (the default one and the ActiveMQ version), by copying certain .jar files, but the process failed.
• The notes for the JMS point to point connection on the website incorrectly define some of the parameters required for the test, including the way for specifying the topics/queues to connect to. As such, follow the configuration for the JMS Publisher/Subscriber Listeners as appropriate.
• A connection to an active broker is required for the test to proceed, else it will simply fail: the error will appear in the log file (jmeter.log).
• Note that the JMeter log file appears in the directory from where the run command was called, not necessary where the jmeter executable lies. This means if you navigate to some other directory A and specify the full path for the executable, the log file is created in A.
• Rather than copying the activemq-all-x.y.z.jar file into the JMeter lib directory, you can make do with only the activemq.jar, geronimo-j2ee-manage.jar, geronimo-spec-j2ee.jar, slf4j-api.jar and slf4j-simple.jar.
• Using the ActiveMQ-JMeter version, it was noticed that when starting the test, it would start smoothly but then halt and continue after a short delay.
• Again, with the ActiveMQ version of JMeter, when running the loop twice, each sample is executed twice but one of them ends up being mostly zero's...

-- MichaelCamilleri - 11-Oct-2011

%RESPONSIBLE% Main.unknown
%REVIEW% Never reviewed