Solace PubSub+ Throughput

Methodology

These performance results look at message throughput for point-to-point and fan-out scenarios. The results measure PubSub+ in terms of messages per second. These message exchange patterns are explained in more detail in our core concepts page but this section briefly describes these two patterns as they apply to the performance scenarios.

Point-to-Point

A point-to-point flow is a single publishing client sending messages to the PubSub+ message broker which are received by a single subscribing client.

Fan-Out

Fan-out is the typical publish/subscribe scenario where a message is published once to the PubSub+ message broker and forwarded to multiple clients who have expressed interest in receiving messages of that kind through topic subscriptions. So the message is then fanned out to the clients by the PubSub+ message broker, and each client receives a copy of the message.

SDKPerf Test Tool

For all scenarios, the Solace test tool SDKPerf was used as the traffic source and sink. Because we take performance very seriously, over many years we’ve built SDKPerf into a very capable tool that can be used to test many different messaging APIs. You can find out more about the tool on our SDKPerf overview and you can download the tool and try it yourself as well. For these tests, sdkperf_c was used.

For each scenario, SDKPerf measures the following:

• The received message rate.
• Message loss using message ordering information that publisher instances of SDKPerf embed in the messages they send.
• Message payload integrity by confirming the received message payloads are unmodified by the messaging system.

This ensures that the messaging system is running fast and correctly delivering all messages to the correct consumers without corruption.

Message Delivery Modes

The scenarios in this article cover two types of messaging delivery modes:

• Non-Persistent – Which is “at most once” messaging. PubSub+ message brokers provide two options for “at most once” messaging. Both direct messaging and non-persistent messaging achieve this level of service. The results in this article use direct messaging which is the higher performance option.
• Persistent – Which is “at least once” messaging. To achieve full persistence, producers must receive acknowledgements from the messaging system when the PubSub+ message brokers have fully persisted the message to non-volatile storage. And messages for consumers must not be removed from the message router until an acknowledgement is received. Fully persisted, means the messages is flushed to disk. Often this option is called failsafe messages in other message brokers. Simply caching the message in the filesystem is not equivalent and can lead to message loss in failure scenarios. Therefore in the article when results are listed for Persistent messages this is for fully failsafe messaging with no risk of message loss.

Performance Scenarios

In the performance scenarios below, the goal is to measure the maximum performance of the PubSub+ message broker in each scenario. To achieve this, we use groups of point-to-point or fan-out clients all running in parallel. Each group of clients is monitored to make sure it is running at a rate which it can handle to avoid an individual client becoming the bottleneck and skewing the performance results artificially.

For point-to-point flows, this means you end up with the scenario illustrated in the following figure.

In this figure, client groups are depicted using colors. The figure shows an orange, blue and grey group. Each group has a single publishing client sending to a single receiving client. The test uses enough groups of clients such that the broker becomes the bottleneck, not the clients. When you monitor the PubSub+ broker and observe the aggregate messages per second across all clients then you determine the maximum message rate a broker can sustain for the point-to-point scenario.

Similarly for fan-out scenarios, you will end up with the scenario illustrated in the following figure.

Again in this figure, client groups are depicted using colors. The figure shows an orange, blue and grey group. And again, the number of client groups is scaled as required until the Solace message router maximum fan-out rate is observed. The maximum fan-out rate is the total output message rate for the Solace message routers. Within each group, the number of subscribing clients is dictated by the fan-out factor being measured. So a fan-out factor of 10 will require 10 subscribing clients per group. To figure out how many messages per second are being sent by clients to the Solace message router, simply take the output message rate and divide by the fan-out factor. This is the input rate.

At Solace we believe in full transparency for all aspects of performance results, so comprehensive details on the exact hardware, setup and methodology are described below in the section Appendix A: Test Setup.

Point-to-Point Non-Persistent (Direct) Messaging

The following table shows the aggregate output message rate (messages/sec) for Solace direct messages (non-persistent) as the message payload size is varied. In this test groups of clients are sending messages in a point-to-point fashion. Twenty parallel sets of clients are used to generate the required traffic.

Table 2 – Point-to-Point Non-Persistent Message Throughput (Output Msgs/Sec)

* The results measured reached the bandwidth limit of the NAB on the 3560 Appliance, or the (v)NIC on the software broker.

From these results you can see that as expected the performance is fastest for small message sizes, as these are easier for the Solace message routers to process efficiently.

It is also interesting to graph these results and add in calculated bandwidth in Gbps. From this you can see that bandwidth utilization in ESXi is always increasing as the message size increases, until it reaches the 10Gbps bandwidth limit of the NIC at the 10K byte message payload size. In AWS, the broker quickly hits the 2Gbps network bandwidth limit of the m4.4xlarge EC2 instance. Larger EC2 instances will perform better. For example, an m4.10xlarge instance, with its 10Gbps of network bandwidth, has a performance profile very similar to the ESXi results.

For the PubSub+ appliance, the results for the bandwidth are similar. The hardware I/O card, called the Network Acceleration Blade (NAB), has a messaging capacity of around 80 Gbps. So as the message size increases, the NAB is able to saturate the network bandwidth and reaches the maximum bandwidth at around the 10K byte message payload size. From there the output message rate is governed more by the bandwidth capabilities of the NAB than other factors.

Point-to-Point Persistent Messaging

The following table shows aggregate output message rate (messages/sec) for fully persistent messages as the message payload size is varied. In this test, groups of clients are sending messages in a point-to-point fashion. Twenty parallel sets of clients are used to generate the required traffic.

Table 4 – Point-to-Point Persistent Message Throughput (Output Msgs/Sec)

As with the non-persistent scenario, from these results you can see that the performance is fastest for small message sizes as these are easier to process efficiently. Each of these messages must be saved to non-volatile storage prior to acknowledging the publisher so the overhead is much higher compared to non-persistent messages, and software performance becomes a function not only of the compute power and network bandwidth available, but also the IOPS and storage bandwidth capacity of the platform on which the software broker is running.

It is also interesting to graph these results and add in calculated bandwidth in Gbps. From this you can see that bandwidth utilization is always increasing as the message size increases.

For the PubSub+ appliance, the results for the bandwidth are a little different. The hardware guaranteed messaging card, called the Assured Delivery Blade or ADB, has a messaging capacity of around 9.5 Gbps. So as the message size increases, the ADB becomes the bottleneck and the system reaches the maximum bandwidth at approximately the 4K Byte message payload size. From there the output message rate is governed more by the bandwidth capabilities of the ADB card than other factors.

Appendix A: Test Setup

Performance metrics are only useful if you understand how they were measured and what test equipment was used. The following two figures outline the lab setup when running the tests. Instances of the SDKPerf tool were run on test host servers which were running Linux. Depending on the test requirements, different numbers of servers were used such that the clients were never the bottleneck. For the appliance scenarios, a pair of 3560 appliances we set up in the typical fault tolerant configuration with attached external storage. Up to 12 test hosts were required to generate the appropriate test load.

For the software results, a VMware ESXi server was used to host the PubSub+ VM image. For these scenarios, less tests hosts were required for the traffic generation due to the difference in the performance capabilities.

In both cases, a Mellanox SX1016 10GigE Ethernet switch was used to provide the LAN connectivity.

The following tables give full details of each of the hardware used in testing.

Solace PubSub+ Software (4 core, 8 GB, ESXi)

Solace PubSub+ Software (Amazon EC2 Instance)

Solace PubSub+ Appliances

Required Test Equipment

Up to 12 performance test hosts were used to generate and sink the required message rate for the various scenarios. They all had characteristics that matched the following specifications:

Connecting all the equipment together was a Mellanox 10GigE Ethernet Switch.