How AWS Outposts and the Solace Event Broker Achieve 23 Million
Messages per second with Ultra-Low Latency for Trading Systems
Co-authored with Jared Thompson, Tim Roser and Ahmed Elhosary with AWS.
In the high-stakes world of financial services, the difference between success and missed opportunity often comes down to the speed at which data moves across systems, and microseconds matter.In this post, we’re excited to share groundbreaking results from a collaboration between AWS and Solace that demonstrates how by leveraging AWS Outposts racks with accelerated networking Amazon Elastic Compute Cloud (Amazon EC2) instances (bmn-sf2e) & Solace Event Brokers, financial institutions can achieve unprecedented performance for their most demanding workloads.
The Importance of Latency and Performance in Capital Markets
In capital markets, latency and throughput are key metrics for trading systems that contributes to firm’s profitability. Whether it’s delivering market data to trading algorithms or routing orders to execution venues, the speed and predictability of data movement directly impact profitability and regulatory compliance. Firms in equities, fixed income, and derivatives trading all rely on lightning-fast dissemination of prices, quotes, and trades to maintain a competitive edge. Even microbursts of latency can result in missed opportunities, stale data, or increased exposure to market risk.
Modern trading infrastructures are built around market data distribution and order flow processing—both of which demand consistent, ultra-low-latency performance. Market data publishers must fan out millions of updates per second across trading systems, while order management systems must ensure every order, cancel, and acknowledgment moves across the network with microsecond precision.
Traditional data center deployments achieve this with purpose-built hardware and finely tuned networks, but as firms modernize, the challenge becomes: how do you retain that deterministic performance while embracing cloud agility and scale?
Solace Event Brokers on AWS Outposts Bare Metal: Bridging Low Latency and Cloud Agility
AWS Outposts racks with accelerated networking Amazon EC2 instances (bmn-sf2e) bring AWS services and infrastructure directly into customers’ data centers or colocation sites, minimizing round-trip latency to while maintaining a consistent operational model with the AWS Regions (Outposts is part of the AWS Cloud). This is where Solace Event Broker excels. Deployed on AWS Outposts racks with accelerated networking Amazon EC2 instances, Solace Event Brokers can deliver high throughput data distribution with the low and predictable latency demanded by trading workloads.
Solace Event Broker’s event mesh architecture enables real-time data distribution across trading floors, co-location facilities, and cloud-based analytics systems, all while maintaining guaranteed delivery and high throughput. With AWS Outposts, firms can deploy Solace brokers physically close to trading venues for deterministic latency, and easily extend event streams to Solace Brokers deployed in AWS Regions for machine learning, risk analysis, or trade surveillance—without compromising performance or control.
This hybrid model empowers capital markets firms to modernize their trading infrastructure—preserving the deterministic performance of on-prem environments while benefiting from AWS’s ease of provisioning and security and operational consistency with AWS Regions. By running Solace Event Brokers on AWS Outposts racks with accelerated networking Amazon EC2 instances. Trading firms achieve the best of both worlds: ultra-low-latency event distribution at the edge, and seamless integration with cloud-native analytics cloud-native analytics services to accelerate innovation.
Topic Routing in Solace
In capital markets, market data distribution involves broadcasting an immense volume of real-time updates — price ticks, quotes, trades, depth-of-book changes — to thousands of consuming systems simultaneously. Achieving this at scale demands more than raw network speed; it requires an intelligent messaging layer that can route information precisely where it’s needed, without unnecessary duplication or processing overhead. This is where the Solace Event Broker’s Topic Routing becomes a game-changer.
Solace uses a hierarchical topic structure to dynamically route messages based on their subject matter — for example:
MARKETDATA/NYSE/IBM/TRADES
MARKETDATA/NASDAQ/AAPL/QUOTES
MARKETDATA/FX/EURUSD/DEPTH
Producers publish data once to these topics, and Solace’s event broker efficiently delivers each message only to subscribers that have expressed interest, using topic filters such as MARKETDATA/NYSE/> delivers all NYSE quotes or MARKETDATA/*/IBM/> delivers all IBM market data. This eliminates redundant data flow, minimizes network utilization, and ensures that each consumer receives only the data it requires — all while maintaining low latency and high message fanout performance.
When deployed on AWS Outposts racks with accelerated networking Amazon EC2 instances, Solace Event Brokers can leverage dedicated network interfaces to route millions of topic-based messages per second locally within the trading environment. This setup delivers wire-speed performance and deterministic latency, enabling trading systems, order gateways, and analytics platforms to receive market data updates in real time without jitter or congestion.
Outposts also ensures that this low-latency infrastructure is fully managed and integrated with AWS services, easily extend event streams to AWS Regions. For example, the same market data published to MARKETDATA/NYSE/IBM/TRADES on-premises can be delivered locally or selectively forwarded to a Solace Broker in the parent AWS Region for consumption for use cases such as historical analytics, real-time dashboards, or AI-driven anomaly detection, while keeping the latency-critical trading workloads on-premises.
Test Environment
The results shown on this post were achieved by performing a set of tests ran under the following environment:
AWS Outpost
AWS Outpost racks with 13 bmn-sf2e instances:
- Operating System: RHEL 9.6
- 32 Core x2 CPUs Intel(R) Sapphire Rapids Lake 3.9 GHz turbo
- 1 TB RAM
- 4 x 4 TB NVMe
- 4 x AMD Solarflare X2522 NI, 8×25 Gbps (200 Gbps Total)
Solace Configuration
Solace Event Broker deployed as a Podman container:
- SolOS: 10.25.0.87
- Tier: 200K connections
- High-Availability Deployment
- Podman 5.4.0
Test Clients Configuration
For simulating traffic from client applications, we used Solace’s testing tool SDKPerf using the C API. The test cases were managed and run with the help of Ansible, making sure they were evenly distributed across multiple accelerated networking EC2 instances located on the same AWS Outpost as the instances hosting the Solace Brokers:
- 10 Amazon EC2 bmn-sf2.metal instances
- 10 publishers spread across 10 nodes
- 100 subscribers/consumer across 10 nodes
- 10 Solace topics = Fan-out of 10
- Each client/process was pinned to a different CPU
Environment Optimization
Although we achieved very good results with an ‘Out of the Box’ environment, we performed the following optimizations to achieve the impressive results presented in the following sections:
Podman Rootful Containers & Host Network.
Although, running a Solace software broker as a rootless container is fully supported, running as a Rootful container delivers the best performance.
Regarding networking, the containers were configured to use HOST network mode (over BRIDGED), which allows the container to use the host network stack instead of having to manage virtual interfaces and NATs.
Open Onload Kernel Bypass
One of the greatest advantages of accelerated networking Amazon EC2 instances is that they are available as bare metal instances, reducing latency impact inherent to virtualization layers. In our testing we made sure to take advantage of this by performing Kernel Bypassing using Solarflare’s Open Onload drivers.
Disable unnecessary services
Same NUMA for CPUs & NIC
To take optimization to the extreme, in our testing we made sure that the CPUs and NIC Interfaces being used by the Solace Software Broker were in the same NUMA node to avoid slow remote memory access across sockets.
Latency testing were performed using the network interface ens8f0
Test Methodology
The tests were run using direct messaging – designed for use with high-rate or very low-latency message messaging use cases that can tolerate occasional message loss. The most common use case in trading systems is market data distribution.
For direct messaging, payload sizes were used, and the following measurements were taken:
- Throughput; both ingress and egress
- Latency; Average, 99th percentile and standard deviation
To ensure test legitimacy, each test was run for at least 3 times during 20 minutes, a valid Direct Messaging test is when publishers and subscribers are capable of maintaining target ingress & egress message rates and no message discard is registered.
Performance Results
Solace’s direct messaging can provide high-rate throughput and very low-latency because it does not require acknowledgments between publishers and consumers, also messages are only maintained in memory and never persisted to disk.
Direct Messaging Throughput
When measuring throughput while using Solace’s Direct Messaging QoS on a Software Broker deployed on general purpose VMs, the limiting factor is usually the NIC bandwidth (10 or 25 Gbps are common VM capacities).
| Msg Size (Bytes) | Ingress (Msgs/Sec) | Egress (Msgs/Sec) | Ingress Bandwidth (Mbps) | Egress Bandwidth (Mbps) |
|---|---|---|---|---|
| 100 | 2,300,000 | 23,000,000 | 1,754.76 | 17,547.61 |
| 512 | 1,200,000 | 12,000,000 | 4,687.50 | 46,875.00 |
| 1,024 | 850,000 | 8,500,000 | 6,640.63 | 66,406.25 |
| 5,120 | 205,000 | 2,050,000 | 8,007.81 | 80,078.13 |
If we analyze the results obtained when testing Direct Messaging, we can see that when deploying a Solace broker on a super high performant bare metal EC2 instance like bmn-sf2.metal, we can push the broker its limits to achieve distribution of millions of messages per second.
Direct Messaging Latency
| Msg Size (Bytes) | Throughput Ingress/Egress (Msg/sec) | Average (µs) | 99th Percentile (µs) | StdDev (µs) |
|---|---|---|---|---|
| 100 | 20,000 | 72.40 | 112.00 | 32.10 |
| 512 | 20,000 | 72.60 | 112.00 | 30.90 |
| 1,024 | 20,000 | 73.60 | 112.00 | 31.40 |
| 5,120 | 20,000 | 88.30 | 138.00 | 37.70 |
Now, for direct messaging latency, we can note that for small payloads (1KB or less) the measured latency is mostly driven by the underlying network latency between the producer and the broker (35µS in average) + the latency between the broker and the subscriber (35µS in average) the latency overhead due to broker processing is minimal.
Conclusion
By combining AWS Outposts racks with accelerated networking Amazon EC2 instances with Solace Event Brokers, we’ve created a solution that delivers the ultra-low latency and high throughput required for mission-critical financial applications—all while maintaining the operational consistency and scalability benefits of AWS cloud infrastructure.
AWS Contributors
Explore other posts from category: Financial Services
Subscribe to Our Blog
Get the latest trends, solutions, and insights into the event-driven future every week.