Breaking Barriers: 8.75 Million Messages a Second…in the Cloud!

In today’s data-driven world, businesses need reliable, high-performance messaging infrastructure to handle event-driven applications and real-time data streams. This is particularly true in capital markets, where microseconds can mean millions in profit or loss. Many organizations in capital markets have found transitions to the cloud to be more difficult than other industries, in part due to regulatory requirements, but primarily due to the latency that is introduced. We know that many of our own customers have experimented with such moves but found the trade-offs to be too large.

This post is intended to help capital markets organizations make informed decisions about their event streaming architecture as they migrate mission-critical workloads to the cloud by giving insight into the throughput, latency, and overall performance characteristics of Solace’s event broker when deployed on Oracle Cloud Infrastructure.

Solace PubSub+ Event Broker has established itself as a leading solution for enterprise messaging needs, combining robust performance with comprehensive features, and are excited to share the results of our performance testing of Solace PubSub+ Event Broker on Oracle Cloud Infrastructure (OCI).

These results show that with Solace and OCI, customers can overcome the primary performance issues and begin migrating some workflows onto cloud infrastructure. With Oracle’s pricing model they can also migrate these workflows without incurring the egress costs that have plagued high volume scenarios. For Solace customers it means that they can finally leverage hybrid infrastructure for some workloads without incurring the penalty they have seen in the past.

OCI and Solace in Capital Markets

Oracle Cloud Infrastructure

Oracle Cloud Infrastructure (OCI) provides comprehensive services designed to handle the most demanding enterprise workloads, delivering exceptional performance, resiliency, security, and cost effectiveness.

OCI stands out for capital markets workloads due to several key benefits:

• Powerful computing capabilities, including industry leading high-performance computing (HPC) and AI infrastructure, enables rapid processing of complex financial models, real-time trading, and market data analysis.
• A flat non-oversubscribed network guarantees consistent bandwidth and predictable low latency, essential for market data distribution and algorithmic trading
• Enterprise-grade security meets the stringent security and compliance requirements of financial organizations
• Extensive global presence with more regions than any other cloud provider allows firms to strategically deploy workloads for optimal performance and regulatory compliance. OCI Points of Presence (PoPs) are available in all key capital market locations.
• Best price for performance ensures low and predictable costs that are the same in all regions globally, with no data egress fees within region and over dedicated connections – crucial for moving large volumes of data.

Financial institutions like Goldman Sachs are leveraging OCI for HPC and significantly improving the price/performance of their Monte Carlo risk simulations.

OCI’s differentiated core infrastructure, network, and cloud economics also make it ideal for market data distribution, where every millisecond counts.

Solace

Solace has been a trusted technology provider in capital markets since its inception, with deployments across global banks, exchanges, asset managers, and broker-dealers. Nine of the world’s top ten tier-1 banks trust Solace to power their mission-critical use cases globally. Solace’s broker excels in capital markets use cases for several reasons:

• High Performance: Supports high throughput and low latency distribution of real-time data
• Event Mesh: Allows brokers to be connected to form an event mesh which enables global data distribution and facilitates both hybrid-cloud and multi-cloud deployments, enabling distribution of data from on-prem to any cloud(s) and vice versa.
• Multiple Quality-of-Service: Supports both ultra-fast in-memory messaging for latency-sensitive market data and guaranteed delivery with disk persistence for trade order processing with zero message loss
• Multi-protocol support: Enables integration between legacy systems and modern applications via multiple open protocols (AMQP, JMS, MQTT, REST, WebSocket)
• High fanout capabilities: Efficiently distributes market data to thousands of consumers simultaneously
• Topic-based filtering: Allows consumers to subscribe to precisely the market data they need without having to discard messages they are not interested in

In capital markets, Solace powers critical use cases including:

1. Market data distribution: Efficiently delivering real-time quotes, orders, and trades to trading applications, risk systems, and analytics platforms
2. Trade order workflow: Ensuring reliable order routing from front-office to execution venues
3. Post-trade processing: Guaranteeing delivery of trade confirmations, clearing, and settlement messages
4. Pricing and risk calculations: Distributing real-time pricing updates and risk metrics across the organization
5. Trader UIs: Distributing real-time data such as market data to frontend web UIs

Solace’s versatility in supporting diverse capital markets use cases, coupled with its Event Mesh technology enabling seamless hybrid and multi-cloud distribution, delivers exceptional value to financial institutions.

Test Environment Setup

Here is what the test environment setup looked like for both throughput testing and latency testing:

Oracle Cloud Infrastructure Configuration

For our benchmarking, we deployed Solace PubSub+ Event Broker (200K tier) across two different OCI instance types to evaluate performance characteristics for different deployment scenarios:

* 1 OCPU = 1 physical core (non-oversubscribed). SMT/Hyperthreading was disabled for all servers.

** A maximum of 45GB of memory was used during all the tests. OCI flexible compute shapes enable you to customize compute instances by choosing any combination of cores and memory, allowing you to create a VM with less memory to lower costs.

Solace Configuration

We deployed Solace PubSub+ Event Broker tier with the following configuration:

• SolOS: 10.11.1.147
• Tier: 200K connections
• High-Availability Deployment
• TLS enabled

Client Configuration

We utilized a testing framework developed with Solace’s sdkperf application to generate simulated traffic, enabling us to measure throughput and latency. The clients (publishers and subscribers) were deployed on a kubernetes cluster (OKE).

• 80 kubernetes worker nodes (kubernetes version v1.32.1)
• Kubernetes node: VM.Optimized3.Flex with 1 OCPU, 12 GB memory and 4 Gbps network bandwidth
• 40 publishers spread across 40 nodes
• 200 subscribers spread across 40 nodes
• Each pod had 1 vCPU and 8GB memory
• Fan-out of 5

OCI Environment Optimization

To optimize the OCI environment for Solace PubSub+ we considered compute instance configuration, network configuration, and operating system configuration.

Compute Instance

The performance of PubSub+ Event Broker is constrained by the performance of individual cores and storage. For this reason, we opted for the following:

• Select a compute shape featuring a CPU with a high base clock speed. VM.Optimized3.Flex and BM.Optimized3.36 have an Intel Xeon processor with a base frequency of 3.0 GHz and max turbo frequency of 3.6 GHz.
• Disable SMT to extract most performance out of each core and avoid thread contention.
• Leverage local high performance NMVe storage, included with BM.Optimized 3.36, which speeds up guaranteed messaging.

Network

To ensure maximum network performance each compute instance was provisioned with hardware assisted networking – Single Root Input/Output Virtualization (SR_IOV). This is ideal for low latency workloads and uses the VFIO driver framework.

The OCI standard MTU is 9000, which was used in all tests.

OCI does not throttle traffic between availability domains and no specific configuration had to be done.

All containers used Docker network_mode: host. This instructs the container to share the host’s network interface directly, bypassing Docker network isolation. This significantly improves container network performance.

Operating System

The default Oracle Linux 8 configuration turned out to be optimal for the Solace PubSub+ application. No specific changes were needed.

The Solace PubSub+ Event Broker does benefit from running on dedicated cores, separate from cores that run everything else. To isolate cores, we changed the kernel boot parameters. For example, the following command was used for the VMs:

grubby --update-kernel=ALL --args "nohz_full=2-15 isolcpus=managed_irq,2-15 irqaffinity=0-1"

In docker-compose.yaml we specified which cores were allocated to the Solace Event Broker:

services:
   solace1:
      image: solace/solace-pubsub-standard:10.11.1.147
      container_name: solace
      hostname: solace
      network_mode: host
      cpuset: 2-15

Note that the bare metal servers have two sockets. It is important to allocate cores on a single socket /NUMA node for best performance.

Test Methodology

Our testing focused on scenarios that simulate real-world capital markets messaging patterns:

• In-Memory Messaging: Simulating market data distribution with:
  • Multiple publishers representing market data feeds
  • Multiple downstream subscribers
  • Fanout of 5
  • 100 bytes payload, typical of market data payload
• Guaranteed Messaging: Simulating trade order flow with:
  • Persistent messaging with queues
  • Fanout of 5
  • 1 KB payload, standard for testing persistent messaging

For each quality of service, we measured:

• Maximum sustainable throughput (messages per second)
• Message latency (min, average, 99th percentile)

Test validity required three consecutive 20-minute runs with consistent performance. For direct messaging, tests maintained target ingress and egress throughput with zero message discards. Guaranteed messaging tests succeeded only when operating without significant message queue accumulation.

Performance Results

Our benchmark testing revealed impressive performance metrics for Solace PubSub+ (200K tier) across different OCI instance types. The results demonstrate that OCI provides an excellent foundation for high-throughput, low-latency messaging workloads typical in capital markets applications.

In-Memory (Direct) Messaging

In-memory messaging provides the speed and throughput essential for market data distribution. Also known as fire-and-forget or at-most-once delivery, it achieves minimal latency by bypassing disk writes and acknowledgment processes. This approach is ideal for market data distribution where receiving the latest price updates with microsecond latency outweighs the importance of capturing every single message.

Throughput

The table below shows the maximum sustained throughput achieved on a highly-available broker deployment across different OCI instance types with 100-byte messages, typical of market data feeds:

It’s worth noting that both standard VM and bare metal configurations delivered identical throughput for direct messaging. This makes sense, as OCI does not oversubscribe compute resources, meaning one core on a VM provides the same performance as one core on bare metal. Given that direct messaging is CPU-bound, the result is comparable performance across both environments.

Latency

For latency testing, we setup a publisher process and a subscriber process on a separate host (VM or bare metal) that would be responsible for publishing to and subscribing messages from Solace broker deployed on either VM or bare metal, respectively. The latency being measured is the time duration from publisher send() call to the subscriber receiveCallback(). In other words, it’s end-to-end latency from publisher to subscriber.

The table below shows results of latency testing achieved across different OCI instance types with 100-byte messages, 100K msg/s background load, and 2,000 msg/s by latency probe for latency testing.

Note that the brokers and clients were not deployed in a cluster placement group. Leveraging a cluster placement group ensures close proximity of compute instances and significantly reduces latency.

As shown in the results above, the Solace broker demonstrated exceptional latency performance when deployed in OCI. Key findings include:

• Exceptional Performance Under Load: Maintained outstanding latency while processing 100K messages per second of background load across both VM and bare metal environments
• Superior VM Performance: Achieved remarkably low 99th percentile latency of 241 μs with minimal variability when both clients and broker operated on VMs
• Bare Metal Advantage: Delivered dramatic performance gains with a 58% reduction in average latency and nearly 37% decrease in 99th percentile latency compared to VM configuration.

Guaranteed Messaging

Guaranteed messaging provides reliable delivery essential for critical transactions such as trade orders. By writing messages to disk on both primary and backup brokers and managing acknowledgments, it ensures zero message loss. While this higher quality of service guarantees transaction integrity, it introduces additional latency compared to in-memory messaging.

Throughput

The following table shows performance metrics for guaranteed messaging for both 1KB and 1MB payloads across OCI instance types:

Both Standard VM and bare metal configurations achieved record-breaking throughput for 1KB and 1MB payload sizes. Bare metal’s significantly superior performance can be attributed primarily to its included local storage, which comes bundled with the instance at no additional cost. In contrast, VM relies on network-attached storage, creating a performance differential.

Powering Capital Markets with Solace on Oracle Cloud Infrastructure

Our benchmarking results demonstrate that Oracle Cloud Infrastructure provides an excellent foundation for Solace PubSub+ Event Broker deployments in capital markets. The performance metrics across different OCI instance types showcase the platform’s ability to deliver the throughput, latency, and reliability characteristics required by demanding financial services workloads.

The combination of OCI’s high-performance compute options and low latency high bandwidth network, and Solace’s dual-mode messaging capabilities creates a powerful platform that is uniquely suited to the diverse messaging needs of capital markets firms. Whether distributing high-volume market data feeds with low latency or ensuring guaranteed delivery of trade orders and post-trade messages, Solace on OCI delivers enterprise-grade performance with cloud flexibility.

For capital markets firms looking to migrate mission-critical applications to the cloud, this pairing offers a compelling solution that doesn’t compromise on the performance requirements that trading applications demand. OCI flexible infrastructure allows firms to specify the exact amount of cores and memory needed for a VM and auto tune block storage from 3k to 300k IOPS. OCI offers bare metal compute as a native service for maximum compute performance at the same cost as virtual machines. Both virtual machines and bare metal instances are available with and without local storage. All this allows firms to right-size their infrastructure based on specific workload characteristics while maintaining the security, compliance, and reliability standards required in financial services.

As capital markets continue to evolve with increasing data volumes, more complex trading algorithms, and growing regulatory requirements, the scalability and performance of the underlying messaging infrastructure become ever more critical. Our testing demonstrates that Solace PubSub+ on OCI is well-positioned to meet these challenges, providing a robust foundation for market data distribution, trade order processing, and other latency-sensitive capital markets applications.

Conclusion

By following the optimization strategies and deployment best practices outlined in this post, financial institutions can maximize the performance of their Solace deployment on OCI while maintaining the operational efficiency and cost control that cloud environments enable.

Note: The performance numbers presented in this post were collected under controlled test conditions. Your actual results may vary based on specific workloads, configurations, and environmental factors. We recommend working with Solace and OCI teams to conduct your own performance testing to validate these results in your specific context.