Enabling large payloads over Solace event brokers
Solace comes from a legacy of delivering ultra-high performance event brokers. We have made the analogy that our brokers are like a fleet of sports cars delivering packages across a city (aka the Event Mesh) of highly targeted addresses as fast as possible and with 100% reliability. This compares to, for instance, Kafka that is more like a freight train bringing all the packages to the city perimeter at once in a large log file.
And like a sports car, in order to win races, we helped our customers by making engineering choices based on optimizing for speed (defined simply as latency and throughput). This comes from our early legacy in financial markets, and it includes optimizing things like routing protocols, CPU utilization, cache usage, memory management, threads and more; all designed to help customers to execute milliseconds faster than the next guy and to enable fluid and massive scaling in production.
In the real world, for many customers data isn’t always perfectly micro-sized. Whether it’s a massive legacy XML batch, a heavy retail inventory update being sent to thousands of stores, or a master data file meant to cross the IT/OT divide for manufacturing, data can have weight. And because event driven architecture has become the standard for asynchronous decoupling of publishers and subscribers across industries, we have seen tons of growth in data streaming across enterprise types.
This has changed the needs for some of our core event broker products where performance, while important, isn’t the only thing that matters. Let’s explore two recent changes and see how they help customers with larger payloads.
Change #1 – Liberating small brokers by enabling larger message sizes
Smaller event brokers were intended for deployment in environments with lower compute capacity (and also generally in non-HA environments) and were de-featured and designed for the smaller footprint use-case.
Regional branch offices, distribution centers, and individual retail storefronts do not always want the cost to deploy heavyweight brokers to receive essential business data. But they do need larger payloads. By removing this size barrier in the new edge broker tier we have democratized edge computing, ensuring that lightweight, cost-effective brokers can seamlessly ingest the larger-value payloads required to run operations. In essence, a lower cost solution for your “lower value” implementations.
Change #2 – Core Ingress Optimization (Smart Client Compression)
The second breakthrough targets ingestion by introducing Smart Client-Side Compression Validation to our Java, JCSMP, and JMS APIs. Previously, the API enforced a strict 30 MB limit on raw, uncompressed payloads, rejecting large files even if native compression would have easily shrunk them down to size. This naive restriction optimized broker performance but forced compression to be added on the client-side – which was not a desirable behavior for some customers.
With this update, the API becomes intelligent, evaluating the highly optimized, compressed size of a message before applying ingress rules. Now, a legacy 40 MB XML file can be compressed by 99% down to a tiny 400 KB packet and seamlessly ingested.
The Use Case for End-to-End Frictionless Flow
When these two advancements are woven together, they create a powerful, end-to-end synergy that is very user-efficient for your teams:
- At the Core: A central ERP system generates large multi-megabyte catalog update. The smart client-side API effortlessly compresses the file, bypassing the old ingress gate without breaking a sweat.
- Across the Mesh: Once inside, that highly compressed, lightweight message is efficiently fanned out across the network to thousands of edge brokers located in physical stores worldwide. This has the added benefit of reducing egress costs.
- At the Edge: Because the edge brokers are no longer bound by size caps, they ingest and process the update flawlessly, delivering large data payloads at a fraction of the traditional infrastructure cost and network bandwidth.
Conclusion: Small changes are part of a new era
Ultimately, these two seemingly small changes represent a philosophical shift in how we support the modern enterprise developer and architect. We listened to the community’s friction points regarding payload constraints, and we eliminated them at both the ingestion point and the edge destination.
Solace remains the fastest event broker on the market, but using the sports car analogy that we started with, we’ve officially given you the option to add a large trunk and turn that sports car into an SUV. Will it suit all applications? No. It’s why you wouldn’t enable it if raw speed is all that mattered and your applications also meet this requirement. However, just like the SUV that has become so popular, this event broker compromise happens to be perfectly suited to many applications; and now you have choice.