On June 22, 2020 Gartner® published yet another insightful report on the event-driven architecture (EDA) market, called “Choosing Event Brokers: The Foundation for Your Event Driven Architecture”, by analyst Gary Olliffe. You can read this report by clicking the link but you must have a Gartner for Technical Professionals subscription to gain access. For those who do not have access, I will summarize some of the key aspects of the report below.
Gartner provides very sage advice on choosing event brokers, because there are a lot of options on the market, from open source options to commercial offerings.
To me, the most important guidance of all is this:
Not all event brokers are created equal. Even if you are already an expert user of one or more event brokers, you should still read, absorb and reference this information. You cannot assume that seemingly similar products will behave in line with your experience, and it is important to know where the differences exist.”1
To put a somewhat finer point on this, Gartner says:
Application technical professionals must assess event brokers’ functional and operational features and make key technical trade-offs to select the right tool.”
“Unlike REST or gRPC for request-response communications, the practices, standards and approaches to EDA are diverse with little consolidation beyond the fundamental pattern of publish-subscribe (pub-sub) communication. Your choice of event brokers is critical because its specific capabilities will dictate the architecture, design and operation of your event-driven applications.”
Event Brokers support publish-subscribe communications, but products have extremely diverse architectures and capabilities. These differences make selection challenging; misalignment of requirements and capabilities is painful and costly to resolve.”
From those quotations from the report, I think the point has been made that Gartner believes it is very important to understand the differences between different event brokers; now let’s see what Gartner says about specific types of brokers.
There are three basic types of event brokers:
Gartner’s highest-level event broker recommendations look like this:
In my opinion, Gartner does an adequate job describing the basic principles of queue-based brokers: “The pub-sub mechanism is typically based on creating queues for each consumer (or shared consumer group) and a routing mechanism to deliver published message to the appropriate queues.”
Gartner notes that, “Queue-oriented brokers are a best fit if you need some combination of:
Gartner describes log-oriented brokers as: “Based on the concept of an append-only logs of messages. Consumers do not remove messages from the log, and the broker will not remove them once they are processed. Instead, the log is retained and messages are purged as they age or as the log reaches a size limit. In some cases, they can be retained indefinitely.
The following shows how a simple nonpartitioned topic is consumed by subscribers in a log-oriented broker.”
Gartner notes “Published messages are routed to one and only one of these partitions by a round-robin process (where ordering is not a concern) or a hashing algorithm using a producer-defined key (e.g., sourceID or customerID) to route all events for a specific key to a specific partition. Each partition is a “mini-log” and can have only one consumer process per subscription, ensuring that events for each key (or hash of the key) are processed in order. Note that topic-wide ordering is not supported across partitions.”
My colleague Ken Barr explains in the blog post Why You Need to Look Beyond Kafka for Operational Use Cases, Part 1: The Need for Filtering and In-Order Delivery.
Gartner also notes, “A log-oriented broker is a good option when you know you will need to retain and replay events (e.g., for stream processing or to support transient subscribers) and when you know your consumers will not always be able to keep up with the volume of published events. A log-oriented broker is also the best option for “fan-in” event ingestion. However, you will also need to tolerate partitioned topics and partition-level ordering of events in order to scale your subscribers’ processing capacity.”
Subscription-based brokers were born out of the need to support cloud-native function platform as a service and serverless architectures. The following image from Gartner shows their logical architecture:
Gartner notes that:
A subscription-oriented broker is a good option when building cloud-native event-driven applications in a single cloud provider’s environment and you are willing to design your solution around the unordered, at-least-once delivery model. This scenario is most likely when you want to implement infrastructure automation or build serverless architectures that rely (at least partially) on platform-specific events emitted by your compute, data persistence or communications middleware services.”
While this high-level guidance is useful, the devil is in the details. In my opinion, the bulk of this Gartner research exposes those details for application leaders and developers to make informed decisions about how to evaluate the capabilities they need, what they can tolerate, and which tools can support their requirements.
As Gartner aptly notes:
Not all event brokers are created equal. Even if you are already an expert user of one or more event brokers, you should still read, absorb and reference this information. You cannot assume that seemingly similar products will behave in line with your experience, and it is important to know where the differences exist.”
Gartner notes event broker features and capabilities fit into the following categories:
- Client connectivity capabilities — These features relate to how your event produces and consumers will connect to the broker, and how they will process event messages. This includes protocols support, transaction support and features for software developers and integrators.
- Message delivery capabilities — These features relate to how messages are defined and handled by the event broker. This includes message structure, metadata and message persistence semantics, topic organization, topic partitioning, routing, quality of service, and subscription definition.
- Broker deployment options — These features relate to how the broker is deployed and configured to meet a variety of non-functional requirements including throughput, latency, resilience and recoverability.
- Management and operations capabilities — These features relate to the day-to-day management of the event broker environment and the event-driven applications you build around it. This includes management interfaces including APIs, command line interface (CLI) and UIs, as well as monitoring and securing the environment.”
In my opinion, under each of these categories are a myriad of individual features and requirements, each of which has implications on application design and resulting capabilities. It is important to understand what is important to you and your needs, what the individual brokers are capable of, and what compromises (if any) you are willing to make. As Gartner states:
The three classes of broker have overlapping capabilities, so your choice of event broker for a use case must take the set of capabilities into account rather than the general classification. Understanding these capabilities will make your compromises clear and prepare you for the architecture and design work needed to accommodate those compromises.”
Without replicating the entire 42-page research note, I will simply highlight one final important section within this research: Event Schema Management. Gartner rightly notes:
The structure or schema of the message payload must be agreed and communicated in the same way as an API specification. In fact, it is useful to apply API design and delivery principles to EDA.”
Gartner refers to this as schema management and it includes the following capabilities:
In fact, Gartner also speaks of “emerging capabilities” that some vendors are just starting to bring to market:
I’ve tried to distill some of the essential advice from Gartner’s 42-page paper on choosing event brokers, but let me leave you with their closing guidance:
When adopting or expanding your use of EDA patterns, you must evaluate your selection of event broker middleware against your known use cases:
- Use the simplest event broker configuration to meet your needs
- Use key features to make the fundamental choice of broker type
- Deploy multiple event brokers when required
- Implement schema management.”
If you are a Gartner client, with access to their Technical Professionals research, I encourage you to read this report in its entirety at https://www.gartner.com/document/3986571.
Solace was mentioned as one of the “queue oriented brokers” in the report.
The interesting thing about Solace PubSub+ is while it is fundamentally a queue-oriented broker, it too has an optional message replay capability, thus retaining messages at the broker and only removing once a pre-determined size limit is reached. You can check out the overview here or a demo on YouTube.
The first two emerging capabilities (Event Discovery and choreography visualization) are capabilities that Solace is bringing to market with PubSub+ Event Portal.
GARTNER is the registered trademark and service mark of Gartner Inc., and/or its affiliates in the U.S. and internationally and has been used herein with permission. All rights reserved. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.