Gartner’s Advice on How to Choose an Event Broker

Note: A new blog post featuring more recent Gartner research has been published, and should be read alongside this one.

On June 22, 2020 Gartner^® published yet another insightful report on the event-driven architecture (EDA) market, called “Choosing Event Brokers: The Foundation for Your Event Driven Architecture”, by analyst Gary Olliffe. You can read this report by clicking the link but you must have a Gartner for Technical Professionals subscription to gain access. For those who do not have access, I will summarize some of the key aspects of the report below.

Gartner’s Take on Event Brokers

Gartner provides very sage advice on choosing event brokers, because there are a lot of options on the market, from open source options to commercial offerings.

To me, the most important guidance of all is this:

Not all event brokers are created equal. Even if you are already an expert user of one or more event brokers, you should still read, absorb and reference this information. You cannot assume that seemingly similar products will behave in line with your experience, and it is important to know where the differences exist.”¹

To put a somewhat finer point on this, Gartner says:

Application technical professionals must assess event brokers’ functional and operational features and make key technical trade-offs to select the right tool.”

Because…

“Unlike REST or gRPC for request-response communications, the practices, standards and approaches to EDA are diverse with little consolidation beyond the fundamental pattern of publish-subscribe (pub-sub) communication. Your choice of event brokers is critical because its specific capabilities will dictate the architecture, design and operation of your event-driven applications.”

and lastly…

Event Brokers support publish-subscribe communications, but products have extremely diverse architectures and capabilities. These differences make selection challenging; misalignment of requirements and capabilities is painful and costly to resolve.”

From those quotations from the report, I think the point has been made that Gartner believes it is very important to understand the differences between different event brokers; now let’s see what Gartner says about specific types of brokers.

The Three Types of Event Brokers, Explained by Gartner

There are three basic types of event brokers:

• queue-oriented
• log-oriented
• subscription-oriented

Gartner’s highest-level event broker recommendations look like this:

• Use the simplest event broker configuration to meet your needs – an event broker cloud service will often avoid most (but not all) configuration complexity. Deploying, tuning and operating highly available, high performance, clustered event brokers is a nontrivial engineering and support effort;
• Choose a queue-oriented broker when you require multiprotocol client connectivity and flexible topic structure definition, and do not require message retention and replay;
• Choose a log-oriented broker when you require long term message retention and replay and/or very high throughput, and can tolerate a flat topic structure;
• Deploy multiple event brokers where your requirements span the capabilities of two or more classes of broker types. Do not assume that ‘one-size fits all’; and
• Implement schema management to govern your EDA by controlling the association between event types and topics using a broker-specific or independent schema registry.

Gartner’s Description of Queue-Oriented Brokers

In my opinion, Gartner does an adequate job describing the basic principles of queue-based brokers: “The pub-sub mechanism is typically based on creating queues for each consumer (or shared consumer group) and a routing mechanism to deliver published message to the appropriate queues.”

Source: Gartner “Choosing Event Brokers: The Foundation of Your Event-Driven Architecture, 22 June 2020, Gary Olliffe

Gartner notes that, “Queue-oriented brokers are a best fit if you need some combination of:

• Multiprotocol support for heterogeneous architectures
• Hierarchical topic structures and flexible subscription patterns
• Support for FIFO queue processing in addition to pub-sub
• Or the constraints of subscription and log-oriented brokers do not fit your needs.”

Gartner’s Description of Log-Oriented Brokers

Gartner describes log-oriented brokers as: “Based on the concept of an append-only logs of messages. Consumers do not remove messages from the log, and the broker will not remove them once they are processed. Instead, the log is retained and messages are purged as they age or as the log reaches a size limit. In some cases, they can be retained indefinitely.
The following shows how a simple nonpartitioned topic is consumed by subscribers in a log-oriented broker.”

Source: Gartner “Choosing Event Brokers: The Foundation of Your Event-Driven Architecture, 22 June 2020, Gary Olliffe

Gartner notes “Published messages are routed to one and only one of these partitions by a round-robin process (where ordering is not a concern) or a hashing algorithm using a producer-defined key (e.g., sourceID or customerID) to route all events for a specific key to a specific partition. Each partition is a “mini-log” and can have only one consumer process per subscription, ensuring that events for each key (or hash of the key) are processed in order. Note that topic-wide ordering is not supported across partitions.”

Source: Gartner “Choosing Event Brokers: The Foundation of Your Event-Driven Architecture, 22 June 2020, Gary Olliffe

My colleague Ken Barr explains in the blog post Why You Need to Look Beyond Kafka for Operational Use Cases, Part 1: The Need for Filtering and In-Order Delivery.

Gartner also notes, “A log-oriented broker is a good option when you know you will need to retain and replay events (e.g., for stream processing or to support transient subscribers) and when you know your consumers will not always be able to keep up with the volume of published events. A log-oriented broker is also the best option for “fan-in” event ingestion. However, you will also need to tolerate partitioned topics and partition-level ordering of events in order to scale your subscribers’ processing capacity.”

Gartner’s Description of Subscription-Oriented Brokers

Subscription-based brokers were born out of the need to support cloud-native function platform as a service and serverless architectures. The following image from Gartner shows their logical architecture:

Source: Gartner “Choosing Event Brokers: The Foundation of Your Event-Driven Architecture, 22 June 2020, Gary Olliffe

Gartner notes that:

A subscription-oriented broker is a good option when building cloud-native event-driven applications in a single cloud provider’s environment and you are willing to design your solution around the unordered, at-least-once delivery model. This scenario is most likely when you want to implement infrastructure automation or build serverless architectures that rely (at least partially) on platform-specific events emitted by your compute, data persistence or communications middleware services.”

Now You Know the Basics… But the “Devil is in the Details”

While this high-level guidance is useful, the devil is in the details. In my opinion, the bulk of this Gartner research exposes those details for application leaders and developers to make informed decisions about how to evaluate the capabilities they need, what they can tolerate, and which tools can support their requirements.

As Gartner aptly notes:

Not all event brokers are created equal. Even if you are already an expert user of one or more event brokers, you should still read, absorb and reference this information. You cannot assume that seemingly similar products will behave in line with your experience, and it is important to know where the differences exist.”

Source: Gartner “Choosing Event Brokers: The Foundation of Your Event-Driven Architecture, 22 June 2020, Gary Olliffe

Gartner notes event broker features and capabilities fit into the following categories:

• Client connectivity capabilities — These features relate to how your event produces and consumers will connect to the broker, and how they will process event messages. This includes protocols support, transaction support and features for software developers and integrators.
• Message delivery capabilities — These features relate to how messages are defined and handled by the event broker. This includes message structure, metadata and message persistence semantics, topic organization, topic partitioning, routing, quality of service, and subscription definition.
• Broker deployment options — These features relate to how the broker is deployed and configured to meet a variety of non-functional requirements including throughput, latency, resilience and recoverability.
• Management and operations capabilities — These features relate to the day-to-day management of the event broker environment and the event-driven applications you build around it. This includes management interfaces including APIs, command line interface (CLI) and UIs, as well as monitoring and securing the environment.”

In my opinion, under each of these categories are a myriad of individual features and requirements, each of which has implications on application design and resulting capabilities. It is important to understand what is important to you and your needs, what the individual brokers are capable of, and what compromises (if any) you are willing to make. As Gartner states:

The three classes of broker have overlapping capabilities, so your choice of event broker for a use case must take the set of capabilities into account rather than the general classification. Understanding these capabilities will make your compromises clear and prepare you for the architecture and design work needed to accommodate those compromises.”

One Final Thought – Event Schema Management

Without replicating the entire 42-page research note, I will simply highlight one final important section within this research: Event Schema Management. Gartner rightly notes:

The structure or schema of the message payload must be agreed and communicated in the same way as an API specification. In fact, it is useful to apply API design and delivery principles to EDA.^”

 Gartner refers to this as schema management and it includes the following capabilities:

• Schema Registry – This is a central or shared store of event or message schemas used whtin the system.
• Schema format support – Different payload formats (such as JSON, XML, Apache Avro and protocol buffers) have different schema definition formats. Your schema registry must support the formats your applications will publish and consume.
• Schema versioning – Event payloads can change as producers are modified and improved. You need to manage these version changes.
• Schema validation – Schema validation of individual messages can happen on the broker or within the client.

In fact, Gartner also speaks of “emerging capabilities” that some vendors are just starting to bring to market:

• Event discovery – Developers have become used to discovering REST APIs using self-service developer portals. When an organization wants to democratize and distribute access to its system or business events, developers expect the same experience.
• Choreography visualization – When processes are driven by choreography (meaning a “chain of events”) rather than by a central orchestration process, it can be hard to visualize the flow or state of the process. It can also change based on which consumers are connected and active at a point in time.  Consistent use of distributed tracing tools to inject and propogate traceable IDs and collect data can provide this visualization.
• Function execution – Event brokers are often used to trigger stateless functions, and in a cloud environment, they are commonly used with serverless function PaaS. A natural extension of this is to have a function runtime integrated with the event broker.

I’ve tried to distill some of the essential advice from Gartner’s 42-page paper on choosing event brokers, but let me leave you with their closing guidance:

When adopting or expanding your use of EDA patterns, you must evaluate your selection of event broker middleware against your known use cases:

• Use the simplest event broker configuration to meet your needs
• Use key features to make the fundamental choice of broker type
• Deploy multiple event brokers when required
• Implement schema management.”

If you are a Gartner client, with access to their Technical Professionals research, I encourage you to read this report in its entirety at https://www.gartner.com/document/3986571.

And if you found this article useful, I encourage you to read a new post that features more recent Gartner research: “Gartner’s Latest Advice on How to Choose an Event Broker”.

Solace was mentioned as one of the “queue oriented brokers” in the report.

The interesting thing about Solace PubSub+ is while it is fundamentally a queue-oriented broker, it too has an optional message replay capability, thus retaining messages at the broker and only removing once a pre-determined size limit is reached. You can check out the overview here or a demo on YouTube.

The first two emerging capabilities (Event Discovery and choreography visualization) are capabilities that Solace is bringing to market with PubSub+ Event Portal.

_{Source: ¹ Gartner “Choosing Event Brokers: The Foundation of Your Event-Driven Architecture, 22 June 2020, Gary Olliffe}

_{GARTNER disclaimers
GARTNER is the registered trademark and service mark of Gartner Inc., and/or its affiliates in the U.S. and internationally and has been used herein with permission. All rights reserved. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.}

Solace Named an Event Broker LeaderIDC MarketScape positions Solace in the Leaders category for worldwide event broker software.Read the Excerpt