On June 22, 2020 Gartner published yet another insightful report on the event-driven architecture (EDA) market, called “Choosing Event Brokers: The Foundation for Your Event Driven Architecture”, by analyst Gary Olliffe. You can read this report by clicking the link but you must have a Gartner for Technical Professionals subscription to gain access. For those who do not have access, I will summarize some of the key aspects of the report below.

Gartner’s Take on Event Brokers

Gartner provides very sage advice on choosing event brokers, because there are a lot of options on the market, from open source options to commercial offerings from vendors like Solace and others.

To me, the most important guidance of all is this:

Not all event brokers are created equal. Even if you are already an expert user of one or more event brokers, you should still read, absorb and reference this information. You cannot assume that seemingly similar products will behave in line with your experience, and it is important to know where the differences exist.”1

To put a somewhat finer point on this, Gartner says:

Application technical professionals must assess event brokers’ functional and operational features and make key technical trade-offs to select the right tool.”1

Because…

“Unlike REST or gRPC for request-response communications, the practices, standards and approaches to EDA are diverse with little consolidation beyond the fundamental pattern of publish-subscribe (pub-sub) communication. Your choice of event brokers is critical because its specific capabilities will dictate the architecture, design and operation of your event-driven applications.”1

and lastly…

Event Brokers support publish-subscribe communications, but products have extremely diverse architectures and capabilities. These differences make selection challenging; misalignment of requirements and capabilities is painful and costly to resolve.”1

From those quotations from the report, I think the point has been made that Gartner believes it is very important to understand the differences between different event brokers; now let’s see what Gartner says about specific types of brokers.

The Three Types of Event Brokers, Explained by Gartner

There are three basic types of event brokers:

  • queue-oriented (like Solace PubSub+, RabbitMQ, Azure Service Bus, etc.),
  • log-oriented (like Apache Kafka or Amazon Kinesis), and
  • subscription-oriented (such as Amazon EventBridge and Azure Event Grid).

Gartner’s highest-level event broker recommendations look like this:

  • Use the simplest event broker configuration to meet your needs – an event broker cloud service (such as Solace PubSub+ Cloud) will often avoid most configuration complexity, as deploying, tuning and operating highly available, high performance, clustered event brokers is nontrivial.
  • Choose a queue-oriented broker if you require multiprotocol client connectivity and flexible topic structure definition;
  • Choose a log-oriented broker when you require long term message retention, and can tolerate a flat topic structure;
  • Deploy multiple event brokers where your requirements span the capabilities of two or more broker types. Do not assume that ‘one-size fits all’; and
  • Implement schema management to govern your EDA by controlling the association between event types and topics using a broker-specific or independent schema registry (such as PubSub+ Event Portal).

Gartner describes at a high-level the architectural differences of the three different approaches and their main benefits and drawbacks.

Gartner’s Description of Queue-Oriented Brokers (i.e. Solace PubSub+)

Gartner does an adequate job describing the basic principles of queue-based brokers like RabbitMQ, ActiveMQ, Solace PubSub+ and others in that the pub-sub mechanism is typically based on creating queues for each consumer (or shared consumer group) and a routing mechanism to deliver published message to the appropriate queues.

Gartner on queue-oriented brokers

Source: Gartner “Choosing Event Brokers: The Foundation of Your Event-Driven Architecture, 22 June 2020, Gary Olliffe

Gartner notes that queue-oriented brokers are a best fit if you need some combination of:

  • Multiprotocol support for heterogeneous architectures
  • Hierarchical topic structures and flexible subscription patterns
  • Support for FIFO queue processing in addition to pub-sub
  • Or the constraints of subscription and log-oriented brokers do not fit your needs (i.e. multi-cloud support, topic-level ordering)

Gartner’s Description of Log-Oriented Brokers (i.e. Apache Kafka and commercial derivatives)

Gartner describes log-oriented brokers as based on the concept of an append-only logs of messages. Neither consumers nor the broker will remove messages when processed. Instead the log is retained and messages are purged as they age or as the log reaches a pre-determined size limit. This allows for what is called “message replay”.

The interesting thing about Solace PubSub+ is while it is fundamentally a queue-oriented broker, it too has an optional message replay capability, thus retaining messages at the broker and only removing once a pre-determined size limit is reached. You can check out the overview here or a demo on YouTube.

The following shows how a simple nonpartitioned topic is consumed by subscribers in a log-oriented broker.

Gartner on log-oriented brokers

Source: Gartner “Choosing Event Brokers: The Foundation of Your Event-Driven Architecture, 22 June 2020, Gary Olliffe

Gartner notes that since only one consumer process can read messages from the log at a time, the way you scale this approach is to partition (or shard) the topic into smaller “mini-logs” as per the following:

Gartner on event brokers, kafka topics and partitions

Source: Gartner “Choosing Event Brokers: The Foundation of Your Event-Driven Architecture, 22 June 2020, Gary Olliffe

It is also noted that topic-wide ordering is not supported across partitions, as my colleague Ken Barr explains in the blog post Why You Need to Look Beyond Kafka for Operational Use Cases, Part 1: The Need for Filtering and In-Order Delivery.

Gartner notes that log-oriented brokers are a good option when you know you will need to retain and replay events (e.g. stream analytics). Log-oriented is also a good option for “fan-in” event ingestion. However, Gartner notes you will need to tolerate partitioned topics and partition-level ordering of events in order to scale your subscriber base.

Gartner’s Description of Subscription-Oriented Brokers (i.e. Amazon EventBridge, Microsoft Azure Event Grid)

Subscription-based brokers were born out of the need to support cloud-native function platform as a service and serverless architectures. The following image shows their logical architecture:

architecture of subscription based brokers

Source: Gartner “Choosing Event Brokers: The Foundation of Your Event-Driven Architecture, 22 June 2020, Gary Olliffe

Gartner notes that:

a subscription-oriented broker is a good option when building cloud-native event-driven applications in a single cloud provider’s environment and you are willing to design your solution around the unordered, at-least-once delivery model. This scenario is most likely when you want to implement infrastructure automation or build serverless architectures that rely (at least partially) on platform-specific events emitted by your compute, data persistence or communications middleware services.”1

Now You Know the Basics… But the “Devil is in the Details”

While this high-level guidance is useful, the devil is in the details. The bulk of this Gartner research exposes those details for application leaders and developers to make informed decisions about how to evaluate the capabilities they need, what they can tolerate, and which tools can support their requirements.

As Gartner aptly notes:

Not all event brokers are created equal. Even if you are already an expert user of one or more event brokers, you should still read, absorb and reference this information. You cannot assume that seemingly similar products will behave in line with your experience, and it is important to know where the differences exist.”1

 

Gartner event broker middleware

Source: Gartner “Choosing Event Brokers: The Foundation of Your Event-Driven Architecture, 22 June 2020, Gary Olliffe

Event broker features and capabilities fit into the following categories:

  • Client connectivity capabilities — These features relate to how your event produces and consumers will connect to the broker, and how they will process event messages. This includes protocols support, transaction support and features for software developers and integrators.
  • Message delivery capabilities — These features relate to how messages are defined and handled by the event broker. This includes message structure, metadata and message persistence semantics, topic organization, topic partitioning, routing, quality of service, and subscription definition.
  • Broker deployment options — These features relate to how the broker is deployed and configured to meet a variety of non-functional requirements including throughput, latency, resilience and recoverability.
  • Management and operations capabilities — These features relate to the day-to-day management of the event broker environment and the event-driven applications you build around it. This includes management interfaces including APIs, command line interface (CLI) and UIs, as well as monitoring and securing the environment.

Under each of these categories are a myriad of individual features and requirements, each of which has implications on application design and resulting capabilities. It is important to understand what is important to you and your needs, what the individual brokers are capable of, and what compromises (if any) you are willing to make. As Gartner states:

The three classes of broker have overlapping capabilities, so your choice of event broker for a use case must take the set of capabilities into account rather than the general classification. Understanding these capabilities will make your compromises clear and prepare you for the architecture and design work needed to accommodate those compromises.”1

One Final Thought – Event Schema Management

Without replicating the entire 42-page research note, I will simply highlight one final important section within this research: Event Schema Management. Gartner rightly notes:

The structure or schema of the message payload must be agreed and communicated in the same way as an API specification. In fact, it is useful to apply API design and delivery principles to EDA.1

 Gartner refers to this as schema management and it includes the following capabilities:

  • Schema Registry – A central or shared of event schemas.
  • Schema format support – i.e. JSON, XML, Avro
  • Schema versioning – for application compatibility
  • Schema validation – to ensure consistent policy

In fact, Gartner also speaks of “emerging capabilities” that some vendors are just starting to bring to market:

  • Event discovery – REST API like developer experiences to democratize access to events
  • Choreography visualization – The ability to visualize your event or process flow
  • Function execution – embedded serverless function PaaS

The first two emerging capabilities (Event Discovery and choreography visualization) are capabilities that Solace is bringing to market with PubSub+ Event Portal.

I’ve tried to distill some of the essential advice from Gartner’s 42-page paper on choosing event brokers, but let me leave you with their closing guidance:

When adopting or expanding your use of EDA patterns, you must evaluate your selection of event broker middleware against your known use cases:

  • Use the simplest event broker configuration to meet your needs
  • Use key features to make the fundamental choice of broker type
  • Deploy multiple event brokers when required
  • Implement schema management.”1

If you are a Gartner client, with access to their Technical Professionals research, I encourage you to read this report in its entirety at https://www.gartner.com/document/3986571.

1 Gartner “Choosing Event Brokers: The Foundation of Your Event-Driven Architecture, 22 June 2020, Gary Olliffe

Roger Sabourin

Roger Sabourin is Solace's analyst relations leader.