The Importance of Message Settlement for EDA

There is no doubt that event-driven architecture (EDA) is going mainstream. Every industry – whether it be consumer goods, retailers, banking, aviation, or capital markets – has come to a realization that EDA is no longer a niche architecture but an absolute requirement in order to remain competitive and relevant in the industry. For example, in this article about the retail industry, my colleague Ush Shukla says “In the coming phygital age, it is the event-driven business that will capture the lion’s share of the market.“

Now there are many EDA platforms out there, each with their own strengths and benefits. An analytics use case for detecting fraud will require a very different event streaming platform than a transactional use case that is processing payments. In this post, I will focus on an important aspect of an event streaming platform for transactional use cases: single message settlement (or acknowledgement). Before we dive into that, let’s start with some basics.

Event Driven Architecture 101

In an event-driven system, you have three main participants:

• Event Producers: These are a set of microservices that ‘push’ events to the event broker
• Event Consumers: These are a set of microservices that ‘subscribe’ to events from the event broker
• Event Broker: The event broker matches relevant events from the publisher to interested subscribers and pushes it to them

An event is an action or change in state, such as a sensor reading from an IoT device, a retail order event originating from an e-commerce system, or a transaction that originates from a trading engine. The event broker would typically be agnostic to the type of the event… it just has the responsibility of routing information about it to the consumer.

Settling a Message

Once a subscriber receives a message, it typically takes some action and notifies the event broker that it wants to receive the next message.

For example, a credit card payment microservice will consume the payment and perhaps write it to a system of record, and then notify the event broker that it wants to receive the next message through an acknowledgement (or “ack”). Once the event broker receives this notification, it will push more events to the event broker.

Two Patterns for Message Settlement

By now, you should see the importance of message settlement to an event driven architecture. Now there are two prevailing patterns that event brokers use to settle events: individual message settlement and batch settlement.

Here’s how these patterns work and what type of systems work best with each.

Individual Message Settlement

In this pattern, a message is individually acknowledged, and the event broker keeps track of what messages haven’t been acknowledged. In the case of a failure, the event broker will only deliver messages to the consumer that haven’t been acknowledged. In addition, some event brokers have the functionality to mark a message as redelivered in the case it was sent to a consumer but not processed. These features make it simpler for microservices/applications to deal with failures.

Transactional systems that require individual message settlement such as credit card systems, order management systems, trading systems all functionally work better with a single message acknowledgement pattern.

Batched Message Settlement

In this pattern, the consumer will periodically send an acknowledgement back to the event broker to notify it that it is done processing a set amount of messages.

In this situation, it is harder for an application/microservice to reason about failures. When the event broker redelivers the message to the consumer, there is no indication that it has been redelivered and so the consumer will have to rely on an external state store to determine what to do with an event once it is received. This significantly increases architectural complexity and therefore is not fit for purpose for transactional use cases.

However, when you are doing analytics/processing on a stream of data – your analytics/processing framework will typically act on records in batches and will store an index/offset of the last consumed message to deal with failures and also have the ability to deduplicate messages. In this case, the burden of tracking consumer state is moved away from the broker to the consumer which is typically par for the course for analytics use cases.

Conclusion

Message settlement or acknowledgement patterns are an extremely critical  consideration for an event driven architecture. As with any technology decision, it’s important to start with your business requirements and work towards a supporting technology platform. For example, are your events transactional in nature (such as retail orders, trades, payments etc.) where every message needs to be individually processed, a single message settlement process will be the ideal architecture. Whereas if you are dealing with an analytics use case where an extra step of post-processing on your event stream is part of your pipeline, a batched acknowledgement architecture can work better.

White Paper: Enabling Modern Digital Retail Banking with Event-Driven ArchitectureLearn how retail banks can benefit from event-driven architecture (EDA) and what you need to know about selecting a platform to support your EDA adoption.Download Paper