Event Mesh as an Architectural Pattern

In 1977, Christopher Alexander changed the way we design buildings, communities and cities with his book A Pattern Language which showed how one can combine “Lego block like components” in different ways to create new structures. This way of thinking made it’s way to the world of software when the authors of the now classic Design Patterns: Elements of Reusable Object-Oriented Software documented coding patterns that helped developers create cleaner and more reusable code. Others applied the idea to other software domains over the years with books like Enterprise Integration Patterns, Service Oriented Architecture Design Patterns, and Microservices Patterns. A higher-order coarser-grained set of patterns emerged alongside them to act as a guideline for assembling the domain-specific patterns, e.g. layered or multi-tier architecture, pipes and filters, model-view-controller (MVC), enterprise service bus, etc.

So architectural patterns are “blueprints” that help one select the most appropriate model or models for a use case. To extend the Lego block analogy, architectural patterns are essentially macro-blocks that can also be assembled to create higher order solutions. In this article I will introduce a rather new one, the event mesh, and how it complements existing patterns.

Event Mesh as a Unifying Architectural Pattern

As the name implies, the event mesh pattern has to do with the handling of asynchronous events. Functionally speaking there are two types of capabilities that certain exiting patterns reflect:

• Event routing
• Event production/publication – consumption/subscription

Due to the variety of experiences and opinions on the topic of architectural patterns, different authors often attribute different names to these functional capabilities. For instance, event routing is often attributed to the broker pattern, and in some cases, it is used more specifically in the context of the message broker pattern. Conversely, the ability to publish and subscribe to events is either captured in the “publish subscribe” pattern (commonly called pub-sub) or the enterprise service bus (ESB) pattern.

Additionally, there are other “message exchange patterns” that complement pub-sub (peer-to-peer queuing, request reply, one-way etc.), that are key aspects of event distribution that are sometimes attributed to the message broker pattern and/or to the ESB pattern. To make matters even more complicated, the event routing capability is sometimes also attributed to complex event processing (CEP), albeit in a broader sense, and pertaining more to content-based event routing.

It’s important to note that pub-sub and routing are two separate capabilities that are embodied by two separate product sets or implementations. In the routing context, one must contend with writing routing rules and conditions, and thus incur development costs. In the pub-sub context, the products/implementations supporting the pattern are “protocol bound” (JMS, MQTT, AMQP, Kafka, etc.) and cannot transcend those boundaries. In both cases there typically are additional operational costs due to the deployment and management of additional components, and geographical hybrid (cloud and on-prem) scalability can be a challenge

Given all these factors, how does the event mesh pattern fit in, and what problems does it solve?

Essentially, the event mesh pattern is an “unifying” pattern, supporting all the capabilities described above at a functional level. Conversely, the implementation of the “physical” event mesh should address all the operational, and scaling challenges faced by other implementations.

In terms of where the event mesh pattern fits and how/where it should be used, the answer is: throughout the event-driven architecture journey. Traditionally many enterprises would begin their event-driven architecture journeys with digital transformations localized to a line of business or a small-scoped pilot and would use some type of message broker to begin that journey. These are typically technology testbeds and do not have many architectural considerations. Arguably though, taking a design first approach from the very beginning, and building the event-driven architecture from a functional, capabilities and re-use perspective can yield great rewards early on. Thus, even for small scale pilots, focusing on where events need to go and how they need to be consumed from a business rather than a technology perspective, could in fact generate a reusable pattern of event distribution, that could shorten future design and development efforts.

As the case for event-driven architecture expands within the enterprise, a natural next step for the event mesh pattern is for it to be used in hybrid environments when scaling from on-prem to the cloud. All the key feature of the event mesh pattern should not be impacted by the physical extension to a new environment, and that is a very important distinction to make: the implementation of the event mesh and its ability to scale from the small and local to the broad and distributed, is what empower the event mesh pattern to be so universally applicable. In short, the event mesh pattern and how it is being used from a design perspective, presumes that those capabilities (discussed in the next section) are applicable to any size infrastructure, thus allowing the design pattern to grow with the infrastructure implicitly.

Event Mesh Pattern Capabilities

As previously indicated, the event mesh pattern is a broad pattern that subsumes the capabilities of other patterns and adds a few new characteristics of its own. One can think of the event mesh pattern as akin to a highway system, as defined by Christopher Alexander. It’s designed to move large volumes of data/event traffic fast, and it connects/bridges many other architectural patterns, much like the highway connect cities, town, districts, neighbourhoods etc.

In a way, the event mesh pattern can be seen as “connecting tissue” or “architectural glue” as it essentially enables other patterns to interoperate and work together. (more on this in the next section…)

The event mesh pattern embodies certain functional capabilities that enable it to act as that connecting tissue. It is important to differentiate between the event mesh pattern and the event mesh physical implementation (layer). The event mesh pattern embodies certain characteristics that allow it to be woven into conceptual distributed architectures. The event mesh implementation will of course support the “abstract” characteristics of the event mesh pattern however it will have many more of its own that have to do with non-functional, performance, scaling and security aspects.  The simplified UML diagram below illustrates this basic “hierarchy”, with the event mesh (physical implementation) inheriting/implementing the capabilities/characteristics of the event mesh pattern, and in turn being composed of multiple “nodes” or interconnected event brokers.

Below I introduce explore some of the key event mesh pattern capabilities; more information on the extended capabilities supported by the event mesh physical dimension can be found here.

• Dynamic event routing speaks to the ability of the event mesh pattern to distribute events to various consumers based on specific criteria. Whereas for instance a Broker pattern needs to have routing rules defined to fulfill that function, the event mesh pattern has this capability “baked in” i.e., the event mesh pattern does not require routing rules, but routing is done as a function of the way consumers express interest in specific Events.
• Support all event exchange pattern (EEP) – speaks to the ability of the event mesh pattern to support the capabilities of the event bus (pub-sub) pattern as well as those of the message broker pattern (one-way, multicasting, anycasting, request-reply streaming etc.).
• Protocol bridging – speaks to the ability of the event mesh pattern to essentially be protocol agnostic in that it that it should treat events in the same manner irrespective of what protocol they are traveling over (i.e. JMS, Kafka, Solace, MQTT etc.)

Below is a potential representation of what dynamic routing capabilities could look like for the event mesh pattern. Events would have “annotations/attributes” akin to hierarchical URL Paths, and consumers can subscribe to parts of the annotation, and thus select the type of information they are interested in. Thus, you “match” published events against “subscriptions of interest”; and through this “matching” process, routing is accomplished.

Because routing is fundamentally an attribute of the event mesh pattern, there is no need for additional “routing” components to be added to the architecture, and there is no need for routing rules.

The following diagram illustrates the protocol agnostic character of the event mesh pattern. The event mesh pattern leverages abstract channels for the propagation of events. In turn, the underlying event mesh infrastructure would provide protocol binding and translation mechanisms, so that an event initiated on say Kafka can be transported over Solace or JMS or any other industry standard protocol.

Earlier I said the event mesh implementation is constituted of multiple event brokers or nodes. The diagram bellow illustrates how the connected infrastructure of a physical event mesh could look like:

The nodes or node clusters are interconnected in a “mesh” like fashion. The nodes can be connected in a way that maximizes event flow to the interested parties. The diagram also speaks to how the nodes in a mesh can be geographically distributed to maximize coverage/distribution and provide localized access.

In essence, the event mesh pattern can be seen as an “intelligent” event distribution framework that is intended to maximize and streamline event flow.

The Bigger Picture: Event Mesh as a Foundational Pattern

Although event mesh has merit as an architectural pattern on its own, it offers much greater value when “assembled” or paired with other architectural patterns.

The best way to understand that is via a more concrete example. Let’s look at order processing, for instance.  When a purchase order is made online, there are typically multiple stages that the order must go through:

1. Order entry (Web or mobile app)
2. Order processing
   1. Check inventory
   2. Send ordered product to packaging/assembly if existing, and if not put order on the supply-chain roster
   3. Send to logistics
3. Logistics
   1. Track and trace order en-route
4. Customer management
   1. Ensure customer receives order and update order status

Although the example above is rather simplistic, what is important to understand is that all of these stages can be fulfilled by various architectural patterns, but what connects them all, is the event mesh.

The diagram below illustrates how the patterns fit together like a jigsaw puzzle:

As it can be seen the event mesh is at the center of the order processing infrastructure.

Here’s how the various stages of order processing are addressed by using various architectural patterns:

• Order entry can be handled by the MVC/layered architecture pattern with the order being received via a RESTful interface from a Web or mobile app and being persisted in some type of data store.
• The data store and the broader order state management aspect, as well as the logistics track-and-trace capability can in fact be managed by the blackboard pattern, where the various steps in the process are updated, and changes are broadcast to all the interested parties.
• Order processing can be accommodated by the pipes-and-filters pattern, as there is a sequence of steps involved in fulfilling the order, and they will occur over variable time.
• Lastly the event mesh pattern literally acts as the “glue” between the other 3 patterns:
  • The layered architecture pattern will need to emit and route events to the blackboard infrastructure as well as initiate Order Processing
  • The blackboard pattern will leverage the event mesh pattern to get updates to the Order state and communicate status outbound.
  • The pipes in the pipes-and-filter pattern are essentially routing channels supported by the event mesh pattern.

As you can see, the event mesh pattern is the quintessential “connecting/glue” pattern that empowers many other patterns. With the event mesh pattern one can create reliable and composable architectures, plugging in any other architectural pattern that one deems relevant for the solution requirements at hand.

Microservices Patterns and the Event Mesh Pattern

As you have seen, the event mesh pattern is a foundational pattern facilitating the composition of higher order application architectures by acting as the connecting glue between other architectural patterns.

However, the event mesh pattern is as useful at the macro level as it is at the micro level. For instance, there are a multitude of “finer granularity” microservices design patterns that are totally compatible with the event mesh pattern.

Below is a simple rendition of some of these compatible patterns:

Essentially all microservice design patterns that depend on some type of asynchronous interaction can leverage the event mesh pattern.

Let’s explore how some of these “dependent” microservices patterns function:

• Saga is usually used to implement distributed transactions, that can span multiple services. The intent of the saga pattern is to ensure data consistency across all the services involved in that transaction. Sagas are usually based on the choreography of services and events are being exchanged between these services to trigger the execution of the next step in the transaction. The event distribution and routing across services falls squarely in the realm of responsibility of the event mesh pattern.
• CQRS & Event Sourcing is a well-known pattern design to optimize data access by dividing the querying of data from the writing of data into separate services. In turn, Event Sourcing is designed to track the state of an entity throughout its life by emitting state change events that are stored in some type of query-able event store. Both patterns are complementary (event sourcing leveraging CQRS to write the events, and CQRS being used to query the event store where the events were written to) and are in fact best implemented by coordinating event exchanges, hence their compatibility with the event mesh pattern.
• Chain of Responsibility (Chaining) simply reflects the dependencies between services g. Service A needs to call Service B which in turn needs to call Service C and so on. Although this pattern can be implemented via synchronous calls, it is better serviced by asynchronous ones, especially if there is no need to maintain state across the services. The event mesh pattern can be used to marshal events between the dependent services.
• Aggregator aggregates data from multiple services and then sends or publishes the response to the interested party. This pattern is very useful when for instance aggregating response from multiple queries. As the request, the responses being aggregated, and the aggregate response itself being send to the interested party are essentially events, the event mesh pattern is a good supporting pattern for the Aggregator.
• Distributed Tracing: is similar to event sourcing but at a more granular system type level. The event mesh pattern can be used to route all tracing events to a centralized storage location. In the same vein, Log aggregation can also benefit from the event mesh pattern as a way to forward service logs to a central storage location.

Event Mesh Use Cases and Benefits

An event mesh pattern can support a wide variety of use cases; one can in fact use the event mesh pattern in virtually any industry. The examples below illustrate the type of impact and value the event mesh pattern can have.

Microservices Patterns Integration

Most modern architectures are based on domain-driven design principles using microservices as building blocks. The event mesh pattern can accommodate all event driven patterns and can act as the connecting fabric between microservice-based patterns and applications, as well as between microservices based applications and legacy technologies.

Customer Experience Management support

The event mesh pattern can act as the conduit that can route notifications and alerts to customers on their device of choice, at the right time and in the right context. The ability to facilitate real time delivery of events leads to better decision making and the rapid identification of opportunities or threats.

Internet of Things (IoT)

Due to its protocol agnosticism, and dynamic routing capabilities the event mesh pattern can support many IoT use cases. It can be positioned as a “connecting fabric” between edge devices and Cloud Based Services or between Cloud Based Services and on-prem back-end systems. For instance, many global logistics companies leverage the event mesh pattern to allow them to track and trace shipments.

Complex Event Processing

A big part of event-driven architecture is event processing. The event mesh pattern helps evolve and mature event-driven architecture. Due to the dynamic routing capabilities of the event mesh pattern, complex event processing (CEP) engines can focus more on content-based routing and event enrichment, and event stream processing (ESP) engines can more easily look at mixed data streams and detect patterns in more diverse data sets.

Benefits

Lower Operating Expenses

Due to the inherent capabilities of the event mesh pattern and the physical event mesh underpinning it, one needs to use fewer products in order to deliver certain solutions. For example, the event mesh pattern obviates the need for separate pub-sub and broker patterns and the respective messaging and routing products supporting them, as it encapsulates both capabilities. This leads to a reduction in technology sprawl, lowers capacity requirements, needs fewer upgrade cycles and fewer personnel requirements.

Higher Developer Productivity

Due to the fact that the event mesh pattern is agnostic to messaging protocols, and to environment types, developers can focus on implementing business logic using the best technologies available. This frees developers to innovate without having to develop complex data distribution networks, implement complex routing logic and removes the limitations imposed by development environments, messaging platform, or types of cloud. Additionally, using the event mesh pattern as part of a broader architectural solution (as in the order processing example above), usually generates an inordinate amount of reusable assets, which can also reduce if not eliminate redundant work, and lead to a faster time to value.

Conclusion

The. event mesh pattern is both a unifying and a foundational pattern. It complements and enhances virtually all other architectural patterns, acting as connecting event fabric and providing real solutions to the challenges posed by the ever-increasing volumes and variety of real time data, and the broad distribution of event processing centers.

Some of the key capabilities of the event mesh are:

• Dynamic routing
• Protocol agnosticism
• Support for all event exchange patterns
• Designed with the 3Vs of data movement in mind (Velocity, Variety, Volume)

These capabilities make the event mesh pattern an ideal architectural pattern that can add value to pretty much any use case or scenario. In fact, no mature event-driven architecture can be complete without the use of the event mesh pattern.

Appendix: The Application Network Relationship

In recent years a new paradigm relevant to the domain of microservice architecture has arisen.  Solidly based on domain-driven design (DDD) principles the application network is a way to organize microservices in a “composable” structure. The premise of the application network is that APIs can be seen as “Lego blocks” that can be assembled and disassembled at will to generate new business applications and offer a better way to adapt to the increasing market demands.  The application network allows the business to be agile and minimize the impact of change by designing modular architectures from reusable building blocks.

The original unit of work of the application network was the RESTful (synchronous) API, and the “composite business APIs” were essentially orchestrations of smaller grained system (data access) APIs. Due to increasing prevalence of event-driven architecture however, the application network needs to be expanded in its scope to also incorporate event-driven APIs.

The extended application network can essentially support any use case, as it allows one to model both synchronous and asynchronous interactions and support any data/event movement pattern. In fact, you can look at the application network as a way to “weave” all other architectural and design patterns together in support of any use case.

The event mesh, the implementation of the event mesh pattern is intended to support fast, varied and voluminous data sets in motion, and it is designed to scale seamlessly across a global geography. Because of its emphasis on supporting volume and speed, it can be placed at the center of the application network Architecture easily integrating with other synchronous aspects of the application network

Thus, event consumers can perform enrichment or persistence of event data via RESTful APIs, and in turn RESTful APIs can in fact act as event producers to the event mesh.  Lastly, the same way that RESTful APIs use standards like Open API, RAML etc. to describe and define the API contracts, asynchronous APIs can use AsyncAPI for event-driven APIs.

The diagram below illustrates a conceptual rendition of an extended application network where one can see AsyncAPIs acting as entry points into the event mesh and interacting bi-directionally with their RESTful counterparts.

A natural follow up question to the role of the event mesh in the application network structure, is where and how does the service mesh fit in?   A service mesh is the infrastructure that benefits the deployment of RESTful microservices, facilitating more direct communication between the deployed microservices.  The event mesh is similar in function to the Serice Mesh in that it facilitates the communication between the event broker nodes that are part of the event mesh. The difference is that whereas the service mesh optimizes the communication between dockerized microservices (running some type of logic), for the event mesh microservices are clients (producers/consumers) of the event mesh and as such are not part of the internal communication of the event mesh.

The diagram below illustrates the relationship between the event mesh and service mesh in the broader application network context:

In conclusion, the extended application network provides a complete foundation for realizing the composable enterprise and its inherent benefits.