Event Driven Integration: Meet the Event Mesh

In the first blog post of this series, Event-Driven Integration: Why it’s Time to Event Enable your Integration, I introduced event-driven integration and why it is now needed more than ever. In the second, Event-Driven Integration: Approach and Advantages, I described the approach in greater detail, and explained how it can change the integration landscape.The foundation of event-driven integration is a pair of principles: the “core” represented by an event platform, which in its scaled out form it becomes an event mesh, and the “edge”, which is represented by all manner of event processors (i.e. connectors, transformers etc.). In this article I will focus on the core, and specifically why an event mesh is the optimal conduit for data movement.

I will explain the idea of an event mesh through the lens of an architectural framework established by an organization called The Open Group, eponymously known by the acronym “TOGAF” which stands for The Open Group Architecture Framework. TOGAF is comprised of 3 layers:

1. Business Architecture looks at the capabilities that the business needs that can be addressed by an event mesh.
2. Information System Architecture looks at how the information flows across an event mesh, optimizations to the event flow and what does the application ecosystem look like.
3. Technical Architecture looks at the structure of an event mesh, including key components and looking at key capabilities related to scalability and agility.

What is an Event Mesh?

An event mesh, in a nutshell, is a set of event brokers connected in a manner akin to the way the routers are in an IP network. Much like in an IP network routers exchange routing tables via a common protocol (e.g. BGP, OSPF etc.), the event brokers in a mesh, exchange subscription tables with each other to streamline event distribution. As such an event emitted in New York will be forwarded to an interested party in Tokyo seamlessly and without any configuration efforts. In a sense an event mesh mimics the behaviour of the IP network at the application level.

The event mesh is designed to linearly scale across any environment and geography. You can find a more formal and in-depth definition of an event mesh here.

Business Architecture

Any technology has to provide implicit value to the business i.e. certain capabilities that can lead the business to either drive new revenue streams or optimize operations in order to reduce costs. The event mesh is designed to help IT departments roll out an event driven infrastructure that can empower the business. In its purest form an event mesh facilitates movement of data across any environment and any system. Given this, the natural question is what does the unobstructed, optimal movement of data get you or rather what type of capabilities does it facilitate?

The table below illustrates some of the key business capabilities that an event mesh enables:

In the subsequent sections I will explore how an event mesh can deliver on the business capabilities outlined above.

Information System Architecture

Now that you understand the key business capabilities that an event mesh enables, it’s time to look at what technical functions are required to make this happen.

As an event mesh is designed to primarily facilitate the optimal flow of information, a functional architecture is in essence an information flow/system architecture. The functional architecture can and should be composed of “functional blocks”. Business capabilities can be supported by combining/composing functional blocks. A basic set of functional blocks that describe an event mesh are as follows:

• Performance: maintain low latency and handle large volumes of events over any environment and any networking infrastructure.
• Scalability: maintain performance and reliability as an event mesh grows in scale and deployment.
• Routing/Filtering: have the ability to dynamically and natively route events to various destinations based on granular subscription criteria.
• Multiprotocol support (protocol bridging): the ability to support multiple client APIs, and move events along over multiple eventing protocols (JMS, MQTT, AMQP, Kafka etc. )
• Support all Event Exchange Patterns (EEP): support varied event-based interactions – peer-to-peer, pub-sub, streaming, request-reply.
• Connectivity: ability to connect to various data sources and destinations, as well as to various protocols
• Shock Absorption: ability to guarantee delivery of events in order even when there is a speed mispatch between producers and consumers.

The table below illustrates the capability-function map:

Now that I’ve explained how the technical functional blocks can help deliver on business capabilities, I’ll dive a level deeper and look at how they map to technology capabilities.

Technology Architecture

Functional blocks are represented by a set of technical features/capabilities. I will use the attributes and capabilities of an event mesh built with Solace PubSub+ Platform to illustrate how its native technical features map to the respective functional blocks described above.

Performance

Solace offers 3 form factors: a hardware appliance, a software broker, and a cloud service model. The appliance features the highest performance, offering consistent microsecond level latency. You can configure an event mesh by combining all three form factors across cloud and on-premises environments. For example, a fast core based on an appliance with slower but more distributed edges based on the software or cloud instances or optimize for fast connectivity at the edge with a combination of cloud and on-prem brokers.

Scalablity

An event mesh powered by Solace technology is the ultimate stage in scaling an event broker network and it encapsulates all other scaling mechanisms (i.e. partitioned queues for dynamic consumer load distribution, clustering for horizontal scaling and HA/FT etc.). The event mesh can scale across any form factor and any environment (cloud-cloud, cloud-on-prem)

Routing/Filtering

A Solace-based event mesh has two key differentiating capabilities:

• Topic structures: due to the hierarchical nature of a topic (i.e. /order/process/product/{state}/{city}/{store}) and the ability to substitute variable in the topic name, one can achieve a great degree of flexibility in publication. Conversely, wildcards (*, >) allow one to have a great degree of granularity in subscription (i.e. /order/process/product/OH/>, /order/process/product/IL/Ch*/* )
• Dynamic Message Routing: is a unique capability that allows all brokers in the mesh to share with each other subscription tables, so that the mesh can route any event to its destination along the necessary path without the need to configure these routing paths.

Multi-Protocol Support

Solace supports multiple protocols in two different ways:

• Client APIs and Open Protocols: A client may connect to a PubSub+ Event Broker in the mesh over a variety of protocols: MQTT, AMQP, JMS, REST, WebSockets etc.
• Protocol bridging: The event mesh supports the seamless transfer of events over various commercial protocols via protocol bridges/maps i.e. Kafka, IBM MQ, Google PubSub AWS SQS etc. For instance, an event emitted on Google PubSub can seamlessly make its way to the mainframe over IBM MQ via these protocol bridges.

Event Exchange Patterns

Solace supports all exchange patterns i.e. pub-sub, peer-to-peer, request-reply, streaming etc.

Connectivity

The event mesh is the best medium for real time data movement, however it can’t move any data unless it can connect to data sources and destinations. Via a wide array of connectors, the Solace event mesh can ingest and dispatch data to/from data bases, applications, files etc. Additionally, via iPaaS connectors Solace can further extend its connectivity reach to an even broader range of applications, data warehouses, etc.

Shock Absorption

One of the key capabilities of an event mesh is its ability to regulate traffic. It does so through its built-in buffering and caching capabilities where it can absorb huge bursts of input events. The mesh can detect slow consumers and not flood them with large volumes of data that they could not otherwise absorb, thereby maintaining the stability and robustness of the overall ecosystem.

As you can see, an event mesh built with Solace natively supports the functional blocks required to fulfill the needs of the business.

Event Mesh Pattern

Event mesh is both a technology platform and an architectural pattern. Even though a powerful technology on its own, an event mesh does not live in a silo. The IT landscape supporting enterprise architectures is often very diverse, and in many cases, there are duplicate tools that support similar activities. In addition, there are many architectural patterns at play (i.e. layered architectures, pipe-and-filters etc.) as well as many specific service design patterns (sagas, aggregators, CQRS etc. ) and they are combined to generate the optimal architecture and technology landscape that can support the operations and the growth of the business.

Event Mesh is a “foundational” pattern as it often helps “glue” other patterns together. No matter the set of patterns included in a given architecture, an event mesh can act as the “core” of the overall architecture and providing support and enabling the seamless composition of multi-dimensional architectures i.e. integrating IoT domains with the (on-prem) enterprise and cloud domains.

The event mesh also has a significant impact on the design of integration architectures. Due to its many native capabilities (dynamic routing/filtering, protocol bridging etc.) it can greatly simplify the integration architecture by eliminating the need to embed infrastructure logic in the integration processes themselves, and allowing them to strictly focus on delivering business value. Additionally, there are many macro integration patterns like data aggregation, distribution, stateful choreography etc. that an event mesh can not only facilitate but also optimize.

Conclusion

The event mesh is an essential part of the enterprise IT landscape. It supports key business needs and metrics such as giving the business real time access to data, improving customer experience, and minimizing the risk and impact of downtimes. It enables the fulfillment of the business capabilities through specific technical functions such as performance, scalability and connectivity. In turn, the actual features of an event mesh (i.e. dynamic message routing, multi-protocol support, etc.) are used to implement these required technical functions.

An event mesh can also be used as architectural pattern in designing large scale event driven systems. In a subsequent article I will explore in more detail the types of broader architectural patterns that an event mesh can be used to realize.