Events meet APIs: Building High-velocity Business Applications with MuleSoft and Solace

The challenges associated with implementing a successful integration strategy are many, especially when you have to design a system that accounts not just for ever changing business requirements, but a continuously evolving technology landscape. Some of the technical factors affecting architectural design decisions are:

• Increase in data volumes, variety and velocity: as a result of the increase in the need for connectivity to an ever-expanding gamma of applications and devices across a wide array of environments (IoT, cloud apps, on-prem apps etc.).
• Increases in cross environment traffic between cloud and on-prem systems: because of data synchronization requirements, and the need for more accurate analysis
• Increase in geographic distribution: because of businesses growing into new markets – and requiring often exponential scaling

In addition to the technical factors there are also specific business drivers that are pushing companies to adopt new approaches to integration:

• Improving customer experience: customers want real time information over their preferred channels so they can take advantage of opportunities and/or detect threats
• Consolidating information: customers need to have all their information (which is typically spread over multiple systems) available to them in real time with minimal effort; whether that information needs to be available in a unified view on demand or delivered to them asynchronously as a result of specific changes.
• Accelerating innovation: the key to having a faster time to market with products and services, is to be agile and flexible, and adopt a composable enterprise attitude; discoverability and reuse of digital assets are key aspects that are required to achieve this.

It is widely accepted that to design well balanced and scalable integration architectures, you need both request-reply and event-driven patterns. Over the years these architectural patterns have evolved in parallel, but not equally. Although most real time enterprise applications that operate in the background (e.g. trading applications, order management, etc.) leveraged an event-driven approach, most user facing applications were based on synchronous/REST-full principles. As demand for web/mobile based experiences increased over the past two decades the REST-full domain has seen increased investment, driving it to reach a high degree of maturity. REST-full APIs, the basic building blocks of modern web-based architectures, have had an entire ecosystem built around them, an ecosystem that addresses the entire lifecycle of the API from design, build, deploy and manage to security, consumption and monetization.

The event-driven domain however has not seen such investment. Although over the years the event-driven space has seen significant growth and adoption, it has remained as a separate domain; whereas REST has seen a complete democratization of access and consumption, the event world remains relegated to the realm of enterprise experts.

In the last few years, however, a shift has begun to occur, a shift driven by the demand for more real-time experiences from customers. Customers now demand to be notified when a sale on their preferred product occurs, or if their credit card is about to be defrauded. Those experiences are fundamentally event-driven, and the demand for these experiences has now pushed the event-driven domain to the forefront alongside its REST-full counterpart.

Given this change, we now need to treat event APIs (specifications encapsulating event definitions and their respective flow direction), in the same way that we treat REST APIs. Having a unified developer and management experience for both these paradigms are essential to ensure that complex and composite integration architectures can be developed.

MuleSoft has been a long time a proponent of “Unified API Management” where REST APIs and event APIs are managed in the same fashion, offering developers a uniform developer experience across both asset categories. Recognizing the ever-increasing market demand for real-time, event-driven experiences, MuleSoft has added support for event APIs to its flagship Anypoint platform. With this addition, MuleSoft can now help its customers expand their existing Application Networks to include event APIs without disrupting their existing development lifecycle

Much like MuleSoft is an industry leader in integration and API Management, Solace is an industry leader in event streaming. They share many customers who want to benefit from the exceptional capabilities of both platforms. As customers wanted a deeper synergy between the two platforms, MuleSoft has added direct support for Solace to its Anypoint platform.

In this article, we will explore how MuleSoft and Solace are paving the way for Event-driven Integration.

API Lifecycle Management

With the October 2024 Release of Anypoint Code Builder, MuleSoft has added support for event APIs to its API-led connectivity principles. One can now build comprehensive “hybrid” architectures, that contain a combination of both Event-driven and REST-full patterns.

MuleSoft sees event APIs as first-class citizens, deserving of the same benefits that their REST counterparts benefit from. It is important to note that event APIs are defined by the Async API specification, REST-full APIs are defined by the RAML, or OAS specifications.

Let’s look at some of the additions that MuleSoft has made to its Anypoint Platform to support event APIs. Following the API Lifecycle process, we will look at how certain steps in the lifecycle have been adapted to offer support for event APIs.

Design

MuleSoft API Designer now supports the design of Async APIs alongside OAS and RAML APIs. The experience is almost identical, making an API designer’s life easy. Support for Async API testing is planned for a future release,  thus offering full capability parity with API Designer for REST APIs.

Publish

Once an Async API is designed, it can be published to Exchange much like an OAS or RAML API is.  In fact, MuleSoft Exchange offers now a “single pane of glass” view of both REST and event APIs, which goes hand in hand with MuleSoft’s philosophy of unified API management.

Build

MuleSoft developers are used to using Anypoint Studio or Anypoint Code Builder to implement REST APIs. Now, both tools can peruse Exchange, discover Async APIs, and scaffold them. The process is identical to the OAS or RAML scaffolding, as such this makes building Event-driven Integrations “trivially” easy.  It is important to note that MuleSoft is also using the same API Kit framework for Async APIs , so developers will be familiar with the experience.

Deployment and Management will be identical to the process that one uses for REST APIs.

If one is using the Solace platform, event API Governance will be provided by the underlying security mechanism of the Solace platform (e.g. access control lists for topics and queues, authentication /authorization, encryption mechanisms etc.). However, as part of MuleSoft’s journey to provide full support of event APIs, “event gateways” will eventually be brought to bear to offer the same governance experience that REST APIs benefit from i.e. policies that can help shape event traffic and apply necessary content-based and volumetric filtering constraints.

Solace Contributions

Solace is a MuleSoft technology partner, and has been named by IDC as the premier event platform in the market. Many other analysts (including Gartner and Forrester) have remarked on Solace’s capability to offer a comprehensive event platform which includes event lifecycle governance.

In this section we will explore the things that Solace brings to the table to complement MuleSoft’s capabilities.

PubSub+ Platform

Whereas Anypoint Platform facilitates the deployment and management of MuleSoft engines (containing either Event or REST APIs), PubSub+ Platform facilitates the deployment and management of event brokers (that underpin event API utilization) to any cloud, or on-prem environment. In essence Solace offers the infrastructure which the MuleSoft event APIs leverage to construct complex event-driven Integrations.

Some of PubSub+ Platform’s key capabilities are:

• Dynamic Routing: PubSub+ simplifies integration architecture, by removing the need to create routing logic in MuleSoft integration projects.
• Shock absorption: PubSub+ event brokers can smooth out traffic allowing slower consumers to not get flooded by faster producers.
• Protocol support: Not only can PubSub+ event brokers be bridged to any other event broker (e.g. Kafka, IBM MQ, etc.), they can also “speak” multiple protocols (i.e. AMQP, HTTP, JMS, MQTT, WebSockets, etc.)
• Event Mesh: An event mesh is a network of event brokers that allows events from one application to be dynamically routed and received by any other application no matter where these applications are deployed.

In short, PubSub+ Platform complements MuleSoft Anypoint Platform in the design and implementation of event-driven integrations.

PubSub+ Micro-Integration for Mulesoft

PubSub+ Micro-Integration for Mulesoft (known until recently as PubSub+ Connector for MuleSoft) can be used to connect with the Solace platform, and thus leverage the capabilities that Solace has to offer in designing complex event-driven integrations. It is available via Solace’s Integration Hub and Anypoint Exchange.

Whether used standalone in a MuleSoft-based integration or used as part of the API Kit through the scaffolding process, the Solace connector is the same. It brings to bear Solace’s connectivity capabilities and leverages the power of hierarchical topic structures, variables and wildcards used in the definition and utilization of topics.

With topic segments as variables for publishers and wildcard-based subscriptions one can achieve a high degree of flexibility and granularity in the way one can segment data channels and optimize data distribution.

PubSub+ Event Portal

PubSub+ Event Portal is a key part of Solace PubSub+ Platform. It does two things:

1. Allows for the holistic design of event-based applications based on core domain-driven design principles
2. Provides governance for Events.

On the first point, Solace takes a domain-driven design approach to designing event-driven application networks. One can take any business domain (e-commerce, inventory management, order fulfillment etc.) and design all the components that make up that domain: events, event schemas, applications (producers/consumers), event flows/distributions etc. An important thing to note is that an application in the Solace context is in fact an event API that is defined by an Async API specification. The diagram below illustrates a hypothetical order fulfillment process:

As such, Solace takes a holistic approach to creating the “event graph”. The event graph is not unlike the MuleSoft Visualizer in that it shows the interactions and dependencies between APIs, with the difference being that whereas the Solace event graph offers a design time perspective, the MuleSoft Visualizer offers an operational/runtime one.

On the second point, the event platform manages the Lifecyle of an event from its creation to its various versions, and to its eventual retirement. Thus, Event Portal lets you manage the lifecycle of events much like Anypoint manages APIs.

Another important thing to note is that one can indeed use principles of API-led integration with event APIs. For instance:

• System event APIs: encapsulate events generated from a system of record, accessible within a business domain i.e. create/update account. Event Storming principles classify these events as “command events”.
• Process event APIs: encapsulate events that contain information valuable at the enterprise level i.e. offer composite customer objects like account, customer etc. Event Storming principles classify these events as “aggregate events”.
• Experience event APIs: encapsulate events that expose information to end consumers i.e. notification, alerts etc. Event Storming principles classify these events as “domain events”.

Deployment

Once the application/event API has been implemented and tested in the Mule developer environment (Studio or Code Builder), it is ready for deployment and management. One can at this point deploy the Mule application in any of the traditional Mule ways. Solace’s contribution to this process is the GitHub workflow that orchestrates the deployment of multiple artifacts. If a Mule project is checked into GitHub, the workflow will be able to provision the asset directly:

• Deploy the Mule project to where it is configured to go to i.e. CloudHub, RTF.
• Configure the event broker bindings identified in the Async API spec (i.e. topics/queues).
• Update the Event Portal configurations with the respective binding configurations.
• 

Once deployed the Mule application can be monitored within Anypoint Monitoring and the event traffic can be monitored with the Solace PubSub+ platform (the Insights component).

From Integrations to Micro-Integrations

The journey of integration over the past two decades has gone from enterprise service busses (ESB) to microservices-based architectures and iPaaSs. The typical differentiating adage between the paradigms has been:

• ESB: smart pipes and dumb endpoints
• Microservices: smart endpoints and dumb pipes.

With the ever-increasing demand for event-driven integrations that require stable and scalable infrastructure, there needs to be a paradigm shift to: “smart endpoints and smart pipes”. This means that along with the microservices-based architecture (which MuleSoft provides) you now need an “intelligent infrastructure” (which Solace provides). This combination is the foundation for a wide variety of integration use cases that can span the edge (IoT), the enterprise, and the cloud/s. The core of PubSub+ Platform is PubSub+ Event Broker, which can be used to build an event mesh that filters and routes information over a globally-scalable interconnected infrastructure and simplifies the development and operation of an event-driven system.

Even though a construct like event mesh brings about great benefits to event-driven integrations, the way we build integrations themselves needs to change to derive the greatest benefits. Borrowing some concepts from the IEEE paper RFC-3439, we can consider changing the way we do integration by adopting the principle of moving complexity to the edge, while at the same time keeping the core simple. Many integrations are designed by coupling components together (connectors, transformers, routers etc.) to create a unified deployable asset. The challenges with this approach are that the deployable artifact is more difficult to scale, changes to one of the components in the “ensemble” may impact the other components and the uptime of the entire service, and failure in one of the components may impact the other components. Additionally, the integration project being in the data path, its scalability challenges may negatively impact downstream dependent services and integrations.

Following the recommendations given in the IEEE paper, one approach would be to decompose the integration project into more granular assets (micro-integrations – i.e. connectors, transformer etc.) that are independently deployable and that reside at the edges of the event mesh. This approach would allow for modifications to one micro-integration without impacting the others, it would allow for each micro-integration to be scaled out individually, and as the micro-integrations are at the edge, the flow of data is unhindered.

The event mesh has destination routing, and filtering built in, and as such you would not need to build infrastructure logic into the integration itself, thereby having a cleaner separation of concerns and a more targeted architecture. The event mesh also has built in shock absorption that allows it to regulate traffic implicitly and eliminates the need to add additional logic to account for edge scenarios.

Potential Use Cases

The combination of MuleSoft and Solace can be used across any industry vertical and can solve many challenges.  They both bring unique capabilities to the table that when combined create a superset that is very difficult to match. The use cases that benefit most from the combination are those that need to leverage the low latency nature of the Event Mesh and its global scale and those that have complex integration needs both at the edge and at the core of the enterprise.

Some of the many use cases that MuleSoft and Solace can address are:

• “Smoothing out” high volume, high velocity data across the “edge” and “core” of the enterprise (e.g. shock absorption for: IoT, Change Data Capture, and generally bursty data)
• High fan out data distribution (e.g. MDM updates)
• WAN distribution: where MuleSoft services need to communicate over a global network
• Distributed Data aggregation (e.g. Customer 360) – parallelizing data access
• Stateful choreographies for long running business transactions (e.g. Order Fulfillment)
• Hybrid environment data synchronization
• Multiplexing/demultiplexing data channels

Some of the areas and verticals where MuleSoft and Solace can have a high impact are:

• IoT: connecting the edge to the enterprise – Automotive (connected vehicles), Logistics (track and trace), Healthcare (tracking patient vitals) etc.
• Energy: smart grids, pipeline management, Resources (connected mines) etc.
• Financial Services: capital markets integration, market data distribution, payments, trade processing etc.
• Manufacturing: real-time visibility into the assembly line floors, supply chain integration, smart manufacturing (Industry 4.0), recall management etc.
• Retail: inventory optimization, eCommerce, customer experience management etc.
• Communications: customer experience, provisioning, activation etc.

In summary, MuleSoft and Solace together can have a tremendous impact and help solve many industry challenges.

Conclusion

MuleSoft has embarked on full support for the event API paradigm and has adapted its API lifecycle management to support both REST and event APIs.  Solace is aligned to MuleSoft’s methodology and can help MuleSoft customers take advantage of the benefits of event-driven integration. A new architectural approach based on the event mesh (and focused on accelerating data movement) can be used to reimagine event-driven integration, by recommending a higher degree of decoupling and moving complex integration processes to the edge. This new approach for designing event-driven integrations coupled with a large, combined set of unique capabilities, enable the MuleSoft and Solace ensemble to tackle a large amount of difficult use cases and provide compelling solutions across most industries.