Supercharge your iPaaS for Better Agility, Resilience, and Real-time Capabilities with an Event Broker

More and more enterprises are moving from on-premises enterprise service buses (ESB) to integration platform as a service (iPaaS) solutions like Boomi, Jitterbit, Mulesoft Anypoint and others. iPaaS solutions like these are a key component of what Gartner calls a hybrid integration approach^[1], and as someone accustomed to creating ESB solutions, it’s been eye-opening see how quickly and easily you can get them up and running. But as I create iPaaS solutions, one thought keeps running through my mind: this would be so much more powerful and elegant if we added an event broker to the mix.

iPaaS and Synchronous REST Calls

At their heart, iPaaS solutions aim to simplify integration by creating a managed experience, particularly when deployed in the cloud. An iPaaS solution typically includes:

• Graphical tools that show how transactions flow through the system
• Connectors that contain logic for popular protocols
• Built-in mapping tools that allow for data transformation
• Defined locations for code snippets that isolate customer-specific modifications, typically in a language like JavaScript
• Integrated source code repositories and deployment capabilities that are usually external in ESBs

Although there are adapters for many different information providers such as databases and EDI, most iPaaS (and most ESB) solution’s bread and butter has been synchronous REST calls between software as a service (SaaS) providers like Salesforce and SAP S/4HANA Cloud. As a result, iPaaS providers have focused on making synchronous API integrations as easy as possible. Architecturally, this generally means the same process that receives an incoming request maps the data using a graphical tool, connects to the information consumer, and sends the transformed message.

Figure 1: Typical iPaaS process flow

Adding an Event Broker to Your iPaaS Solution

Adding an event broker allows you to separate (decouple) that process, such that the iPaaS receives information via one process and writes it to a queue or topic^[2], then a second process picks up the message and performs additional logic, typically including data transformation, before forwarding it to a consumer.

Figure 2: iPaaS flow with event broker

That decoupling can enhance traditional ESBs, but in some ways it’s even more important for iPaaS solutions. The architectural shift to decoupling using an event broker can improve an iPaaS’s resilience, agility and real-time capabilities—all without complicating its architecture or operation. The approach is a lot like breaking up a monolithic application into multiple microservices.

Event-Driven Integration with iPaaS - The Architect's Guide to ImplementationIn this guide, you will learn how pairing an iPaaS with an event broker will help you create a robust and scalable architecture and the technical steps you need to take to start reaping the benefits of event-driven integrationDownload

Some brief asides:

• Many iPaaS providers have begun to integrate lightweight event brokers into their products. While I think there are advantages to having an event broker live outside the iPaaS itself, I will save that for another day and focus on the benefits of all event brokers, including those they include.
• While event brokers provide great benefits for iPaaS architectures, they also present challenges. In a future blog I’ll discuss those challenges, the trade-offs they entail, and give guidance on architectural choices that can help you overcome them.

Planning for Failure – How an Event Broker Helps an iPaaS Stay Simple

One of the most complex and time-consuming tasks for any integration architect is making sure a system performs when things go wrong – in fact some experts say up to 25% of an integration project should be devoted to designing exception handling, and iPaaS solutions are no different. That isn’t surprising given that architects need to consider complex questions including:

• What if the iPaaS itself fails? Will I lose in-flight transactions?
• What should happen if a consumer goes down and I need to retry? How can I prevent it from decreasing performance and blocking incoming?
• What if a consumer can’t process my incoming message until a human intervenes (e.g. adding a value to a cross reference table)? How can I easily move on to the next transaction and replay it later?

You can try to solve these issues within the iPaaS itself, perhaps by creating a library of custom code snippets in a language like JavaScript that dictate architectural standards like manually compensating for failed transactions and using adapters to ensure that transactions are persisted to disk. However, in doing so you sacrifice some of the simplicity that makes iPaaS so advantageous.

Introducing an event broker as part of a decoupled architecture, on the other hand, can answer some of the questions while keeping the iPaaS simple. As shown in Figure 3, by persisting messages to a queue or topic after arrival at the iPaaS, the event broker takes responsibility for guaranteeing that the message will not be lost in the event of a failure that brings down either your process or the entire iPaaS. Without an event broker, an iPaaS either requires custom persistence logic or accepts that messages will be lost in the event of failure, requiring the information provider to resend as necessary.

Figure 3: Event brokers allow for message persistence in the case of failure

Once the message is persisted by an event broker, the iPaaS receiver (not the end consumer) can acknowledge that the message has been successfully processed and becomes available for the next incoming message.

This ability for the iPaaS receiver to quickly and independently process messages comes in handy if the downstream information consumer isn’t available. Typically, an information consumer being unavailable results in multiple attempts to retry the transaction until it succeeds. As shown in Figure 4, if the receiver and sender are on the same thread, this can quickly cause the iPaaS to run out of resources, creating a chain reaction that blocks incoming transactions. By buffering incoming transactions, an event broker helps to keep your solution up and running. This buffering is even more valuable as we add in additional consumers, which we discuss in the next section.

Figure 4: Event brokers conserve resources during retry

Lastly, if the error turns out to be one that will take a while to resolve (e.g. someone to add a value to a look up table), the message can be placed into an error queue. When the issue is resolved, the message can be requeued for processing.

Figure 5: Event brokers simplify replay after human intervention

The Value of Event-Driven Architecture

To this point, we’ve focused on the benefits of decoupling a single sender and receiver from one another, and the benefits of that decoupling are worthwhile on their own. But the value really adds up as you begin distributing transactions amongst multiple diverse consumers using a fully event-driven architecture. Making your integrations more agile lets your architects and developers innovate more freely, solving business needs and exploring new frontiers for your business.

Architects need to consider complex questions including:

• How can I quickly introduce new business functionality while making sure I’m not affecting existing functionality?
• How do I handle situations where one consumer processes information faster than another? Or one consumer rejects the transaction, while another processes it?
• How do I make sure I’m not locking myself into a deployment pattern (on-premises, cloud, etc), cloud provider, or even iPaaS solution?

How Event-Driven Integration is Helping HEINEKEN Become the World’s Best Connected BrewerTo overcome challenges associated with "bursty" data flow and to meet their ambitious digital transformation targets, HEINEKEN is taking an event-driven approach to integration.Read Now

To aid agility, iPaaS providers typically feature integrated source control and deployment capabilities. Including these components reduces the amount of additional infrastructure needed to start building an iPaaS solution – no more setting up a Git repository and configuring Jenkins.

Most iPaaS solutions include the ability to call multiple sender processes from a single receiver process, but as Figure 6 shows, even with the code promotion infrastructure in place, if the solution tightly couples senders and receivers of information you still need to have a complete test cycle and deployment with every change. That cycle is represented by a link icon in the diagram below. This tight coupling also makes error handling more complex, particularly in cases where one consumer rejects the transaction and additional consumers process it successfully.

Figure 6: iPaaS solution with multiple tightly coupled consumers

An event broker allows you to move into the realm of fully event-driven architecture. At the outset, the addition of an event broker may seem like it increases complexity—after all, there are now more components—but the addition of those components is more than offset by the fact that an event broker makes code more modular and reduces integration challenges.

By playing middleman between receivers and senders, an event broker lets you change a sender without redeploying and retesting the entire process.  What’s more, you can very easily add new senders and consumers, and seamlessly integrate them into an existing solution.

With an event broker, you can design a new sender connecting to an innovative new application using iPaaS integration tools like graphical mapping, content-based filtering and pre-built connectivity. Then once you add a new subscription to the event broker, events begin flowing to the new process and onward to the information consumer.

Figure 7: iPaaS with loosely coupled consumers using event broker

Handling Slow Consumers and Routing Information with an Event Mesh

At runtime, decoupling lets consumers process information independently. This can help to resolve several difficult architectural challenges:

• The first challenge decoupling alleviates is information consumers that process transactions at different rates of speed. An event broker creates a buffer that allows slow consumers to process messages at their own pace without affecting or constraining those applications that can process messages much more quickly.
• Another challenge is dealing with instances in which one consumer rejects an incoming transaction, but other consumers are still able to process it. An event broker can handle that scenario without resorting to adhoc error handling. That said, error handling in an event-driven architecture can be tricky, and I look forward to diving into that topic in a future blog post.
• By linking multiple brokers in different locations to form an event mesh, event brokers can efficiently route information between on-premises and cloud applications, different clouds and even different iPaaS solutions—without having to explicitly route information between them within your iPaaS.
• Such an event mesh can also address the challenge of unreliable WAN connectivity by provide reliable delivery without each iPaaS application needing to handle this. If your deployment pattern changes, the event mesh dynamically reroutes events to the new topology.

Reacting to Real-Time Event Sources….in Real-Time

Beyond decoupling and event-driven architecture between iPaaS components, an event broker provides an on-ramp to real-time capabilities for your iPaaS. The internet of things (IoT) and other real-time information sources such as Workday and Salesforce present a massive opportunity to increase your business’s ability to react, but you typically need to act on such information in real-time to fully exploit its value. Real-time information sources can also send a massive spike in traffic during periods of high demand.

The need to process incoming events in real-time, and deal with sudden unpredictable bursts of data can be challenges for any iPaaS. In the absence of an event broker, iPaaS solutions are typically triggered either through a synchronous API request, or on a polling interval defined when the process is developed and deployed. In the case of a process invoked by a synchronous API request, dealing with traffic spikes within the structures of an iPaaS typically means increasing the number of processing threads available to handle the incoming flow of information and over allocation of memory and processing power. In the case of a polling interval, it is difficult to predict how often to poll far in advance, especially if you can’t change that interval when conditions change.

Figure 8: Using polling for real-time event processing

An event broker helps you overcome these challenges by providing an entry point for event-driven applications. Messages that are placed on event broker topics can trigger your iPaaS immediately, without having to worry about selecting a static polling interval. An event broker can also buffer incoming events and provide a throttling mechanism that prevents downstream processes from being overwhelmed in periods of high traffic.

Figure 9: Using an event broker for real-time event processing

Conclusion

Event brokers are a great potential partner for your iPaaS, enhancing their resilience, agility and real-time capabilities. Recognizing those potential benefits is the first step in the journey. The next steps are recognizing and overcoming some challenges that event brokers can introduce. In the future posts, I’m looking forward to answering questions such as:

• What are the trade-off between using an iPaaS-provided message broker and an external one?
• What does error handling look like when using an event broker?
• What are some architectural “gotchas” associated with using event brokers with iPaaS?
• How can be my business make the leap to event-driven?

[1] For more on HIP, see https://www.gartner.com/smarterwithgartner/use-a-hybrid-integration-approach-to-empower-digital-transformation/

[2] For a more general introduction to messaging concepts and patterns, see https://www.enterpriseintegrationpatterns.com/patterns/messaging/Introduction.html