More and more enterprises are moving from on-premises enterprise service buses (ESB) to integration platform as a service (iPaaS) solutions like Boomi, Jitterbit, Mulesoft Anypoint and others. iPaaS solutions like these are a key component of what Gartner calls a hybrid integration approach, and as someone accustomed to creating ESB solutions, it’s been eye-opening see how quickly and easily you can get them up and running. But as I create iPaaS solutions, one thought keeps running through my mind: this would be so much more powerful and elegant if we added an event broker to the mix.
At their heart, iPaaS solutions aim to simplify integration by creating a managed experience, particularly when deployed in the cloud. An iPaaS solution typically includes:
Although there are adapters for many different information providers such as databases and EDI, most iPaaS (and most ESB) solution’s bread and butter has been synchronous REST calls between software as a service (SaaS) providers like Salesforce and SAP S/4HANA Cloud. As a result, iPaaS providers have focused on making synchronous API integrations as easy as possible. Architecturally, this generally means the same process that receives an incoming request maps the data using a graphical tool, connects to the information consumer, and sends the transformed message.
Adding an event broker allows you to separate (decouple) that process, such that the iPaaS receives information via one process and writes it to a queue or topic, then a second process picks up the message and performs additional logic, typically including data transformation, before forwarding it to a consumer.
That decoupling can enhance traditional ESBs, but in some ways it’s even more important for iPaaS solutions. The architectural shift to decoupling using an event broker can improve an iPaaS’s resilience, agility and real-time capabilities—all without complicating its architecture or operation. The approach is a lot like breaking up a monolithic application into multiple microservices.Event-Driven Integration with iPaaS - The Architect's Guide to ImplementationIn this guide, you will learn how pairing an iPaaS with an event broker will help you create a robust and scalable architecture and the technical steps you need to take to start reaping the benefits of event-driven integration
Some brief asides:
One of the most complex and time-consuming tasks for any integration architect is making sure a system performs when things go wrong – in fact some experts say up to 25% of an integration project should be devoted to designing exception handling, and iPaaS solutions are no different. That isn’t surprising given that architects need to consider complex questions including:
Introducing an event broker as part of a decoupled architecture, on the other hand, can answer some of the questions while keeping the iPaaS simple. As shown in Figure 3, by persisting messages to a queue or topic after arrival at the iPaaS, the event broker takes responsibility for guaranteeing that the message will not be lost in the event of a failure that brings down either your process or the entire iPaaS. Without an event broker, an iPaaS either requires custom persistence logic or accepts that messages will be lost in the event of failure, requiring the information provider to resend as necessary.
Once the message is persisted by an event broker, the iPaaS receiver (not the end consumer) can acknowledge that the message has been successfully processed and becomes available for the next incoming message.
This ability for the iPaaS receiver to quickly and independently process messages comes in handy if the downstream information consumer isn’t available. Typically, an information consumer being unavailable results in multiple attempts to retry the transaction until it succeeds. As shown in Figure 4, if the receiver and sender are on the same thread, this can quickly cause the iPaaS to run out of resources, creating a chain reaction that blocks incoming transactions. By buffering incoming transactions, an event broker helps to keep your solution up and running. This buffering is even more valuable as we add in additional consumers, which we discuss in the next section.
Lastly, if the error turns out to be one that will take a while to resolve (e.g. someone to add a value to a look up table), the message can be placed into an error queue. When the issue is resolved, the message can be requeued for processing.
To this point, we’ve focused on the benefits of decoupling a single sender and receiver from one another, and the benefits of that decoupling are worthwhile on their own. But the value really adds up as you begin distributing transactions amongst multiple diverse consumers using a fully event-driven architecture. Making your integrations more agile lets your architects and developers innovate more freely, solving business needs and exploring new frontiers for your business.
Architects need to consider complex questions including:
To aid agility, iPaaS providers typically feature integrated source control and deployment capabilities. Including these components reduces the amount of additional infrastructure needed to start building an iPaaS solution – no more setting up a Git repository and configuring Jenkins.
Most iPaaS solutions include the ability to call multiple sender processes from a single receiver process, but as Figure 6 shows, even with the code promotion infrastructure in place, if the solution tightly couples senders and receivers of information you still need to have a complete test cycle and deployment with every change. That cycle is represented by a link icon in the diagram below. This tight coupling also makes error handling more complex, particularly in cases where one consumer rejects the transaction and additional consumers process it successfully.
An event broker allows you to move into the realm of fully event-driven architecture. At the outset, the addition of an event broker may seem like it increases complexity—after all, there are now more components—but the addition of those components is more than offset by the fact that an event broker makes code more modular and reduces integration challenges.
By playing middleman between receivers and senders, an event broker lets you change a sender without redeploying and retesting the entire process. What’s more, you can very easily add new senders and consumers, and seamlessly integrate them into an existing solution.
With an event broker, you can design a new sender connecting to an innovative new application using iPaaS integration tools like graphical mapping, content-based filtering and pre-built connectivity. Then once you add a new subscription to the event broker, events begin flowing to the new process and onward to the information consumer.
At runtime, decoupling lets consumers process information independently. This can help to resolve several difficult architectural challenges:
Beyond decoupling and event-driven architecture between iPaaS components, an event broker provides an on-ramp to real-time capabilities for your iPaaS. The internet of things (IoT) and other real-time information sources such as Workday and Salesforce present a massive opportunity to increase your business’s ability to react, but you typically need to act on such information in real-time to fully exploit its value. Real-time information sources can also send a massive spike in traffic during periods of high demand.
The need to process incoming events in real-time, and deal with sudden unpredictable bursts of data can be challenges for any iPaaS. In the absence of an event broker, iPaaS solutions are typically triggered either through a synchronous API request, or on a polling interval defined when the process is developed and deployed. In the case of a process invoked by a synchronous API request, dealing with traffic spikes within the structures of an iPaaS typically means increasing the number of processing threads available to handle the incoming flow of information and over allocation of memory and processing power. In the case of a polling interval, it is difficult to predict how often to poll far in advance, especially if you can’t change that interval when conditions change.
An event broker helps you overcome these challenges by providing an entry point for event-driven applications. Messages that are placed on event broker topics can trigger your iPaaS immediately, without having to worry about selecting a static polling interval. An event broker can also buffer incoming events and provide a throttling mechanism that prevents downstream processes from being overwhelmed in periods of high traffic.
Event brokers are a great potential partner for your iPaaS, enhancing their resilience, agility and real-time capabilities. Recognizing those potential benefits is the first step in the journey. The next steps are recognizing and overcoming some challenges that event brokers can introduce. In the future posts, I’m looking forward to answering questions such as:
 For a more general introduction to messaging concepts and patterns, see https://www.enterpriseintegrationpatterns.com/patterns/messaging/Introduction.html