Why Your iPaaS Event-Driven Integration Architecture Is Broken—and How to Fix It

Ahhh, the wonderful world of event-driven integration. Combining an iPaaS solution with an event broker to create a beautiful partnership that gets real-time data to your customers faster. But like so many other things, it’s not a silver bullet—poorly implemented event-driven architecture can be just as bad as a point-to-point solution. Fortunately, most of the potential gotchas with iPaaS event-driven integration are avoidable with some architectural foresight and a valuable toolset – an event portal.

These are some of the challenges you will run into with event-driven architecture and an iPaaS solution:

• You can’t understand which events available or which applications are consuming them
• You can’t trace an event through the system
• You struggle with error handling

Luckily, there are a few things you can implement to make your life easier.

Overcoming the Challenges of Decoupling: Event Management

We have covered the benefits of a decoupled architecture extensively in other blog posts. Decoupling lets you bring innovative solutions to market quicker, simplify error handling, and more easily exploit the power of mobile devices and Internet of Things (IoT) sensors. Decoupling an architecture through events can transform a business, making it more responsive to customer demands. It’s a good thing.

But decoupling also has drawbacks. In a traditional point-to-point architecture, iPaaS solutions graphically show you what applications connect to one another. You’ve got a GUI and there is a line between Salesforce and SAP, showing exactly how the iPaaS connects the two, maps the data, and other business logic.

In a decoupled architecture, you still have some of those nice graphical elements. But to get things decoupled, the iPaaS also publishes messages to an event broker, which then distributes the event to interested downstream applications. Especially at the beginning of an event-driven transformation, it can seem like there is a black box in the middle of your architecture. And that can make it hard to figure out what information is available in your enterprise and what applications are consuming it.

Event Driven Integration with iPaaS - The Architect's Guide to ImplementationIn this guide, you will learn how pairing an iPaaS with an event broker will help you create a robust and scalable architecture and the technical steps you need to take to start reaping the benefits of event-driven integrationDownload

Manage Asychronous Events with iPaaS and an Event Portal

That’s why an event portal is key for an event-driven architecture and, by extension, event-driven integration with iPaaS. Think about API gateways for synchronous REST and SOAP APIs: you could track the details of what APIs are available with spreadsheets, save the schemas describing data formats in shared drives and use email to notify people of changes to implementations. But it would be a confusing mess. API gateways put all the API management capabilities that you need into one convenient, integrated tool. To top it off, the best API gateways will integrate with your iPaaS solution, letting you pull information about APIs, endpoints, and data formats right into the GUI. It’s one stop shopping.

An event portal serves the same purpose for asynchronous events. Rather than having event definitions, topic strings, and applications strewn across shared drives and spreadsheets, event portals put it all in one convenient location. An event portal manages, visualizes, and governs events.

Given the emphasis on GUIs in an iPaaS, it’s also key that an event portal also presents this information in graphical form. This provides the “missing link” between what’s shown in your iPaaS GUI and how the event broker distributes incoming events.

Event Portal view of how applications are connected.

Finally, an event portal can seamlessly integrate with your iPaaS, just like synchronous API solutions. Rather than hunting for the latest version of a schema, or wondering what the current topic string is, an event portal provides the up-to-date configuration within the iPaaS GUI itself.

Overcoming the Challenges of Decoupling: Event Traceability

Once a solution moves into production, decoupling can also make it harder to determine what happened to a transaction that entered your iPaaS. One of the best features of an iPaaS solution is built-in transaction monitoring, which shows the exact steps that a transaction took as it moved through the system and what it looked like along the way. If a single process calls multiple other processes, you can see it nested within the monitoring report.

In contrast, once an iPaaS publishes an event, it can be hard for developers and support personnel to see where it went next. In a crucial situation, having immediate clarity about where an event went can mean the difference between a 2 minute-support call and an extended investigation. When moving to a combined iPaaS/event broker solution, you need to plan for how to keep things simple.

Add and Track Unique IDs

There are several steps to providing this functionality. It starts with adding a unique tracking id (sometime called a globally unique identifier or GUID) to an event’s header as soon as it enters the system. Or even better, include the GUID generated by the event source moves through your integration layer. No matter where the GUID comes from, it acts as bread crumb trail: as an event moves between iPaaS processes and external endpoints, you can always identify where it originally came from.

In addition to the GUID, extract out business-relevant fields (think person, account, or transaction identifiers) from the body of the message as quickly as possible and put them into the event’s header. This is one of the best aspects of having both an iPaaS and an event broker: an iPaaS is great at parsing out the body of a message, which event brokers struggle with. Frequently, people requesting support won’t know the GUID, they will just know business relevant information. For instance, if it’s a salesperson, they will know the account number, if it’s a nurse they will know the patient’s last name. Having business-relevant fields available makes everyone’s job a lot more pleasant.

Once you’ve got the fields you need, you can use functionality within your iPaaS to capture them in a searchable format. The exact implementation varies by iPaaS provider; for MuleSoft, check out transaction tracing; for Boomi, use Tracked Fields. Once the database captures the fields, support personnel can use them to troubleshoot and nail down where an event went. If internal tools don’t provide enough flexibility, or you want a graphical view of your events coursing through your iPaaS provider, then check out external tools like DataDog or Jaeger, which provide more customizable, powerful capabilities.

Overcome the Challenges of Decoupling: Error Handling

As mentioned previously, event brokers enhance iPaaS error handling in many ways. An event broker helps to handle situations like:

• Processes timing out after an extended retry
• Losing information when an iPaaS node fails
• One SaaS update failing after several other SaaS

In a traditional architecture, distributed transactions ensure that everything either succeeds or fails together. If the third database update fails, the first two roll back, simplifying the error handling architecture. However, distributed transactions have huge costs in terms of speed and flexibility, and modern applications communicating with REST don’t support it.

Full decoupling means that distributed transactions can’t be part of our toolset: every connection to an endpoint is independent. That has a lot of positive qualities, but it also begs the question: what takes its place?

Embrace Eventual Consistency… Where You Can

In contrast to strong consistency, which says all updates either succeed or fail at the same time, eventual consistency states that updates to various systems will “eventually” process all updates. Particularly when backed by an event broker that guarantees not lost messages, eventual consistency gives your iPaaS the flexibility it needs to deal with a distributed enterprise (check out this post for a more complete discussion of eventual consistency—and my front yard).

Even though eventual consistency is powerful, it requires a mindset adjustment on the part of developers and architects. It also requires a deeper understanding of what truly needs to be strongly consistent, and what can be left to eventual consistency.

There are situations where a transaction must absolutely have strong consistency. If a bank is wiring a million dollars out of my bank account (I wish!) and into yours (you wish!), it better be strongly consistent. In those cases, keeping things synchronous may be the right answer. Frequently though, once the crucial transaction has completed, there are follow-on actions that can complete asynchronously. The email alert to parties involved, the dashboard update to the loan officer, etc. can occur with eventual consistency.

A couple more architectural techniques you may be able to employ to get to eventual consistency:

• Change updates into insert/updates (e.g. upserts), to add flexibility
• Instead of required values, specify default values for SaaS applications (think a default geographic region in Salesforce) that later events can update

Conclusion

If you’ve been reading this series, you know that an iPaaS and an event broker are a powerful combination. But there will still be challenges along the way to a mature event-driven integration architecture. If you can dodge these common pitfalls, you’ll be ahead of the game.

How Event-Driven Integration is Helping HEINEKEN Become the World’s Best Connected BrewerTo overcome challenges associated with "bursty" data flow and to meet their ambitious digital transformation targets, HEINEKEN is taking an event-driven approach to integration.Read Now