How Event-Driven Integration can De-silo your Data and put it into Motion

As I talk to application developers and integration architects with existing and prospective customers, many are seeking a way to enable and manage the distribution of data between all of the disparate real-time data silos that exist across their business.

The reality is that some data is being shared in real-time with modern event brokers such as Kafka and Solace, while other times it’s being distributed via legacy integration tools like Tibco EMS, Oracle Service Bus/Weblogic JMS or IBM Integration Bus/IBM MQ. The challenge is each person looks at the problem differently, uses different terminology to explain their problem, and is looking for different business outcomes.

Just yesterday, I talked to one of the world’s largest grocery retailers about their challenges. Here’s how they described their problem:

Each of our domains have their own events, and they use a variety of different brokers (such as Solace, Kafka, Weblogic JMS, and IBM MQ) and ESB and iPaaS solutions to get them where they need to be. So our integration strategy is fragmented, which is fine, but we need the ability to share real-time events across the enterprise so each domain can easily find events to integrate with, and we also need the ability to govern things to prevent unauthorized data access.”

Sounds hard, right? Their quandary is becoming more and more common, but how do you solve it? Event-driven integration can help. Here’s how:

• By turning your integration architecture inside out
• By establishing an event mesh that links everything
• By giving you and your team self-service access and governance

I’ll explain what I mean by each of those…

Turn your Integration Architecture Inside Out

Today, many enterprises integrate systems almost exclusively with synchronous, orchestrated integration flows – flows that are centralized by the nature of their iPaaS or ESB implementation. We believe you should think of your enterprise architecture differently by adopting event-driven integration, which looks at integration infrastructure with two different lenses:

• Distribution at the Core: refers to part of the system that focuses on facilitating the movement of data and event.
• Connectivity at the Edge: refers to the part of the system that has to do with event processing i.e. connectors, transformers, content-based routers, etc.

In essence, event-driven integration turns integration inside out by pushing integration and connectivity to the edge while the event-driven core focuses on moving data across cloud, on-premises and IoT environments and geographies.

This approach maximizes the flow of events and information while improving your system’s scalability, agility, and adaptability. It also allows you to build small, micro-integrations with a variety of integration solutions such as Boomi, SAP Integration Suite, Mulesoft, Camel, etc. Finally, it enables a decentralized approach to integration and data sharing since each team/domain can choose their own tooling and pace.

Embrace the Power of an Event Mesh

Moving events in real-time across your distributed enterprise is hard, as is providing a universal connectivity layer. Doing both with stability, reliability and the ability to support future use cases feels impossible. Or is it?

Solace pioneered the concept of event mesh – linking event brokers into a network that dynamically routes information from any source to any system that needs it, anywhere in the world, running in any kind of cloud, on-premises or IoT environment.

What you should avoid considering is doing point-to-point integrations between your existing estate of different event brokers without an event mesh. The reason is that the cost/complexity increases, there is no uniform policy management and governance and finally it is brittle as application evolution happens.

So, if you avoid point-to point integrations and adopt an event mesh, all those things that were hard, now become easy. You cannot implement this type of an architecture with Kafka, AnypointMQ, or Cloud Native Messaging (Amazon SQS/SNS, Google Pub Sub, etc).

Offer Self-Service Access and Governance

So you’ve flipped your architecture inside out and picked your event broker – now for the last mile. How do you make it easy to create new integrations and keep it all governed?

This also feels hard, but it doesn’t need to be! Solace invented and brought to market the world’s first event portal that leverages AsyncAPI to make the developer and integration experience easy. Imagine being able to discover business events, define your integration flow’s inputs/outputs and seamlessly generate the flow scaffolding in the technology of your choice.

That is exactly what we do for Mulesoft via the Solace Micro-integration Generator for MuleSoft Anypoint Platform that generates integration flows for Mulesoft and several other iPaaS and development technologies such as Spring Cloud Stream. You can see it in action by watching this brief video, or learn more and download it on Github.

But how do you keep it from spiraling out of control by having sensitive data leakage or breaking other integrations with changes as things evolve? Event Portal helps with that too via a complete lifecycle management capability that supports approvals, versioning and promotion. You get to make life easier for your development and integration users,  and give yourself the kind of control and governance you need.

Conclusion

The three considerations above will hopefully help you decompose your own problem statement, determine how to think about the problem and guide you to the best possible solution which I strongly feel is by turning to event-driven integration. Do you have these types of challenges in your organization? Do you want to integrate more systems in real-time? Do you want to change the way your business and customer interacts? If so, let’s chat!

Event Driven Integration with iPaaS - The Architect's Guide to ImplementationIn this guide, you will learn how pairing an iPaaS with an event broker will help you create a robust and scalable architecture and the technical steps you need to take to start reaping the benefits of event-driven integrationDownload