How to Event Enable your Boomi iPaaS in Six Simple Steps with PubSub+ Connector for Boomi

At its heart, event-driven integration means enhanced real-time customer engagement and better-informed, predictive decision-making. The Solace PubSub+ Connector for Boomi enables organizations to reap those benefits faster by creating a combined Boomi and Solace experience that’s greater than the sum of its parts. Boomi developers get to stick with the UI they know and love, while getting added benefits through the event-driven Solace platform. For Boomi architects, the connector fits hand-in-glove with a time-tested methodology and comprehensive platform powering your enterprise’s event-driven evolution. In this post, I will go over the following 6-step methodology that will help you get the most out of your event-driven integration investment.

Step 1: Formalize and Evangelize Your EDA Goals
Step 2. Understand Your Event-Driven Integration Foundation
Step 3. Speed Date Pilot Candidates and Pick the Lucky Winner  
Step 4. Event-Driven Design 
Step 5. Implement a Quick Win 
Step 6. Repeat 

(Async) API-First Development

Architects using synchronous connections like REST and SOAP have long acknowledged the importance of APIs. There’s a reason that tooling like API Management is such a crucial part of the Boomi platform. API Management enables modelling synchronous APIs, establishing service level agreements, regulating security, enforcing SLAs, registering data formats and ensuring that API users are notified of changes.

Despite their importance, traditional development methodologies (and the realities of rushed modern development) typically leave API definitions until late in the development process: after coding is complete, the developer exports the data structure schema and uploads it to API management.

Lately, thought leaders have turned this development process on its head. Instead of a late–in–the–game export, these thought leaders urge enterprises to define APIs at the start of the development cycle, then implement them using coding tools such as Boomi Integrate. This philosophy is called API-first.

Boomi responded by both modifying the platform to allow an API-first methodology and describing it’s use in prominent blog posts such as this one written by Boomi Field CTO Rene Klomp. While you should read the whole post, this is a summary of their approach to API-first development:

• Create a schema using a graphical mapping tool like Swagger.io
• Test the schema
• Promote the schema to Boomi API Management
• Create a process in Boomi Integrate, using the import capabilities of Boomi to grab the schema
• Complete the business logic implementation using standard shapes available in Boomi Integrate

Until recently, asynchronous implementations haven’t had the tooling or methodologies necessary for an API-first development process. That is no longer the case when pairing Boomi and Solace. Much like synchronous API management, architects and developers can now develop event-driven, asynchronous APIs and model them using graphical tools provided in Solace PubSub+ Event Portal. The newly released Solace PubSub+ Connector for Boomi can then seamlessly integrate your freshly created APIs into Boomi and use them as the basis for creating processes that follow best-practices for event-driven integration.

The enhanced tooling aligns with a standardized, repeatable methodology tailored for use in event-driven integration, both with organizations new to the concepts of event-driven architecture (EDA), and those well along the journey to a mature EDA implementation.

Step 1. Formalize and Evangelize Your EDA Goals

Before you begin the journey, it’s helpful to review and solidify why you are moving towards event-driven integration with Boomi and Solace. These goals can serve as key performance indicators in your new architecture and provide a helpful list of reminders of why you pursued this initiative in the first place.

Adding Solace to an existing Boomi infrastructure brings:

• Operational simplicity: integration that adjusts to different topologies such as on-premises and cloud-based deployments, while offering built-in high-availability and load-balancing capabilities
• Agility for adding features: simplifies adding innovative features on the fly
• Simplified error handling
• Realtime, global responsiveness: integration that starts immediately, rather than using polling or batch processing
• Increased robustness: integration that absorbs a sudden influx of data and simplifies error handling
• Agility for a changing business: integration that distributes incoming events to multiple information consumers and makes it easier to add more information consumers as your business changes
• Scalability: integration that grows with your business, while insulating legacy systems from increased traffic
• Design simplicity: integration that componentizes your Boomi processes for simpler choreography
• Next generation innovations: integration that allows for easier adoption of technologies like microservices, cloud, artificial intelligence, and machine learning
• Asynchronous API management: management through a graphical interface </>

Adding Boomi to an existing Solace infrastructure brings the ability to:

• Transform data from any format to any format using standardized, visual tooling
• Add content-based routing and route events based on their content
• Enrich events with information from databases, CRMs, flat file look ups, caches cross reference tables and more
• Split events from a single incoming event into multiple events for more targeted downstream processing
• Combine multiple transactions: into a single transaction, providing a more contextualized view of an event
• Bridge information coming from Electronic Data Interchange partners into the world of events
• Enhance connectivity options to databases, flat files and others
• Accelerate your web and app presence by using low-code application development
• Create a single “golden record” of information within an enterprise using master data management
• Manage synchronous APIs through a graphical interface

While event-driven integration provides numerous benefits, enterprises need to exercise care in their evolution towards event-driven maturity. Making sure that your organization is on board, from executive stakeholders down to junior developers, requires thought leadership and persistence. The integration team needs to start thinking about choreography, rather than orchestration. The API teams need to start thinking about event driven APIs, rather than just request/reply. Support personnel need to understand how events move through the enterprise. Leadership needs to grasp the benefits of real-time information flow across multiple clouds.

Fortunately, Boomi and Solace have collaborated on a wide range of materials to help you evangelize an organization. Here are some good ones:

• Four Factors to Consider When Choosing Between a Built-in vs. 3rd-Party Event Broker for iPaaS 
• Supercharge your iPaaS for Better Agility, Resilience, and Real-time Capabilities with an Event Broker

It’s important to note that evangelism is not a one-time activity. Keeping an EDA evolution on track requires a continuing process of education and realignment to goals as the journey continues. As your company onboards more events and starts to see the fruits of your labor, the evangelism shifts from “why are we doing EDA?” over to “check out these events available within our enterprise, how can your group use them?”. An event portal plays a crucial role in that.

The remaining steps in the methodology ensure that your enterprise captures the benefits of event-based integration, while minimizing the uncertainly, resistance and pain associated with any large architectural shift.

Step 2. Understand your event-driven integration foundation

While nothing is more exciting than the possibilities of event-driven integration, bringing the concepts of EDA into reality take a firm foundation. While there are multiple options for EDA, this paper assumes that you’ve decided to use the Boomi and Solace product stack. It’s a potent combination. But getting the most of your foundation requires the strengths of each of the components.

Boomi Components:

	Integration: Drag and drop UI, data mapping tools, and a comprehensive library of connectors. Coupling this with support for various integration patterns enables you to build any integrations with exceptional speed.
	API Management: Supporting the full lifecycle of synchronous APIs in a hybrid environment. First, configure synchronous APIs and expose their functionality to both internal and external clients, then test and deploy synchronous APIs and enforce contracts and policies with an API gateway.
	Flow: A low code platform for apps that allows everyone to build workflow applications, without needing to be a software developer. Simply drag and drop your workflow as a flow diagram, and the Boomi Flow platform converts this into a fully functioning workflow application.
	Master Data Hub: A cloud-native master data management (MDM) solution that sits at the center of the various data silos within your business and provides easy to implement, scalable, flexible, and secure master data management hub as a service.

Solace Components

	Solace PubSub+ Event Broker: Modern messaging layer that can be deployed across every environment (appliance, on-premises software, cloud-based software, messaging-as-a-service) and component of the distributed enterprise. Event Mesh: A collection of event brokers in different physical locations, woven together that dynamically routes events from one application or connected device to any other— no matter where they are deployed. Whether it’s in legacy on-premises systems or in the public cloud, private cloud, or somewhere in between, an event mesh takes the guesswork out of getting events where they need to be.
	Event Portal: Event management toolset that enables you to create, design, share, and manage various aspects of your event-driven architecture. Event Portal brings to the async world the same functionality that lets enterprises create API-first architectures.
	Cloud management: PubSub+ Event Broker: Cloud is a software-as-a-service (SaaS) platform that gives you access to advanced features like the publish-subscribe messaging pattern, queuing, streaming, replay, and request-reply, while doing  all the heavy lifting to dynamically deploy, upgrade, patch, and operate your event broker services. It is engineered to be secure, resilient, and cost-optimized so you can focus on building applications.

These components interact to create a complete ecosystem for event-driven interaction:

At the heart of the Boomi-Solace relationship is the PubSub+ Connector for Boomi. This connector lives inside the Boomi Integration GUI, both establishing runtime connectivity and letting you carry out common Solace tasks like discovering event definitions without ever leaving the friendly confines of Boomi. This integration creates a better experience for developers and keeps your EDA design vision on track. As part of the PubSub+ Connector for Boomi, there are multiple sample processes available in the Integration Process Library, which in turn link to extensive walk-through documentation. We’ll talk much more about the connector and process library later in Step 5 where we make our designs a reality.

The Basics of Events

While we are getting acquainted with the Solace and Boomi components, it might be helpful to review the terminology of event-driven integration. We’ll illustrate using a scenario where a fictional taxicab company hires a new driver through the SaaS provider Bamboo. In turn there are three downstream applications that are all interested in a new hire: Slack, Salesforce and ServiceNow.

_{Note: there are many ways you can implement an event-driven architecture with Solace. However, the typical scenario and best practice is: publish to a topic, read from a queue. Some of us have tattoos that say exactly that. So, to simplify things, that’s what we’ll describe here. There’s much more in the documentation about other use cases.}

Event: an event is a business moment where something changes somewhere in the enterprise. Here, the event is the taxi company hiring the driver. The key information about the hiring event is put into the payload (or content) of the event. For instance, the driver’s name, social security number, and address would be in the payload in a JSON or XML format.

Topic: a topic is a hierarchal string that briefly describes the content of the event to make routing more efficient. The topic is meta-information, separate from payload. The format of the string depends on the organization. The topic for this message is: /taxico/hr/bamboo/Driver/hired/v1/349034

Which can be broken down like this:

• taxico: Name of the company
• hr: Division within the company
• Bamboo: Application that created the event
• Driver: what the event is about (the noun)
• Hired: what happen (the verb)
• V1: the version of the event
• 349034: The person ID for the driver

Subscription: downstream applications indicate what events they are interested in using subscriptions. These subscriptions follow the same format at the topic string, but the true power comes in being able to use wildcards. So if Slack wants to know about all drivers that are hired, it can subscribe to the topic string: /taxico/hr/bamboo/Driver/hired/v1/* where * is the wildcard for “all”

And if SalesForce needs to know about drivers that are both hired, fired and promoted, it can use the subscription: /taxico/hr/bamboo/Driver/*/v1/*

Queue: When the event broker finds a subscription that matches the topic string of the incoming event, it creates a copy and places it on the queue. The queue holds messages until the downstream application is ready to process them. Some remarkable things about queues: they guarantee order, they preserve messages if there is an issue with the application and they serve as a buffer if there is heavy volume for a period of time and the application can’t handle it.

There might be many matching subscriptions, in which case all associated queues get a copy of the message, which makes publishing to a topic and reading from a queue a terrific way to spread information from one source to multiple interested consumers.

Step 3. Speed Date Pilot Candidates and Pick the Lucky Winner

No matter how great your foundation, biting off more than you can chew is sure way to sink a promising event-driven integration project. The key is a juicy pilot project that shows off event-driven integration without a big commitment. But with most enterprises featuring hundreds if not thousands of integrations, how do you pick a winner?

If your organization already has Boomi and wants to expand into event-driven integration with Solace…

One excellent feature of Boomi is that you can view deployed processes at any stage of the lifecycle through the “Deployments” view of the UI. This can give you a sense of how your organization is using Boomi, and which processes would be good candidates for Solace. These are signs that your Boomi process would be a strong candidate for introducing Solace for event-driven integration:

• Information incoming from a sole source using the process route node is distributed to multiple endpoints
• Long running processes
• Challenging error handling
• Processes that can use eventual consistency rather than strong consistency
• Processes that leverage multiple clouds or have cloud-based applications that need to access onsite resources
• Uses Internet of Things (IoT) or mobile phone applications
• Process where volume significantly varies over the course of a day or a week or is seasonal
• It’s using low latency mode to improve performance
• Already use messaging, or can use messaging technologies like JMS

If your organization already has Solace and wants to enhance your event-driven architecture with Boomi’s capabilities…

If you’re using Solace PubSub+ Event Portal, then you have an excellent catalog of applications and events moving through your enterprise. Use this as a starting point to locate good candidates event flows. These are signs that Boomi could enhance your existing Solace event integration:

• Data transformation requirements that are either currently unmet or addressed in an adhoc fashion
• Desire to provide better context for an event using data enrichment from a lookup table, in-memory cache or other techniques
• Want to dig into the payload of the event to route more efficiently
• Splitting single events into multiple events with different structures to better pinpoint information
• Need API Management for synchronous interactions
• Using custom coding to connect to databases, legacy applications, flat files and more.
• Need MDM capabilities
• Need a low code app

Applications featuring these characteristics (the more the better) should form a short list of potential candidates from which to choose. From there, winnow down further. The best event-driven integration pilot projects deliver a potent combination of:

• High business value, ideally through an improved customer experience
• Involves real-time data
• Rapid time from design to development to production
• Addresses a common use case within your organization (e.g. information coming out of Salesforce, distributed amongst multiple consumers)
• Resides in the happy medium between insanely complex and boringly easy

A quick win that delivers an improved customer experience can breathe optimism into both technical and business teams, relieve anxious stakeholders and provide momentum for additional projects. If you can simultaneously make developers and support personnel’s lives better by alleviating obvious technical pain (brittleness, performance, resiliency) so much the better.

Step 4. Event-Driven Design

Now that you’ve landed on a single project for a pilot, it’s time to roll up your sleeves and start designing. As mentioned at the beginning, a key part of the event-driven methodology is an API-first philosophy. Prior to PubSub+ Event Portal, there was no API gateway equivalent for events. With the introduction of our event portal, the designs you create here will serve to document your architecture, but also serve as the raw material for the PubSub+ connector as you implement your design in the next section using the Boomi platform and Solace PubSub+ Platform.

Solace PubSub+ Event Portal contains three main entities: events, schemas, and applications. When developing event-driven integrations with Boomi, you can think of the components like this:

	An application represents a piece of software that produces and consumes events. For us, that will be a Boomi Integration Process or an external entity like Salesforce or Bamboo. Applications connect to the event broker in an event-driven architecture and communicate with other applications via events.
	An event represents a business moment that can be communicated to interested applications. An application (here a Boomi process or external application) produces an event, containing a payload that matches a schema on a specific topic.
	A schema describes the payload of an event. Producers and consumers of an event can trust that the payload of the event matches the schema definition assigned to that event. Schemas define a type of payload through JSON, XML, Binary, or Text. JSON and XML schemas have content that describes each property of the schema. The content is either in JSON Schema format or XSD/DTD format.

Going from pilot project choice to a full-fledged design means knowing the details of your applications, events, and schemas. That means asking a lot of questions. In some ways it’s like a high school journalism class: figuring out the 5Ws+H (who, what, when, where, why, and how) of the events flowing through the enterprise. Equally important is documenting all the answers in a repeatable way that developers can use our design-time processes later.

Each of the key questions below flesh out some aspect of the applications, events and schemas. We can then enter them into PubSub+ Event Portal.

What information is available from the source application and what does it look like?

The first step is figuring out what information from the source application is available that could be used by the rest of your enterprise. A schema represents what that data looks like. If you’re using a SaaS or other commercial product, there is a well-defined schema available that you can find in its documentation. If it’s an internal application, you might need to dig a little more to find or create a schema representation. Event Portal catalogues schemas for easy access, and allows Boomi to import them when designing processes:

How can the source application send data?

This is also known as the protocol. Luckily for us, Solace and Boomi support many protocols—from event-driven protocols like JMS and MQTT, to REST over HTTP, to application specific ones like database connections with JDBC or just reading from a flat file. The protocol is key for determining the structure of your Boomi processes and picking the right entry from the process library, which will be addressed shortly.

What destination applications are interested in the data?

Flexible, one-to-many event distribution is a key benefit of event-driven integration. That means Solace can distribute an event arriving from a single information source to many interested consuming applications. Each of those applications can be defined within Event Portal.

What do destination applications need the payload to look like?

Just because an event starts with a certain data format, doesn’t mean that consuming systems can accept it in that format. That’s especially true for commercial SaaS applications. The first step is adding the destination schema to the Event Portal. This catalogues it and makes it available to the PubSub+ connector at design time. At which point, Boomi’s graphical data transformation tools can convert the data as required.

When are destination applications interested in data?

By using wildcards wisely, the flow of events to a consumer isn’t all or nothing. Some consumers may only be interested in a geographic region. Others may only care about events involving a large amount of money. Fortunately, with event-driven architecture, the event broker can handle the details of event distribution. This feeds into topic string design. As you start your EDA journey, it’s important to pay attention to topic taxonomy – setting up a topic naming convention early on and governing it. A solid taxonomy is probably the most important design investment you will make, so best not to take short cuts here. Having a good taxonomy will help with event routing, and the conventions will become obvious to application developers. For expert advice on this, refer to my colleague Ken Barr’s post on topic best practices. Event Portal calls the combination of a schema and a topic an event. As you input them into Event Portal, you start to build a catalog of events that you can use in this pilot and in future projects.

Boomi’s graphical tooling often comes into play here, as it lets you extract information from the payload and create a topic string with it.

How can destination applications receive data?

Again, this is the protocol, but this time it is the destination protocol that the consuming application will use. The combination of Boomi and Solace gives you a wide array of supported protocols.

How should we structure the Boomi processes?

One guiding principle: decouple information sources from information consumers.

What does decoupling look like for a Boomi implementation?

A traditional Boomi process receives information, then uses the branch or process route shape to interact with multiple information consumers. Moving to decoupling unbinds information receivers from information senders, with separate Boomi processes for both. PubSub+ Event Bbroker distributes information between them.

Why is decoupling so important?

Decoupling allows you to reap the benefits of event-driven architecture you evangelized in Step 1, including simplified error handling, preserving data in the case of disaster, allowing free movement of data from on-premises to the cloud, and rapid deployment of innovative solutions. While decoupling is the key, it can take several different forms, depending on what protocol your source and destination applications can use. If it’s an event-driven protocol like JMS or MQTT, then Solace might be the right choice to interact directly within the information source.

In this architecture, an information provider – which could be an internet of things device like a sensor – interacts directly with the event broker. The event broker routes the incoming event and places copies of it on queues for individual Boomi sender processes. The queuing insulates the Boomi processes from sudden rushes of traffic; if a downstream system can’t handle an event at the time, the event broker will hold it. And because each Boomi sender process gets its own copy of the message, it can 1) complete independently, without a dependency on other processes, and 2) implement business logic that is solely related to the end consumer. So, the Boomi sender for Salesforce logic is completely isolated from the logic for ServiceNow. If it’s not an event-driven protocol, then using Boomi to interact directly with the information source is probably the right call.

If the information provider doesn’t use an event-driven protocol like JMS, then connectors within Boomi can liberate information in legacy protocols into events by grabbing the incoming payload, determining the correct topic string, and publishing the event to Solace. In this case the information provider interacts with Boomi first, then Solace.

What does the overall flow of events look like in Event Portal?

Finally, you’ve completed the design work. The reward is a top down view through Event Portal of how your event-driven integration will work. That view is invaluable for communicating your vision for the pilot out to the team and re-evangelizing and re-educating your enterprise for your next project.

Instead of the haphazard Visio diagram that lives only on the architect’s laptop and is likely outdated, we now have a living, visual document that developers, architects and support personnel can access.

An outdated Visio diagram crammed with information

A standardized Event Portal diagram

What’s more, we can now dive into each of the entities to get more information about the schemas, applications, and events that make up our architecture. For example, we can see what schema, topic string, and applications interact with the ResourceHired event that was discussed earlier on.

And as we turn to implementation in the next section, the PubSub+ Connector for Boomi seamlessly brings this information into the Boomi UI.

Step 5. Implement a Quick Win

Now that the Async-API first design completed, it’s time to implement the design. Event Portal work continues to be important; you’ll be using it to import event definitions that ensure that the topic strings and data formats that are used match the design. But now the toolset expands to the Cloud Management capabilities of Solace, the Integration component of the Boomi Platform, and the crucial the Solace PubSub+ Connector, which allows them to integrate.

Note, the implementation and configuration instructions here are intentionally high-level. You can get much more information after importing an appropriate process from the process library, which will guide you step-by-step through the process or from more technical blog posts from Mike Hilmen.

Match your decomposed processes with process libraries

To help you get started, Solace and Boomi created a process library leveraging the PubSub+ connector and Boomi Integration. These processes bring together best practices developed through years of shared experience between Solace and Boomi—all you do is fill in the blanks and enhance with your business logic.

Many of these processes feature key technical aspects specifically aimed to meet the business objectives (Step 1) that made event-driven integration appealing in the first place. The key is to match the use case of your application to a use case in the process library.

Step through the code lab configuration

For each process in the Process Library, in addition to the skeleton code, there is extensive documentation in the form of a Google Code Lab. The documentation shows how to configure the Connector, Connector Operation, and the process itself.

Configure the PubSub+ Connector

This is where the design-first work you did in the earlier step comes into play. The PubSub+ Connector for Boomi integrates with Event Portal, which gives visibility into your event-based architecture. After picking an event, the connector imports the schema and the format of the topic string, then generates a recommended input queue name based on the name of the connector operation.

Round out business logic using Boomi’s integration capabilities

Even though the process library covers many use cases, you’ll still need to enhance the processes to make them your own. That could include custom data mapping, enriching the event with information from a flat file or database, or running business rules against them.

Create the underlying Solace Infrastructure

Now that you’ve created the Boomi process, it’s time to create the underlying Solace infrastructure event services, the queues that your applications will read off, and the subscriptions that will attract messages to the queues. You can create all of these using the Cloud Management console. Again, the code lab guides you through each click you need to create the objects

Step 6. Repeat

With the pilot, as you define events and refine the flow of events through your processes, your Event Portal event catalog also takes shape. The initial event catalog serves as a starting point of reusable events which future applications can consume off the event broker queues. Here, the cycle begins again: searching for additional business uses cases that could benefit from event-driven integration. Only this time, you are ready with some lessons learned from the pilot and a handful of events already available on your mesh.

Conclusion – Why Use PubSub+ Connector for Boomi?

To briefly recap: why would you use the PubSub+ Connector for Boomi? The main reason is that it guides you towards repeatable best-practices to help you get the most out of your event-driven integration investment. It serves as the glue in a combined Boomi/Solace environment, making your design-first artifacts living, breathing entities that your developers can utilize as they make life better for your customers.