Governing Event-Driven Systems with Solace PubSub+ Event Portal

Dunith Dhanushka is a Solace Scholar and seasoned IT professional with over ten years of experience in architecting, building, and consulting large-scale event-driven systems by utilizing cloud-native microservices architectures.

Event-driven architecture can be seen as a double-edged sword. When implemented right, it can scale applications to virtually unlimited capacity while ensuring high throughput and more reliability. It also allows for new apps/services (for new use cases) to easily be integrated into the architecture to participate and extend overall solution set, by just subscribing to and leveraging the event flows that are already being published by other existing apps. Hence, event-driven architecture has been the architect’s first choice for building systems that demand high scalability, performance, and reliability.

But since event-driven architecture promotes a loosely-coupled, decentralized architecture with many moving parts, it can be challenging to manage the complexity of the application as it evolves. Especially, finding the ownership of events and applications, event schema evolution, architecture documentation, and dependency visualization becomes challenging as more components and people are added to a project.

Solace PubSub+ Event Portal addresses those problems by providing a central place to design, organize, and govern the event-driven architecture components around message brokers and event streaming platforms. This article takes a fictitious use case and walks you through the areas where Event Portal solves the above problems.

Use Case: Order Fulfillment

Picture a large-scale retailer with a growing customer base. They are in the process of implementing an e-commerce store to improve the overall customer experience. The implementation process takes a phased approach where different e-commerce use cases are on-boarded one-by-one.

E-commerce order fulfillment is a use case designed with event-driven architecture in mind. It will be implemented as a collection of event-driven microservices that collaborate by exchanging business events via Solace PubSub+ Event Broker as the central event broker.

Figure 01: High-level solutions architecture

John is the solution architect leading the design of the fulfillment use case. John is joined by Matt and Kayla, two developers who will be implementing the Orders and Shipping microservices, respectively.

Figure 02: The project team

Planning the Project

John is a seasoned architect who has been working on several large-scale event-driven projects. So, he precisely knows where event-driven architecture excels and needs more attention. John plans to follow a collaborate-design-build-iterate model to implement the fulfillment use case.

Simply put, that model envisions the following.

1. John collaborates with the business stakeholders to capture their expectations around the fulfillment use case. E.g., what information is included inside an order? How many orders have to be processed per hour?
2. John designs the high-level architecture of the use case to include the microservices, business events being exchanged, external dependencies, and the event broker environment.
3. Matt and Kayla collaborate with John over the high-level architecture. They refine it by adding microservice specific information, such as the events being published and subscribed by individual services and the schema of the payload of each event. Once all are satisfied, Matt and Kayla start working on the implementation.
4. Whenever John wants to make a design decision, John repeats the same process.

Figure 03: The delivery process

How can PubSub+ Event Portal help?

To facilitate the process described above, the project team decided to use Event Portal as a central place to design and document the fulfillment use case and a consistent way to communicate the changes in the architecture.

Event Portal is a web application that is a product within Solace PubSub+ Platform. It helps developers better manage their event-driven applications around event brokers. Event Portal serves both architects and developers at a very high level as a medium to visualize, discover, and catalog various components in event-driven applications.

Soon after the project is kicked off, John’s team got the accounts for Event Portal where they can log in, create and browse project artifacts, and collaborate while having appropriate levels of read and write permissions.

Throughout the latter part of this article, I’ll explain how John and the team leverage Event Portal’s capabilities to manage the fulfillment use case.

Step 1: Architectural design

As the first step of the process, John logs in to Event Portal and starts designing various components of the fulfillment architecture. For that, he uses the integrated Designer.

Application domains

An application domain is a bounded context for a particular business domain. John starts by creating the E-Commerce Fulfillment application domain to group all related project artifacts.

Figure 04: Application domain

Events and schemas

Once the application domain is created, the business events in the domain can be defined. An event contains a corresponding schema and version. Also, each event contains a topic address field which represents the topic that the event is published to by the applications at runtime.

John creates the OrderCreated event to represent the order information originating from the OrderUI web application. Similarly, the OrderConfirmed and ShipmentCreated events are also created.

Figure 05: Event definitions

Event Portal also allows for defining payload schemas for the events. In our use case, all events should be JSON formatted.

Figure 06: Schema definition for the OrderConfirmed event.

Applications

Applications represent the individual processes that publish or subscribe to the business events defined above. They can be web applications, APIs, microservices, or pretty much any application runtime.

In our use case, John defines two applications to represent the Orders services and the Shipping service. At this point, the applications include very high-level details. They will be completed when implementation-specific details are added.

Figure 07: The graph view visualizes the applications and their dependent events

Step 2: Building the Applications

Once the design is finalized, John invites his team members, Matt and Kayla, to review what he has done and refine the high-level design with implementation-specific details.

The implementation boils down to two Spring Boot microservices as follows.

1. OrderService: Consumes the OrderCreated event, validates the order, and produces either OrderConfirmed or OrderRejected events based on the result.
2. Shipping service: Consumes the OrderConfirmed event, creates a shipment for the order, and produces the ShipmentCreated event.

Matt owns the implementation of the OrderService, while Kayla owns the ShippingService.

Matt first logs in to Event Portal and uses the graph view to understand the inter-dependencies of the components. He also uses the event catalog to browse for events his application should subscribe to and publish to the event broker.

Once he understands the bigger picture of how everything works together, he configures the OrderService application.

1. Subscribe to the OrderCreated event.
2. Publish the OrderConfirmed event.

Up to this point, everything is in the design stage. Now Matt wants to take the design to the implementation level. How is that possible?

Event Portal allows generating the corresponding AsyncAPI specification for a given application. The generated AsyncAPI specification defines the event interface for the application. It contains the events and schema definitions, channels/topics to which the events are published, subscriptions by which the events are consumed, and other event broker details.

Figure 08: PubSub+ Event Portal allows downloading the AsyncAPI specification of a given application.

Matt then uses that specification file to generate the corresponding Spring Boot microservice. Once it is done, further business logic related to order validation can be added.

The tooling with the AsyncAPI specification increases the developer productivity by directly generating the backend service implementation. That eliminates the slow, error-prone manual coding. Currently, the code generation supports multiple languages, including Java, NodeJS, Python, etc.

In the meantime, Kayla also logs into the Portal and follows the same process as Matt did. She configures the ShippingService application to:

1. Subscribe to the OrderConfirmed event.
2. Publish the ShipmentCreated event.

Once the code for both services is generated, Matt and Kayla continue to add business logic and test them as needed.

Propagating Changes in the Architecture

The previous sections described how Event Portal could be instrumental in executing an event-driven project with a top-down approach: define the architecture, break it down into components, and implement them. So far, so good.

But enterprise systems can’t stay in their rigid forms forever. New business requirements come in, which expects the architecture to adapt with a minimum business impact. That is where most of the event-driven architecture projects start the downfall.

The ability to detect the downstream dependencies among components saves a lot of time when it comes to planning and executing architectural changes. The dependency visualization provided by event Portal helps detect dependencies among applications components and notify the teams who own those dependencies about the incoming changes.

Releasing a New Version of the OrderCreated Event

Let’s assume there’s a requirement to add a new field sales_region to the OrderCreated event. The business requirement is to calculate sales figures against each sales region.

John wants to update the existing event schema to accommodate this new field and communicate it to the applications who have subscribed to the OrderCreatedevent.

Fortunately, Event Portal serves as a one-stop shop for this. It allows John to create a new revision for the schema with the sales_region field and attach it to the event.

Figure 09: When editing an event, its schema can be changed.

Then, John finds out that the OrderService will be impacted by this change and informs Matt to use the latest version of OrderCreated in the OrderService.

Matt acknowledges the change request, applies the change and makes a new release.

Extending the Architecture: Adding the AnalyticsService

With the fulfillment use case completed and deployed in production, the business wants to measure the sales KPIs to get more insights and identify the areas that need further attention. John is asked to capture all business events and feed them into a BI system. How does he plan to do that?

As a seasoned architect, he thinks of reusing the current business events in the fulfillment use case. So he logs into Event Portal and discovers all the events using its event catalog.

Figure 10: The event catalog allows viewing and searching all the events, schemas, and applications from a single place.

Once events are identified, John creates a new application called AnalyticsService configures it to subscribe to the following events.

• OrderCreated
• OrderConfirmed
• ShipmentCreated

John assigns Anita as the owner for the analytics service, as she will own the service implementation in the future.

Conclusion

Hopefully this example has shed light on how PubSub+ Event Portal can help you design, collaborate, implement, and govern a large-scale event-driven system at scale.

Event Portal’s strength is the areas where event-driven architecture requires more manual, laborious tasks to document itself. For example, cataloging and versioning are crucial for clear ownership of events and well-documented payload schemas. The graph view will be helpful to identify dependencies among application components before making an architectural change. Also, having a user access level with proper role-based access controls (RBAC) allows team members to collaborate within the proper scope.

If you are planning an event-driven project around the Solace event broker, I highly recommend using Event Portal as the management plane.

About the Author

Dunith Dhanushka

Solace Scholar

Dunith Dhanushka is a seasoned IT professional and Solace Developer Community member with over ten years of experience in architecting, building, and consulting large-scale event-driven systems by utilizing cloud-native microservices architectures. He’s interested in exploring innovations in the enterprise data management space including streaming platforms, real-time analytics, and machine learning. Read more of his work on Medium.