Event-Driven Architecture Demands Event Lifecycle Management

Event-driven architecture (EDA) has become extremely popular in the last several years, and is being adopted in processes that have traditionally been batch oriented, across many industry verticals. This improves the ability to react to changes in real-time and improves customer experience, while enabling better corporate agility and reducing costs.

In working with many of these companies, I’ve noticed that they generally have a good experience when applying EDA to their first use case, but run into challenges when they aim to:

1. Add additional use-cases to the ecosystem, and
2. Try to extend or reuse events that are already in use.

Both of these challenges are caused by the lack of a well-defined lifecycle model for events, and associated tooling.

That’s why I’d like to introduce a lifecycle model that can lend some structure and address these challenges with events. I’ll identify the problem and solution in each step and demonstrate how this model and tooling can help you be much more successful as you apply EDA to more and more use cases.

Model for Lifecycle Managing Events 

At a high level, managing the lifecycle of events can be categorized into four main areas that follow the sequence of creating events and deriving business value from them: Discover, Define, Develop and Operate.

Discover

• Event Streams: Search and locate events which are of interest
• Relationships: Understand the relationship between events, their sources and who is consuming them

Define

• Use Case: Determine desired business outcome
• Schema: Create business objects that represent the event context
• Event: Construct event metadata including addressing (topic) and headers
• Specify Application: link events as inputs and outputs to the business logic

Develop

• Generate code: Use code generators to create application “scaffolding” based on the application specification
• Add business logic: Create application which performs the required function

Operate

• Secure: Enforce authentication and authorization policies
• Deploy: Start and connect to the runtime event broker
• Audit: Determine if there are any runtime vs. design time violations
• Monitor: Understand the utilization of events, schemas and applications based on historical and real-time metrics
• Improve: Enhance applications, events and schemas; deprecate events that aren’t being utilized

In addition to these three core areas, the diagram also hints at some more functionally specific aspects of event lifecycle management: discovery, collaboration, development and informing. In the next section I’ll describe the core challenge you’ll face in each area, and how Solace aims to help.

The Four Stages of Event Lifecycle Management 

Discovery 

If your organization is already hooked on the idea of becoming event driven, congratulations! You’re given a business problem to solve and are looking to leverage some of the events which you think already exist. Where do you go to find them? Many organizations rely on Excel spreadsheets or homegrown databases that tend to be out of date, and that’s a big problem. If the developers of events can’t register their availability, discover them and use them, you’re looking at a disconnected mess of event-driven use cases, not an event-driven enterprise.

What’s the solution? You need a catalog that stores event definitions and their associated payload schemas, and make them available for discovery. This catalog must support keyword searches, and needs to support version control because in the event-driven world change is inevitable. I’ve gone down the road of building such a system several times in my career, and trust me it’s not as easy as it sounds. And without the insights you can get from metrics generated by the runtime brokers, its going to get out of date AND not have the full picture. Luckily, Solace is launching its PubSub+ Event Portal this fall so you can discover all of your events and get value out of their reuse.

Design and Collaboration 

Once you’ve identified some event driven use cases and come up with a way to document and discover events, it’s time to get cracking. There are some key questions you’ll run into right away though: Are the data schemas existing and reusable? Do you need to specify new event definitions or leverage existing? What are the inputs and outputs of the new application? Are multiple applications required, thus forming event driven choreography?

Each of these questions must be answered though organizational collaboration. Data architects specialize in defining the payload schemas (JSON schema, Avro schema, XML schema, Text or Blobs), while application architects and enterprise architects excel at understanding the use case and creating a design that meets requirements – and success requires collaboration between all three. What if the payload schema is missing fields, or the application consuming events only needs a subset? This feedback loop must occur in a timely manner to ensure timelines are met.

Organizations need to collaborate though tooling built with this in mind. The Solace PubSub+ Event Portal’s designer was conceived though feedback from many of our customers and their experiences bringing real-time, event driven solutions to life. What they’ve told us is that the ability to keep the question, answers and comments provides context. It’s like using Stack Overflow but tailored for events. The questions and comments are just as important as the solution and enabling this communication and tracking it is really the gift that keeps on giving.

Development 

Once you’ve defined your use case and created an event-driven design for it, it’s time to implement. As it happens, writing an event driven application is not as easy as you would think. In fact until now it’s been a bit of a black art –not nearly as easy as the synchronous API world with its shiny frameworks, code generation tools and simple programming model. You don’t want to have to make your developers ninjas in Paho for MQTT, the JMS API or any number of other event driven APIs. Rather, you want them to get busy so the organization can realize the benefits of being event driven.

So wouldn’t it be nice if you could export the event driven application you defined and generate code, just like Swagger/OpenAPI lets you do in the synchronous RESTful world? Solace PubSub+ Event Designer will allow you to export the AsyncAPI spec definition of your application, which you can run though code generators to generate your application scaffolding and associated event driven API code. This means your development team can focus on writing the code, using the language they are the most comfortable with and create business value instead of taking a bunch of time to become an expert in an event driven API.

Once the code wizards are done, and it’s time to get real (and real-time !) Once you push the application to test/QA/Prod… wherever…what next? Are the new events your application is producing being consumed? Who is consuming them? Is your sensitive data being protected? Can you audit who consumed each event and when? Do you know if the rate of events is increasing, decreasing? What are the most and least consumed events?

While the delivery of events between applications is well handled by runtime event brokers, you need information from that runtime layer to make decisions and optimizations. In fact, without answers to these questions you aren’t agile, and things can get really messy from an auditing and security perspective.

To help you keep the event-driven train on the rails, Solace and its PubSub+ Event Platform deliver all kinds of data about what is happening with your events, and in your business. By defining your event driven application landscape in a controlled and governed way, the Event Portal can create access control lists (ACLs) which enable you to easily implement the principal of least access. That in and of itself is a huge win – now imagine where you can see real-time metrics about your event driven applications, associated choreographies and understand where runtime problems and bottlenecks exist. All of this will be a reality with the Solace PubSub+ Platform.

Conclusion 

I hope I’ve helped you understand that the road to being an event-driven organization is not as easy as writing an application and installing an event broker; but, there are simple steps you can take proven to succeed. You just need the tooling and “guardrails” to keep you firmly on the event driven road. Solace is committed to providing you a platform that ensures success so you can experience the advantages EDA has to offer. Let’s go on this journey together!