If your company has started to embrace event-driven architecture (EDA), you’ve probably been considering or struggling with how to manage your “events” with some semblance of governance, reuse and visibility. If that’s you, then read on!
In 2020, Solace introduced PubSub+ Event Portal to make it easier for enterprises to efficiently visualize, model, govern and share events across their organization. Why?
While organizations are embracing EDA, many are finding it hard to realize the intended benefits of better customer experiences, more efficient operations, and greater agility due to a lack of tools that help them manage the lifecycle of events, integrate with software development tools, and expose event flows for reuse.
Forrester advises that organizations should “implement governance and lifecycle management for event streams. As they grow in importance, you can include enterprise business events in the portfolio management you should already have in place for APIs. Establish an event portal where users of event streams can discover events, understand their functionality, and subscribe.”(1)
If you agree that there is a need for a toolset that helps you manage events, the next question is – what should you look for in such a toolset? And which one should you use?
I caught up with Shawn McAllister, Solace’s chief technology and product officer, to get his take on what enterprises should look for when they look for a tool to manage their events, and how our own offering – PubSub+ Event Portal – meets them and compares to alternatives like Confluent’s Stream Governance product.
Shawn: First of all, it’s not about hosting a conference or a wedding. Event management is like API management for your event-driven system. Many companies are building event-driven microservices and/or integrating real-time data. To achieve either of these significant initiatives, there are certain things architects and developers need to be able to do. One is to have an event broker to route your events where they need to go as they stream across your enterprise in real-time. That need has been met by event brokers like Apache Kafka and our own PubSub+ Event Broker, but there’s been a lack of tools to manage your events like you do your RESTful APIs.
Without such tooling, there’s no way for your architects and developers to collaborate, making it hard to find events to reuse them, to know what events are available where, to keep documentation up to date, and to make complex model changes without impacting upstream and/or downstream applications. There’s no way to manage the lifecycle of events and other assets that are part of an event-driven system. Having a solution to this problem is a critical factor that made the API economy so successful and filling that gap that existed in 2020 for the event driven world is what we set out to do and we call it event management.PubSub+ Event Portal for Apache KafkaManage, visualize, and catalog your Kafka Event Streams with PubSub+ Event Portal for Kafka.
We believe that an event management tool must accelerate and simplify the entire event lifecycle, from design and development right through operation and retirement. It’s innovative, disruptive, and customer-centric, and it’s the right thing to do to ensure EDA can become mainstream!
There are some core functions we think every event management tool should do. Get ready, it’s a long list:
It was with those capabilities in mind that we created PubSub+ Event Portal, which we launched in 2020.
Event Portal can do most of them today and will be able to do all of them by early next year. It’s always been our vision to deliver a product that provides all this functionality, and since it’s been on the market for two years we’ve learned a lot from our customers on how it can be improved and what else they need it to do.
While I am not an expert on Stream Governance, I’d say that while at first glance they seem to serve a similar purpose, in reality they are quite different. Stream Governance only works with Confluent Cloud, and really only adds value in the area of understanding your live operational environment. Its catalog interrogates a particular Confluent Cloud environment to show you what’s happening in its clusters at that time – which is after an application has been designed, developed and deployed – at a stream and stream processing level.
This is useful for operational reasons once an application or a stream is deployed, but it doesn’t help developers in creating or modifying existing applications or streams, guiding best practices, helping with governance at design time and does not provide a “design vs actual” comparison to ensure what you have deployed is what was intended. This is all because it provides visibility of what is deployed – not what you are designing. It is tightly integrated with Confluent Cloud and combines other functionality to provide runtime lineage and data quality capabilities – again both are operationally focused.
Event Portal, on the other hand, helps architects and developers across all phases of the software development lifecycle, not just after deployment, and is designed to support many types of brokers so you don’t need to have a different tool for each event broker you use. We view Event Portal as the central point that helps you design, discover, catalog, manage, govern, visualize and share your event driven assets. It has specific functionality to help you build event driven microservices applications and to define event driven interfaces/APIs for real-time data integration, data meshes and analytics.
Event Portal works not only with Solace brokers, but with Apache Kafka, including Confluent and MSK brokers. It discovers and imports topics, schemas and consumer groups from these brokers. The open source Event Management Agent, a component of Event Portal, has a pluggable architecture so you can add your own plugin to perform discovery from other brokers too.
So, I say use the right tool for the right job, – these tools serve a complementary purpose. Operational visibility into your Confluent Cloud cluster stream architecture is best done with Stream Governance. But if you want a tool to support cradle to grave design time management of your event driven assets across environments and with different types of event brokers, then Event Portal is what you want. If you are a Confluent Cloud user, then maybe you want both.
In terms of design concerns, here are a few examples of what people want to do at design time in event-driven systems that they need tools for:
Here are some of the areas in which Stream Governance does not address design time concerns:
However, from an operational point of view, Stream Governance is better suited for managing schema registries, validating schemas at runtime, determining runtime data lineage and resolving operational issues involving the flow of streams in Confluent Cloud.
Lifecycle management is defined as the people, tools, and processes that oversee the life of a product/application/event from conception to end of life.
Most enterprise software development teams follow a software development lifecycle management process in delivering high value applications to their organization. We believe that event management must integrate with these development processes and tools to facilitate and provide added governance and visibility into EDA application development and deployment. As a result, new application and event versions can be created efficiently and without negative impacts to downstream applications.Supercharge your Software Development Lifecycle with PubSub+ Event PortalLearn how PubSub+ Event Portal can work with your existing SDLC tools and add crucial EDA perspective with our open source plugins.
This graphic shows the typical stages of development of an event driven application or an event driven API and Event Portal plays a role in each of these stages. From supporting the definition of streams, microservices and visualization of their interactions during the design phase, to versioning as you evolve artefacts on day 2, to environments and promotion between them, to audit of design intent vs what is actually deployed. Open-source integrations we have created integrate Event Portal with tools like Confluence for design, Intellij for development, Git for GitOps and you can use Event Portal’s API to integrate its catalog and repository with your own favorite developer tools using our open source integrations as examples.
In comparison, Stream Governance provides visibility into what is deployed in Confluent Cloud clusters – and it does give you great visibility of that – but not across the entire development lifecycle and not beyond Confluent Cloud deployments.
Event Portal and Stream Governance solve different problems in overlapping situations. Event Portal focuses on cradle-to-grave design, discovery, reuse, audit and lifecycle management of your event streams, schemas and applications across Solace brokers and Kafka brokers including Confluent Platform and Confluent Cloud along with Apache Kafka, MSK and others that can be added via the open-source Event Management Agent. Stream Governance focuses on visibility of your runtime streams in Confluent Cloud deployments. Which tool is best for you depends on the problem you are trying to solve.
I hope this Q&A with Shawn has helped you understand the difference between PubSub+ Event Portal and Stream Governance. You can learn more about Event Portal at https://solace.com/products/portal/kafka/