Why You Need More than a Schema Registry to Become an Event-Driven Enterprise

In this post I’ll introduce some challenges associated with the enterprise-scale implementation of event-driven architecture, and break down the reasons you need more than a schema registry to overcome them. I’ll wrap things up by letting you know what you do need to successfully event-enable your enterprise and IT infrastructure.

Background

In today’s rapidly evolving technology landscape, event-driven architecture and streaming have shown their value in their ability to reach customers in real-time. To succeed, all businesses must react to and predict the needs of their customers, which means constantly creating new applications and capabilities.

The ability to alert and inform applications via the publish/subscribe distribution of events is well established, but many enterprises struggle with making events easily reusable and governable across their entire organization and partner ecosystem. API paradigms like REST are supported by tooling that enables the advertising of service capabilities, but the event broker/messaging world is not. That shortcoming is what inspired Amazon to introduce EventBridge, and Confluent to roll out Schema Registry. I’ll start by explaining the role these new tools serve.

AWS EventBridge and Confluent Schema Registry

Let’s say you’re an architect or a developer (pretty good chance if you’ve gotten this far…) and you know your organization is using AWS and/or Kafka. You’ve been tasked with creating a new type of analytics application and need to feed it a bunch of data.

Amazon EventBridge and Confluent Schema Registry enable you to discover the events/messages that exist within your organization and understand the payload schemas so you can A) understand if they include information your app needs, and B) create business logic that extracts those data elements. Without this “schema” payload contract, how would you know the format of the data? This is the primary paradigm for which these tools were created. Sounds pretty useful right?

There’s no doubt that the ability to auto-discover events and let developers and architects review a “menu” of what’s available is useful, but it’s not the end-all be-all. You still need to factor for the following:

1. The requirement to govern the behavior and usage of events
2. The need to support a wide variety of payload schemas
3. The ability to collaborate on the design and deployment of events

I’ll explore each of these areas.

Three Reasons You Need More Than a Schema Registry

Security and Governance – Are You Behind the Curve?

First, let’s look at governance – the ability to control and manage the lifecycle of schemas. The management of schemas and their corresponding events isn’t just about being able to check forward and backwards compatibility – although that’s important – but being able to understand the lineage of the data, control runtime access to it, and understand how it’s being used.

Midsize and large organizations are constantly dealing with security control requirements and emerging regulations in the areas of data privacy and tracking. I am looking at you, GDPR! It’s important to govern applications and their associated interfaces (such as asynchronous publish/subscribe) because they impact the schema and event definition and context. If you can’t manage and govern the application lifecycle because you only have a schema registry, you’ll be behind the curve of gaining control and ensuring continuity when applications and application designers make changes. Simply put, it’s really important that you are able to govern all architectural components: applications, events and payload schemas.

Payload Schema Format – Betcha Can’t Eat Just One!

Have you ever worked on a system that only has one payload schema format? Neither have I. You have some applications still using custom text and binary formats, others were created during the heyday of SOA using XML, then there’s more modern applications that use JSON. Other teams, depending on their technology stack and use cases, may be using Protobuf, Avro… who knows what else!

Uniformity of schema language and serialization/deserialization formats would be great, but that’s a pipe dream for any organization, let alone a large enterprise. Do you really want to run Amazon EventBridge Schema Registry for OpenAPI described schemas (subset of JSON Schema format) and Confluent Schema Registry for your Kafka and Avro schemas? What about your other schemas?

This is the world we find ourselves in today – cloud and software vendors are offering tools to support their ecosystems instead of providing a holistic solution for the management of ALL payload schemas.

Managing Interactions and Dependencies – Beware the Ripple Effect

The advantages of loosely coupling applications can go up in smoke if you lose track of how they’re interacting. It’s really important to be able to understand application dependencies, both for change analysis and to know which application is producing which event and why. I frequently see people design event-driven architectures using PowerPoint and Visio diagrams. They work… until someone changes something! Each change causes a ripple effect and soon no person, diagram, or tool knows the actual state of the ecosystem. That’s both scary and inefficient.

You Need More Than a Schema Registry – Event Portal to the Rescue

Imagine a world in which you design your event-driven applications, events, and schemas in one tool. Those designs are automatically visualized into a set of interconnected network diagrams your team can go over in design reviews. When you deploy to production it’s easy to see if the design is in sync with the reality in runtime, and changes are all version controlled and tracked. Sounds nice, right?

We at Solace are bringing to market a product we call PubSub+ Event Portal that does all that to meet the needs I described above. We see events being federated between cloud providers, into and out of Kafka clusters and into “legacy” messaging brokers like IBM MQ every day, and we’ve got the chops to make it way easier to manage such a system.

Our PubSub+ Event Broker helps organizations create an event mesh that streams events between applications, connected devices, and people anywhere in the world. We are extending that platform and vision with Event Portal (which will integrate with schema registries to retrieve and push schema definitions) so you can do all your governance, discovery, and design centrally and never lose control of your applications, events, or schemas again!

Be sure to check out our PubSub+ Event Portal Early Access Demo below.