Using PubSub+ Event Portal for Support and Visualization of your Kafka Streams

One of the persistent challenges with a distributed architecture—like with Kafka—is the ability to design your streams for reuse and supportability. This is especially true for large enterprises, where you may have hundreds, if not thousands, of streams, publishers, and consumers interacting with each other. In this post I’ll briefly introduce PubSub+ Event Portal and explore several features that help streamline the visualization, development, and support of your Kafka streams.

PubSub+ Event Portal

PubSub+ Event Portal is a revolutionary event management tool to design, create, catalog, discover, share, visualize, secure, and manage your enterprise’s entire event-driven ecosystem.

For more information about PubSub+ Event Portal, read Solace Documentation.

In the next sections, I will touch on a few of the salient aspects of the Event Portal which make it a great tool for application development and support.

Event Designer: Improved Application Visibility

Visibility is key to robust distributed applications. It allows you to easily get answers to questions like:

• Who are the publishers and consumer groups?
• Which topic(s) are they connecting across?
• What data is being exchanged, and in what format?

Event Designer, a tool in the Event Portal, significantly enhances visibility for support engineers and developers with microservices visualization – a feature which lets them visualize which microservices are members in Kafka event streams as well as the events and schemas (Avro, JSON, XML, …) they are exchanging across topic(s).

From the developer’s perspective, they can now build their microservice with a fuller understanding of how it plays into the broader enterprise ecosystem. Similarly, support engineers can immediately narrow their focus to only the systems that may be affected by a misbehaving service.

Another key aspect of Event Designer is its ability to enumerate events and schemas. Since the portal validates schemas during design time, developers can be certain they are publishing data that is structurally consistent, which then significantly reduces errors related to schema mismatches between publishers and consumers.

Event Catalog: Event and Application Searchability

Event Catalog, another Event Portal tool, provides a searchable index of all event-related objects within an enterprise.

For developers, this increases opportunities for reuse since they need only search for an object (event, topic, schema, application) of interest to determine what is already available for use. From a support perspective, Event Catalog again lets engineers quickly sift out related objects during a troubleshooting exercise.

AsyncAPI Code Generation: Application Consistency

A final feature that enables consistency from design through to development and support is the Event Portal’s code generation capabilities. Once architects complete their modeling in Event Designer, developers can export an AsyncAPI spec of the model to any number of code generators. The resulting code stubs use the schemas, topics, and names as envisioned by the enterprise architect, leaving the developer to focus on core business logic. This level of deep consistency greatly reduces the number of elements that the application support staff needs to consider when doing root cause analysis.

For example, a developer could pass the model through a Spring Cloud Streams generator and apply a Kafka binder to the stream. This results in a publisher/consumer in far fewer steps—and with potentially greater design integrity—than writing the code from scratch.

Conclusion

PubSub+ Event Portal is truly disruptive. It offers architects, developers, and support engineers an unprecedented level of control when creating distributed applications that are better designed for reuse and supportability across the enterprise. Its features are ideal for Kafka streams visualization and development. You can get started with a 30-day free trial by creating an account.