Got Kafkatosis? Stop Accepting the Status Quo of Kafka Event Streaming

If you’re like me, your annual physical isn’t high on your list of favorite things, especially as the aches and inconveniences associated with middle age start to pile up. But growing old can feel so inevitable that the intake forms turn into something like “yeah yeah, everything’s fine doc, I’m just getting old!” This is a good example of “normalcy bias” – a common cognitive behavior by which we underestimate the likelihood or impact of a potential hazard based on the belief that things will continue as they have in the past.

This bias makes it important for the doctor to suss out potentially problematic conditions by asking seemingly innocuous questions about how well you’re sleeping, and listening to your breathing and heartrate, and gradually asking more pointed questions based on what they learn, including some you’re not excited to answer, about things like your intimate relationships and visits to the bathroom.

Is “normalcy bias” masking problems with your Kafka estate?

Normalcy bias follows us to work, too — in your organization, I bet you’re using Kafka or some other broker for event/data streaming, and I bet if I simply asked about the health of your Kafka estate, you’d say: “Everything is fine, not perfect of course but lots of teams are using it, and we are getting to production.” As Lee Corso says on ESPN college gameday: “Not so fast my friend!” Would it surprise you to know things might actually not be as hunky dory as you think? In fact, you might have a condition we call “Kafkatosis.” Don’t be scared, because it’s easy to diagnose, and totally curable, but let me start by explaining what it is.

Definition of Kafkatosis

Kafkatosis
[ kron-ik kahf-kuh-toh-sis]
noun Pathology.

1. A potentially serious systemic condition that causes an enterprise’s Kafka
   estate to become increasingly difficult to control and scale over time.

Good News: Kafkatosis is Easy to Diagnose

Back to the metaphor of a checkup with your doctor, I’ve got some preliminary questions I’d like to ask so we can see if you might be at risk of Kafkatosis:

1. How many different distributions of Kafka does your company have, and how many clusters are running across your organization?
2. How many Kafka topics do you think your company has, and how sure are you that all of those topics have at least one consumer?
3. Does your organization have a schema registry in place?
4. Can members of your team check the impact of changes they make to apps or event streams on their own?
5. How easy is it for team members to find event streams that they can reuse?

With those question and that kind of thinking in mind, you might start to recognize some common symptoms, such as:

1. Users can’t find or understand real-time data/streams that are available for reuse
2. No self-service automated onboarding process for producers and consumers
3. Can’t make changes without impacting upstream or downstream applications
4. Complaints about stale or missing documentation

I bet one or more of these problems have cropped up, but you may not have recognized or sought to fix them because normalcy bias made you think that’s just the way it is, and they’re an inevitable part of dealing with events, Kafka, etc.

If so, you’ll be happy to hear that our Event Portal for Kafka product includes a discovery agent that can help you understand if and exactly how you’re at risk. It scans your system to ascertain within just minutes the size, scope and state of your Kafka event streams and infrastructure: how many clusters and streams you’ve got, how many have topics and schemas, of what kind (avro, json, primitives, etc.) how many are subscribed to — or not — and so on.

Last year my colleague Jesse Menning helped a prospective customer run such a scan on their system, and it revealed a shocking state of affairs. He documented the process, what they discovered, and how we’ve helped them clean things up, in this excellent blog post:

How a Financial Services Giant Cleaned Up their Kafka with PubSub+ Event PortalOne of the biggest wealth management banks in the United States scanned their Kafka environment using Event Portal and was shocked to find out that they over six thousand topics, including 552 that weren’t being utilized by any applications, and many that contained circular dependencies! That's when the clean-up started! Read the blog to find out how they got there.Read Now

Better News: We’ve Got the Cure for Kafkatosis

If you’re worried that you might be suffering from Kafkatosis, never fear: Event Portal for Apache Kafka can help you get things in shape by:

1. Discovering all your data streams and cataloging them in a way that lets the rest of your organization find and reuse them.
2. Giving developers a self-service way to more rapidly create new streaming applications and business capabilities, on par with how they work with APIs.
3. Ensuring that your catalog is kept up to date and representative of reality (aka the runtime)

That’s just a summary of a summary, of course…visit the product page to learn how Event Portal for Kafka can accelerate the design of Kafka event streams and event-driven applications, improve the governance and lifecycle management of those assets, and boost developer productivity by integrating Kafka development into your SDLC processes. If you’re a visual learning like me, be sure to watch this awesome 10-minute demo that Jesse put together.

Conclusion

I hope I’ve helped you realize that the Kafka status quo doesn’t have to be, and there’s a straightfoward way to make life way easier for the architects, developers and administrators who run your Kafka estate. If you’re ready to see if you’re at risk of Kafkatosis, let’s talk!