If you’re like me, your annual physical isn’t high on your list of favorite things, especially as the aches and inconveniences associated with middle age start to pile up. But growing old can feel so inevitable that the intake forms turn into something like “yeah yeah, everything’s fine doc, I’m just getting old!” This is a good example of “normalcy bias” – a common cognitive behavior by which we underestimate the likelihood or impact of a potential hazard based on the belief that things will continue as they have in the past.
This bias makes it important for the doctor to suss out potentially problematic conditions by asking seemingly innocuous questions about how well you’re sleeping, and listening to your breathing and heartrate, and gradually asking more pointed questions based on what they learn, including some you’re not excited to answer, about things like your intimate relationships and visits to the bathroom.
Normalcy bias follows us to work, too — in your organization, I bet you’re using Kafka or some other broker for event/data streaming, and I bet if I simply asked about the health of your Kafka estate, you’d say: “Everything is fine, not perfect of course but lots of teams are using it, and we are getting to production.” As Lee Corso says on ESPN college gameday: “Not so fast my friend!” Would it surprise you to know things might actually not be as hunky dory as you think? In fact, you might have a condition we call “Kafkatosis.” Don’t be scared, because it’s easy to diagnose, and totally curable, but let me start by explaining what it is.
[ kron-ik kahf-kuh-toh-sis]
Back to the metaphor of a checkup with your doctor, I’ve got some preliminary questions I’d like to ask so we can see if you might be at risk of Kafkatosis:
With those question and that kind of thinking in mind, you might start to recognize some common symptoms, such as:
I bet one or more of these problems have cropped up, but you may not have recognized or sought to fix them because normalcy bias made you think that’s just the way it is, and they’re an inevitable part of dealing with events, Kafka, etc.
If so, you’ll be happy to hear that our Event Portal for Kafka product includes a discovery agent that can help you understand if and exactly how you’re at risk. It scans your system to ascertain within just minutes the size, scope and state of your Kafka event streams and infrastructure: how many clusters and streams you’ve got, how many have topics and schemas, of what kind (avro, json, primitives, etc.) how many are subscribed to — or not — and so on.
Last year my colleague Jesse Menning helped a prospective customer run such a scan on their system, and it revealed a shocking state of affairs. He documented the process, what they discovered, and how we’ve helped them clean things up, in this excellent blog post:How a Financial Services Giant Cleaned Up their Kafka with PubSub+ Event PortalOne of the biggest wealth management banks in the United States scanned their Kafka environment using Event Portal and was shocked to find out that they over six thousand topics, including 552 that weren’t being utilized by any applications, and many that contained circular dependencies! That's when the clean-up started! Read the blog to find out how they got there.
If you’re worried that you might be suffering from Kafkatosis, never fear: Event Portal for Apache Kafka can help you get things in shape by:
That’s just a summary of a summary, of course…visit the product page to learn how Event Portal for Kafka can accelerate the design of Kafka event streams and event-driven applications, improve the governance and lifecycle management of those assets, and boost developer productivity by integrating Kafka development into your SDLC processes. If you’re a visual learning like me, be sure to watch this awesome 10-minute demo that Jesse put together.
I hope I’ve helped you realize that the Kafka status quo doesn’t have to be, and there’s a straightfoward way to make life way easier for the architects, developers and administrators who run your Kafka estate. If you’re ready to see if you’re at risk of Kafkatosis, let’s talk!