The Scalability Downside of Static Topics; Learning from LinkedIn’s Implementation of Apache Kafka
Kafka was born as a stream processing project of LinkedIn’s engineering team, so a recent
This is the first in a series of blog posts in which we’ll compare Solace PubSub+ Event Broker with Apache Kafka. Our goal is to help you understand the salient differences between the platforms, arming you with the knowledge you need to select the right streaming platform for your organization. For a quick 5-minute overview, check out this video.
About Apache Kafka
Apache Kafka is a distributed data-ingest and event streaming platform developed in 2011 by engineers at LinkedIn for consuming large volumes of log data. Although commonly called a broker, Kafka is more akin to a distributed data store than a traditional message broker. For example, Kafka only supports publish/subscribe messaging by default, and uses the concept of an ever-increasing commit log, where messages are generally retained indefinitely.
Operationally, Apache Kafka brokers run in a cluster coordinated by Apache Zookeeper. Producers publish records to consumers on named topics. Messages, termed records, are prepended to predefined topics and identified with an offset. Topics may be spread across multiple Kafka brokers using a strategy known as topic partitioning, wherein each broker stores a subset of records intended for a given topic. Consumers ingest messages from all partitions (i.e., the entire topic).
About PubSub+ Event Broker
Solace PubSub+ Event Broker offers event streaming along with traditional message broker functionality. Initially developed in 2001 as a purpose-built messaging appliance for the capital markets, the broker has since been virtualized and implemented in both standalone software and SaaS form-factors.
Solace PubSub+ supports several messaging exchange patterns including publish/subscribe, FIFO queueing, and asynchronous request/reply, and both persistent and non-persistent qualities of service. With persistent messaging, producers emit events which are attracted to consumer queue(s) using subscriptions (more on the power of subscriptions later); events arrive at the queue in FIFO order and are marked as read only once receipt is confirmed by the consumer. Subscriptions allow us to decide at runtime which messages will be attracted to each consumer. They give us true pub-sub decoupling of events. Watch the video below for more detail.
Non-persistent messaging supports pub/sub whereby one or more publishers send messages to multiple subscribers across topics. Unlike Kafka, Solace is more flexible. It supports both persistent and in-memory messaging as well as multiple messaging patterns such as pub/sub, FIFO queueing, request/reply, and point-to-point. To implement pub/sub messaging pattern, publishers simply publish to Solace’s hierarchical topics. These topics are simply metadata tags attached to the payload and do not need to be provisioned or partitioned beforehand. Consumers can consume data directly using topics if they don’t require persistence, or they can utilize queues to enable persistence, and map topics they would like to enqueue data for.
To understand more about Solace topics vs Kafka topics, check out this popular video by Developer Advocate Aaron Lee or check out his blog post Solace Topics vs. Kafka Topics: What’s the Difference?
Conclusion
As you can see, Kafka and Solace PubSub+ have two very different internal architectures. In the next post, my colleague Himanshu Gupta will compare each platform’s publish/subscribe functionality in greater detail. Meanwhile, you can visit our website to read more about how Solace PubSub+ compares to Kafka or see what your peers and other real users have to say about the two platforms on Peerspot.
Check out all the posts in the Solace vs. Kafka series:
- Solace PubSub+ vs Kafka: Implementation of the Publish-Subscribe Messaging Pattern
- Solace PubSub+ vs Kafka: Filtering
- Solace PubSub+ vs Kafka: High Availability
- Solace PubSub+ vs Kafka: Multi-Site Architecture
And for more details regarding operational use cases and why you need to look beyond Kafka, check out this other series:
Subscribe to Our Blog
Get the latest trends, solutions, and insights into the event-driven future every week.
Related Posts You Might Be Interested In…
Solace Topics vs. Kafka Topics: What’s the Difference?
Ever wondered what the difference is between a Solace topic and an Apache Kafka topic?
When Connecting Kafka to Solace PubSub+ Makes Sense
Update: We now offer an integrated Kafka bridge! Learn about it here. Many customers tell
Why You Need to Look Beyond Kafka for Operational Use Cases, Part 1: The Need for Filtering and In-Order Delivery
Preface Over the last few years Apache Kafka has taken the world of analytics by
How Solace Compares to Other Event Streaming Platforms for EDA
We often get asked how Solace PubSub+ compares with alternative event streaming
Stop Playing Whack-a-Mole with Your Kafka Event Streams!
An architect once told us that managing his Kafka event streams was like playing a
Why You Need to Look Beyond Kafka for Operational Use Cases, Part 2: The Importance of Flexible Event Filtering
Preface Over the last few years Apache Kafka has taken the world of analytics by
Why Architects Need Tools for Apache Kafka Service Discovery, Auditing, and Topology Visualization
You’re out of control. I hate to be the bearer of bad news, but sometimes
Why You Need to Look Beyond Kafka for Operational Use Cases, Part 3: Securing the Eventing Platform
Preface Over the last few years Apache Kafka has taken the world of analytics by
Using PubSub+ Event Portal for Support and Visualization of your Kafka Streams
One of the persistent challenges with a distributed architecture—like with Kafka—is the ability to design
Why You Need to Look Beyond Kafka for Operational Use Cases, Part 4: Streaming with Dynamic Event Routing
Preface Over the last few years Apache Kafka has taken the world of analytics by
Solace PubSub+ vs Kafka: Implementation of the Publish-Subscribe Messaging Pattern
This blog post is part of a series comparing Apache Kafka with Solace’s PubSub+ Event
Solace PubSub+ vs Kafka: Filtering
This blog post is part of a series of blog posts comparing Apache Kafka with
Solace PubSub+ vs Kafka: High Availability
This blog post is part of a series of blog posts comparing Apache Kafka with
Solace PubSub+ vs Kafka: Multi-Site Architecture
This comparison of how Apache Kafka and Solace PubSub+ each enable event streaming across multi-site