When Connecting Kafka to Solace PubSub+ Makes Sense

Many customers tell us they love our PubSub+ event broker because it lets them establish event-driven interactions between applications and microservices across hybrid cloud environments using open APIs and protocols. At the same time, they acknowledge that their developers are committed to using Kafka.

I’ve got good news!

We’ve just introduced two Kafka Connectors, one source and one sink, that make it really easy to combine the unique advantages of these two technologies.

There is some functional overlap between Solace and Kafka, of course, but if you look at the underlying architecture and operational characteristics of the two tools you’ll realize they solve different problems in very different – and frequently complementary – ways.

Here are a few ways Solace and Kafka can complement each other in the enterprise ecosystem, which is now easier than ever thanks to our new Kafka Connectors.

Scenario 1: Connectivity across hybrid cloud

Imagine your company is using an on-premises Kafka to aggregate log data and perform offline analytics on it. Kafka meets this need perfectly well, but what if you want to use a real-time analytics service running in multiple public clouds to promote products to your website visitors based on sentiment analysis, weather patterns, news feeds, etc.?

How can you take the event stream from Kafka, filter it, and connect to the cloud-based analytics engines?
One easy way would be to introduce Solace between Kafka and the cloud. Here are a few advantages of this approach:

1. Solace connects applications and information sources across multiple cloud and on-premises environments without additional adapters or bridges and topic/queue hierarchy.
2. Solace reduces bandwidth consumption by using fine-grained topic filtering that enables the delivery of exactly and only the messages required by the analytics service. With Solace you just send the data needed for a promotional offer for a segment of your customers. You could, for example, present a winter tires promotion to customers in Ottawa that came in to get a flat fixed.
3. Solace lets you establish an event mesh (a network of advanced event brokers) that dynamically routes events from one application to another no matter where these applications are deployed (cloud, private cloud, public cloud) or what API they use.

Scenario 2: Terminate web and MQTT connections

Let’s say you want to connect your Kafka-based analytics application to millions of vehicles to perform real-time analytics and distribute traffic alerts or diagnostics data. You’ll need an MQTT broker to terminate these connections and move the data to the Kafka analytics system.

For that you can use PubSub+ to terminate these MQTT connections and push events to your Kafka environment via the new Solace Source Connector. Why?

1. PubSub+ supports the federation of tens of millions of MQTT connections.
2. Solace can terminate many standard protocols and open APIs, which means you can stream events from those applications to Kafka via the same Solace Source Connector.
3. Using a single Solace Source Connector means you don’t have to write your own connectors for each service. Isn’t this easier than what you’re currently doing?
4. These IoT devices can be connected to any cloud or any number of clouds, public or private, as well as on-premises systems, and PubSub+ will automatically get the information where it needs to go.

Scenario 3: Send command and control messages to IoT devices

In addition to supporting the inbound aggregation of events from millions of connected devices, PubSub+ supports bi-directional messaging and the unique addressing of millions of devices through fine-grained filtering.

For example, with Solace and Kafka working together you could send a tornado warning alert to a specific vehicle, or all vehicles in or approaching the affected area.

Looking under the hood, a single Solace Sink Connector allows PubSub+ to send record events placed in a single Kafka Topic to whatever vehicles satisfy a given condition or topic, whether that’s as general as being in the tri-county area or as specific as a VIN#, i.e., a single vehicle.

Conclusion

Those are three ways you can achieve exciting capabilities and benefits by combining Solace and Kafka using our new Kafka Connectors. Watch the demo below to see how it works under the hood.

Then check out this video on how to get started and give it a try.

Sandra Thomson

Sandra Thomson is the Director of Product Marketing at Solace.

See all posts written by Sandra Thomson