Solace offers two ways to connect to Kafka clusters. The first is the broker integrated Kafka Bridge, native in PubSub+ Event Broker software and cloud managed versions. The second is the external embedded source and sink connectors that run in your Kafka Connect infrastructure. The broker-integrated connector provides the most simple, secure, performant, and cost-effective option. It is the perfect option for those that don’t want to run a Kafka connect service due to cost and complexity.
However, some organizations prefer to operate Kafka connect infrastructure and use it to manage other connectors for Kafka. While this can be managed more efficiently in PubSub+ via a Kafka mesh, this option provides flexibility to do what you want how you want it.
For PubSub+ appliances, the external embedded connectors are the method to connect to your Kafka clusters.
Why a Kafka mesh helps with event streaming?
If you use Kafka, you’re probably wondering why you need PubSub+ Platform. Although there may be some functional overlap between the two technologies, they excel at event streaming for different use cases. Kafka excels at event streaming for storage and analytic use cases such as log aggregation, event stream processing, event sourcing, website activity tracking and aggregating statistics from distributed applications for metrics and monitoring. PubSub+ Platform excels at event streaming and management for transactional and operational use cases such as enabling event-driven microservice, hybrid/multi-cloud event distribution and management, event driven integration, IoT enablement and enabling real-time applications. PubSub+ delivers:
- Flexibility: Easy access to the rich functionality and multi-protocol support of PubSub+ eliminates the need to write connectors for other services.
- Scalability: Enables connections and communications between a virtually unlimited number of applications and IoT devices.
- Dynamic Routing: Support for hierarchical topics and wildcards, combined with dynamic intelligent routing, enables all kinds of fan-in, fan-out and filtration.
- Global/WAN Distribution: PubSub+ lets you establish an event mesh that dynamically routes events between applications and connected devices no matter where they are deployed, what API they use, or which broker they are connected to.
- Bi-Directional MQTT: The ability to not just collect data via MQTT, but to send alerts and instructions to specific devices or vehicles, enables sophisticated command and control use cases.
Use Cases
Create new versions of these pics removing the part with the x-out. It keeps it generic and allows for both options to be plopped into the diagram by the reader.
Running Real-time Analytics across Hybrid Cloud
Connect a Kafka event stream to PubSub+ Event Broker to route a filtered set of information to a cloud analytics engine. PubSub+ keeps bandwidth consumption low by using fine-grained topic filtering to deliver exactly and only the events required. For example, if you’re using Kafka to aggregate log data and perform offline analytics on it, but want to use a real-time analytics service running in the cloud to promote products based on sentiment analysis or real-time weather conditions, PubSub+ can take the event stream from Kafka and route a filtered set of information to the analytics engine.
Ingesting High-Volume Web and Mobile Data to Kafka for Aggregation and Analytics
PubSub+ supports MQTT connectivity at massive scale, able to establish reliable, secure, real-time communications with tens of millions of devices or vehicles so you can collect data and hand it off to Kafka for aggregation or analytics. Since PubSub+ can also stream events via other popular open protocols and APIs, you can stream events from all of your applications, running in all kinds of cloud and on-premises environments, to Kafka via the same source connector.
Streaming Events Recorded in Kafka to Connected Devices or Vehicles
In addition to supporting the inbound aggregation of events from millions of connected devices, PubSub+ supports bidirectional messaging and the unique addressing of millions of devices through fine-grained filtering. For example, with PubSub+ and Kafka working together you could send a dangerous weather warning alert to a specific vehicle, or all vehicles in or approaching the affected area. The sink connector allows PubSub+ to send record events placed in a single Kafka topic to whatever vehicles satisfy a given condition or topic, whether that’s as general as being in the tri-county area or as specific as a single vehicle.
Connector Options
Integrated Kafka Bridge
(beta released June 2023 in PubSub+ Event Broker Software)
Solace PubSub+ Integrated Kafka Bridge works natively in the software event broker. It can be configured directly in the event broker via the GUI, CLI or via the SEMP API. This provides bidirectional connectivity with no additional software to configure or manage. PubSub+ supports popular messaging APIs and protocols like AMQP, JMS, MQTT, REST, WebSocket and more so the connector can move any message to a topic (keyed or not keyed) on the Kafka broker.
Source Connector
Solace PubSub+ Connector for Kafka Source uses the Kafka Connect API to consume PubSub+ queue or topic data events and stream them to a Kafka topic. The source connector uses Solace’s high performance Java API to move PubSub+ data events to the Kafka Broker.
Sink Connector
Solace PubSub+ Connector for Kafka Sink uses the Kafka Connect API to consume Kafka topic records, then stream the data events to PubSub+ event brokers as a topic and/or queue data event. From the sink connector, any Kafka topic (keyed or not keyed) sink record is instantly available for consumption by any consumer that uses one of the many languages, protocols or APIs that PubSub+ supports.