Databricks

The Solace Connector for Databricks is an Apache Spark DataSource V2 library that lets an Apache Spark job read from and write to a Solace event broker. The Connector for Databrickssupports both directions:

Source — Stream messages from a queue on a Solace event broker into a Spark DataFrame using spark.readStream.
Sink — Publish rows from a Spark DataFrame to a Solace event broker using writeStream or forEachBatch.

Features & Use Cases

Supported Apache Spark Releases

This connector supports two release lines of Apache Spark. Configuration options and read/write behavior are identical between the two lines: only the runtime prerequisites, Maven coordinate, and packaging differ.

Use the 4.x line for new deployments. Continue using the 3.x line only if your environment is fixed at Apache Spark 3.5.x.
The 3.x line is supported until approximately September 2027.
If you are moving from the 3.x line to the 4.x line, see Migrating to the 4.x Line.

	3.x Line	4.x Line
Apache Spark	3.5.x	4.0.x
Scala	2.12	2.13
Java	11	17
Maven coordinate	`com.solacecoe.connectors:pubsubplus-connector-spark`	`com.solace:solace-connector-spark-4`

Resources

Apache Spark Micro-Integration

Still have questions?

Explore Other Micro-Integrations Get in Touch