Databricks Zerobus Ingest

The Solace Micro-Integration for Databricks Zerobus bridges data between a Solace event broker and Databricks Zerobus. The Micro-Integration for Databricks Zerobus provides you (as an operator) a flexible and efficient way to load Databricks Zerobus application data from your Solace-backed, event-driven architecture and the event mesh.

The Databricks Zerobus Micro-Integration is available as both Cloud-managed and self-managed and supports multiple flows in each Micro-Integration to load multiple tables with events from your Solace queues in the same runtime.

• Target-only data flow: events move strictly from Solace PubSub+ to Databricks Delta tables. Each flow is bound to a target table defined by its fully qualified table name (catalog.schema.table), configured via the destination, or catalog, schema, and table producer properties.
• Dynamic destination: set the header “scst_targetDestination” with an expression that evaluates to a fully qualified Databricks table (catalog.schema.table) to route events to different Delta tables at runtime, per event, consistent with other PubSub+ connectors.
• OAuth 2.0 Client Credentials authentication using a Databricks service principal (clientId/clientSecret). Token acquisition and refresh are handled transparently.
• Health check capability (accessed via Actuator endpoint) that gives health status per workflow, per Databricks table stream.
• At-least-once delivery guarantee: Solace messages are acknowledged only after Databricks confirms the records are durably persisted.
• Automatic stream recovery with exponential backoff for retriable errors (e.g., network timeouts, rate limiting). Non-retriable errors (e.g., table not found, schema mismatch) NACK the message to the Solace DMQ. Note: a DMQ must be configured on the Solace queue.
• Payload format: records must be valid JSON (UTF-8); schema validation is enforced server-side by Zerobus against the target Delta table schema.
• Limitations: target Delta tables must pre-exist (the connector does not create tables); JSON over gRPC only.

Zerobus is a high-throughput streaming service for direct data ingestion into Databricks Delta tables, optimized for real-time data pipelines and high-volume workloads. More information on Zerobus is available from Databricks here.