Integration Hub
>
Apache Spark
Asset Type:
Micro-Integrations
Provider:
Category:
Analytics & Stream Processing
Endpoint:
Target
Support:
Solace – Support Available
Platform:
External Embedded
Apache Spark
The Solace Spark Micro Integration enables seamless integration with Solace PubSub+ brokers, supporting both event ingestion and message publishing using the Apache Spark DataSource V2 API ensuring reliable, secure, and scalable stream processing in Apache Spark.
Features & Use Cases
- Batch Message Processing
Ingest messages from Solace PubSub+ in micro-batches, enabling efficient integration with Spark Structured Streaming workloads. - Reliable Acknowledgement Workflow
Messages are acknowledged to the source broker only after Spark completes a successful commit, ensuring at-least-once delivery. - Checkpointing and Deduplication
Message IDs are persisted in Spark’s checkpoint location post-commit, supporting deduplication and exactly-once processing across restarts and failures. - Supported Platforms
Compatible with Spark runtimes, across both cloud-native and on-premise deployments. - Message Publishing (Sink Support)
Supports publishing messages from Spark back to Solace PubSub+ topics, enabling closed-loop processing and feedback-driven event pipelines. - Dynamic Topic Publishing
Messages can be routed to dynamically generated Solace topics based on record fields or transformation logic. This allows content-aware event routing and supports topic hierarchies for scalable message distribution. - Secure Authentication Support
The connector supports multiple authentication mechanisms for secure communication with Solace brokers:
-
- Basic Authentication – Username and password.
- Client Certificate Authentication (mTLS) – Uses client certificates for mutual TLS.
- OAuth 2.0 (Client Credentials Flow) – Authenticates using access tokens obtained from an OAuth authorization server via the client credentials grant.