The Solace connector for ingesting event data from Solace PubSub+ into Spark Streaming / Databricks is available as a free, community-supported connector.
Features
- It supports processing of specific number of messages in a partition (BatchSize option can be used to configure the number of messages per partition)
- Message are acknowledged to Solace once the commit method is triggered from Spark. Commit is triggered from Spark on successful write operation.
- Message IDs are persisted in checkpoint location once commit is successful. This helps in deduplication of messages and guaranteed processing.
