You can save events on Azure Data Lake (Gen 2) from PubSub+ event brokers by leveraging its REST Delivery Points (RDPs) to store and analyze your events as part of your event-driven architecture (EDA) journey.

In the 9.11 release of Solace’s PubSub+ event broker, the Broker Manager can create REST connectors directly to Azure Data Lake (Gen 2).  This is done through the creation of an RDP with the following configuration:

  • Create the RDP (create rest-delivery-point <name>)
  • Link to an existing client profile (client-profile <client profile name>)
  • Create A Rest Consumer (create rest-consumer <name>)
  • The consumer designated as remove (remote)
  • The Rest Consumer Host (host <Azure data Lake Gen 2 container dest-ip-addr-or host DNS name>)
  • SSL on <ssl>
  • HTTP method put <http-method put>
  • Authentication with OAuth Client credentials (RFC 6749) with the oauth client-id, client-secret, token-endpoint, and scope as provided by your Azure administrator:
    • exit
    • authentication oauth-client
    • client-id <client-id>
    • client-secret <client-secret>
    • token-endpoint <token-endpoint>
    • scope <scope>
    • exit
    • exit
    • exit
  • Put at least one queue binding to a queue (create queue-binding <queue-name>) at the rest-delivery-point
  • The post-request-target for the queue binding to Azure Data Lakes Gen 2 Container path and file name you want to use for each of the events.  To determine a file name that changes for each event, you will need to use substitution expressions.  See the link below for more details
  • The request-target-evaluation for the queue binding must be set to substitution-expressions to ensure it is evaluated properly (request-target-evaluation substitution-expressions)
  • The following configuration of headers is also necessary starting from the queue binding:
    • create request-header Content-Type
    • header-value application/json
    • exit 
    • create request-header x-ms-version
    • header-value 2017-11-09
    • exit 
    • create request-header x-ms-blob-type
    • header-value BlockBlob
    • exit 
    • exit
  • Then from the rest-delivery-point, you enable the config with:
    • rest-consumer no shutdown
    • no shutdown

Event enabling data lakes is an important aspect of PubSub+ Event Broker, fulfilling a complete event-driven architecture!

It is FREE to use, all you need is a valid PubSub+ Event Broker support plan and you are all set!

Provider: Solace
Endpoint Service
Azure Data Lakes Gen 2
Provider
Solace
Connector Type
Connector Endpoint
License

Still have questions?

Explore other connectors Get in touch

What's New with PubSub+? Find out on December 7!