Store events on Amazon Simple Storage Service (Amazon S3) from PubSub+ event brokers by leveraging REST Delivery Points (RDPs) to generate Amazon data lakes to archive and analyze your events as part of your event-driven architecture (EDA) journey.

The 9.13.0 release of Solace’s PubSub+ event broker enables Authentication with AWS Signature Version 4!  This allows PubSub+ to directly integrate (without API gateway or Lambda) with Amazon S3 via RDP!

You can Directly integrate Amazon S3 through the creation of an RDP on PubSub+ with the following configuration:
  • Create the RDP (create rest-delivery-point <name>) from within your message-vpn configuration
  • If not using the default client profile, identify the desired client profile (client-profile <name>)
  • Put at least one queue binding to a queue (create queue-binding <queue-name>) in the rest-delivery-point
  • Generate the post-request-target for the queue binding to Amazon S3 file naming for each of the events.  To determine a filename that changes for each event, you will need to use substitution expressions.  See the link below for more details (post-request-target <substitution expression>)
  • Enable request-target-evaluation for the queue binding must be set to substitution-expressions to ensure it is evaluated properly (request-target-evaluation substitution-expressions)
  • Step back from the queue-binding configuration (exit) to the rest delivery point level
  • Create the REST Consumer (create rest-consumer <name>)
  • Configure the REST consumer’s remote characteristics (remote)
  • Identify the REST Consumer Host (host <Amazon S3 bucket dest-ip-addr or host DNS name>)
  • Turn on SSL (ssl)
  • Set the port to Amazon’s S3 HTTPS/SSL port (port 443)
  • Configure the HTTP Verb to “Put” (http-method put) for Amazon S3
  • Authentication with AWS Signature Version 4.  You will require your AWS region, access key id, and secret access key information from your AWS administrator:
    • Step back from the remote configuration to the rest-consumer configuration (exit)
    • Identify that you want to use AWS Signature Version 4 (authentication auth-scheme aws)
    • Start the AWS Signature Version 4 configuration (aws)
    • Specify S3 as the service (service s3)
    • Configure your AWS region (region <AWS region>)
    • Enter your access key id (access-key-id <AWS access-key-id>)
    • While ensuring you have privacy, type in your secret access key (secret-access-key <AWS secret-access-key>)
    • Back out of the AWS Signature Version 4 configuration (exit) to the authentication level
    • Step back from the authentication level (exit) to the rest-consumer configuration level
  • Turn up the rest-consumer configuration (no shutdown)
  • Leave the rest-consumer config (exit)
  • Activate the RDP (no shutdown)
…And that’s it!  If you have preconfigured your Amazon S3 service, your PubSub+ broker will now be delivering messages from the bound queue to Amazon S3!

Event enabling data lakes is an important aspect of PubSub+ Event Broker that enables a complete event-driven architecture!

This connector is FREE to use, all you need is a valid PubSub+ Event Broker support plan and you are all set!

Provider: Solace
Endpoint Service
Amazon S3
Provider
Solace
Connector Type
Connector Endpoint
License

Still have questions?

Explore other connectors Get in touch