Store events on Amazon Simple Storage Service (Amazon S3) from PubSub+ event brokers by leveraging REST Delivery Points (RDPs) to generate Amazon data lakes to archive and analyze your events as part of your event-driven architecture (EDA) journey.
The 9.13.0 release of Solace’s PubSub+ event broker enables Authentication with AWS Signature Version 4! This allows PubSub+ to directly integrate (without API gateway or Lambda) with Amazon S3 via RDP!
You can Directly integrate Amazon S3 through the creation of an RDP on PubSub+ with the following configuration:
- Create the RDP (create rest-delivery-point <name>) from within your message-vpn configuration
- If not using the default client profile, identify the desired client profile (client-profile <name>)
- Put at least one queue binding to a queue (create queue-binding <queue-name>) in the rest-delivery-point
- Generate the post-request-target for the queue binding to Amazon S3 file naming for each of the events. To determine a filename that changes for each event, you will need to use substitution expressions. See the link below for more details (post-request-target <substitution expression>)
- Enable request-target-evaluation for the queue binding must be set to substitution-expressions to ensure it is evaluated properly (request-target-evaluation substitution-expressions)
- Step back from the queue-binding configuration (exit) to the rest delivery point level
- Create the REST Consumer (create rest-consumer <name>)
- Configure the REST consumer’s remote characteristics (remote)
- Identify the REST Consumer Host (host <Amazon S3 bucket dest-ip-addr or host DNS name>)
- Turn on SSL (ssl)
- Set the port to Amazon’s S3 HTTPS/SSL port (port 443)
- Configure the HTTP Verb to “Put” (http-method put) for Amazon S3
- Authentication with AWS Signature Version 4. You will require your AWS region, access key id, and secret access key information from your AWS administrator:
- Step back from the remote configuration to the rest-consumer configuration (exit)
- Identify that you want to use AWS Signature Version 4 (authentication auth-scheme aws)
- Start the AWS Signature Version 4 configuration (aws)
- Specify S3 as the service (service s3)
- Configure your AWS region (region <AWS region>)
- Enter your access key id (access-key-id <AWS access-key-id>)
- While ensuring you have privacy, type in your secret access key (secret-access-key <AWS secret-access-key>)
- Back out of the AWS Signature Version 4 configuration (exit) to the authentication level
- Step back from the authentication level (exit) to the rest-consumer configuration level
- Turn up the rest-consumer configuration (no shutdown)
- Leave the rest-consumer config (exit)
- Activate the RDP (no shutdown)
…And that’s it! If you have preconfigured your Amazon S3 service, your PubSub+ broker will now be delivering messages from the bound queue to Amazon S3!
Event enabling data lakes is an important aspect of PubSub+ Event Broker that enables a complete event-driven architecture!
This connector is FREE to use, all you need is a valid PubSub+ Event Broker support plan and you are all set!
