Integrating Amazon Cloud-native Services with Solace

Those Cloud Environments are really starting to catch on

It’s no secret developers create applications in Amazon Web Services (AWS) or other cloud environments because it helps them to move faster, lower IT costs, and scale without the hassle of configuring, deploying, or maintaining the resources themselves (or having to go through corporate IT to get them).

But connecting applications – both on-premises and in the cloud – to one or more of those AWS cloud services requires an understanding of the specific connection protocols used for each, and for individual bridges to be built and maintained to those services for each application. This can be complex and time consuming, and takes resources away from building business value into the application itself. Moreover, the application data created within AWS may also be of interest to applications or consumers that are outside of the AWS ecosystem – in other regions, other clouds, on premises – and distributing that data to other locations using only AWS messaging isn’t sufficiently flexible, performant or cost effective.

Making it easier to connect Apps to AWS

That’s why we’ve developed a solution to move events into and out of AWS services to and from enterprise apps, streaming and in real-time, without requiring the development, deployment and management of bridges and bespoke integrations.

It’s a two-part solution made up of:

• an event mesh that moves events from apps on premises or in private or public clouds into AWS services, and
• a no-code/low-code integration from the event mesh with key AWS application integration services, including SQS, SNS, Kinesis, S3, and Lambda, which is the topic of this specific blog.

The low-code/no-code integration was developed by Solace Labs. Developers can use Solace-created, open source CloudFormation templates to configure connectivity between the Solace PubSub+ event broker and the desired AWS service (e.g., SQS, SNS, S3, Kinesis and Lambda) and enable message exchange. This means developers don’t have to know any of the underlying connection details or develop, deploy or maintain bridges from the application to those services.

The PubSub+ broker deployed in AWS can connect to Solace event brokers located in other public or private clouds or within customer premises to form an event mesh. This allows information assets produced anywhere within the event mesh to easily be consumed by AWS services, and also share the processed data from these services on the event mesh to applications or consumers that reside in other Amazon regions, in other public or private clouds, or on premises in a well-architected, governed, secure, manageable, and reusable manner.

In addition to the simplicity of event distribution, the WAN optimization inherent in the event mesh keeps costs much lower than using a network connection into Amazon from each application.

How the Solace AWS Integration works

The Solace AWS Integration takes advantage of the REST delivery point built into our PubSub+ broker which, when combined with Solace’s use of open APIs, provide the flexibility required for successful integration. The necessary scripts, CloudFormation templates and instructions used to configure the necessary AWS infrastructure – such as VPC endpoints and an AWS API gateway – are available free of charge as an open-source product on GitHub and available with community support.

Connecting to AWS-native services requires that a PubSub+ broker be deployed in the AWS region in which events are to be moved to or from. To attract events to a particular AWS service, a queue and subscriptions on that queue must be configured. If the same event needs to be delivered to multiple services, then one queue per service must be configured. REST delivery points must be configured on the PubSub+ broker to push the events from this queue to the REST service. Cloud Formation templates and scripts are used to configure the necessary AWS infrastructure – such as VPC endpoints and an AWS API gateway – between the PubSub+ broker and the specific AWS service.

The Solace Integration uses the Amazon API Gateway to connect to a virtual private cloud (VPC) endpoint. For security purposes, only the PubSub+ broker can send information to that VPC endpoint, which sends the data into the API gateway. The API gateway contains all the logic to take a Solace message, wrap it in the correct format for the service it is being sent to (different for each service) and then sends it to that service.

Note that the use of AWS API Gateway limits the performance of the integration to approximately 1000 messages per second, which may not be suitable for all customer use cases.

This solution can be used directly from the SolaceLabs GitHub repo when using PubSub+ broker. The integration also works with PubSub+ Cloud, however you’ll need to contact the Solace Cloud team from within the console for assistance setting it up.

The AWS Integration joins our Kafka Connector and Spring Cloud Stream binder as integrations you can use to connect events to and from any environment.