Extend AWS Scalability Beyond the AWS Cloud with an Event Mesh

AWS gives you unparalleled flexibility and scalability in managing and processing your data. It can help you extend your reach, build innovative solutions, and reduce your time to market. However, when you look at getting your AWS data into motion and into other public clouds, their complex array of messaging and queueing tools can limit your scalability, introduce security gaps and risks, impact your time-to-market, and build-up avoidable expenses.

Guaranteed Delivery, Fan Out, Ordering – Why Not SNS/SQS?

AWS offers a few options for sending persistent, time-sensitive messages and for supporting event-driven architecture. At first glance, it may seem like Simple Notification Service (SNS) and Simple Queue Service (SQS) have all your bases covered for fanout, guaranteed delivery, reliability, scalability, and ordering – but there are limitations you should be aware of that could impact your business.

You may already know that SNS only guarantees delivery if the endpoint is accessible, with a limited number of retries before being lost. SNS and SQS are commonly used together to overcome the challenges of asynchronous messaging – including guaranteed delivery – but, unfortunately cannot tackle the challenge of message ordering.

When working with data that is real-time and needs to be delivered in order – in a market data scenario, for example – the AWS messaging services will typically give you a choice between SNS (to publish the values of a stock) and SQS (to accept and deliver the stock price). When the order of messaging is important, AWS offers SQS FIFO (first-in-first-out). This sounds like an ideal match for its distributed publish-subscribe system (SNS), especially since SQS FIFO supports exactly once delivery. Unfortunately, SNS and SQS FIFO are incompatible – SNS can only feed into standard queues (best-effort ordering, delivered at least once).

In an event-driven system, there are many cases where data is time-critical, has many consumers, and the order of delivery is important. A combination of these solutions or a few workarounds may get you by, but one needs to also consider the other limitations of a solution that relies only on AWS messaging.

How AWS Limits Your Ability to Use Other Clouds and Platforms

The bigger challenge emerges when you are looking to use best-of-breed services from multiple platforms and clouds. The AWS messaging services cannot extend to other clouds, which forces you to egress all data at significant cost, compromises data lineage, and requires your developers to learn how to implement multiple APIs within the AWS cloud and in other platforms.

The bottom line is this: you need multiple AWS messaging/integration services to deliver common business use cases. Naturally, this makes design and management more difficult. Even then, it’s mostly restricted to inside the AWS cloud – which may not be easy to work with.

This diagram shows the utility of Solace PubSub+ vs. the various AWS tools at your disposal, in context of how easy it is to deploy them in AWS.

For use cases that span multiple platforms and require real-time data distribution, enterprises benefit from a loosely coupled, distributed system that uses the same API and consistent features to ingest and deliver traffic. Standardization of features, code, pricing, etc. would help developers focus on the use cases rather than the plumbing.

This can be accomplished by deploying code through the cloud, your virtual private cloud, or an on-premises appliance to gather and deliver events across a network of event brokers – or an event mesh.

How PubSub+ and an Event Mesh Gets Data Moving Better than AWS’s Own Services

PubSub+ is an event streaming and management platform that helps enterprises design, deploy, and manage event-driven applications across on-premises, hybrid cloud, multi-cloud, and IoT environments so they can be integrated and event-driven.

PubSub+ lets you:

• Build services that can be replicated or ported between clouds and platforms without code changes
• Scale up or down and still control your costs
• Achieve excellent performance even when you implement message ordering, guaranteed delivery, and high availability

Single Platform Standardization

AWS provides 5 different services to distribute information and events, each of which meets some data distribution requirements. Here’s a summary of what most customers need in this area, and how the AWS integration/messaging services (and PubSub+) can meet those needs.

Taken together, AWS services check all the boxes. But they don’t work on the same set of APIs, aren’t priced the same way, and can’t be deployed in any and all environments that you choose for your distributed enterprise. Relatively speaking, PubSub+ is a Swiss Army knife that meets all these needs in one platform and in a distributed environment.

A significant factor here is application portability.

When microservices interface with PubSub+ Event Broker, they can be ported as-is between data centers and cloud platforms. These services are coded not to interface with SQS/SNS/EventBridge APIs or Kinesis, but to interface over – for example – JMS with PubSub+.

While AWS mainly has a cloud-only software form factor, PubSub+ can be deployed as an appliance, software, cloud, or a combination of all.

Giving You Performance Without Added Complexity

AWS provides scalability, global reach, and unparalleled cost-efficiency across its wide spectrum of services.

Working in conjunction with AWS, PubSub+ can help move events and data in and out of the AWS cloud. It will also glean relevant events/messages from outside the AWS cloud and push them to relevant AWS services. A customer can architect and efficiently govern data mobility through topic filters.

In AWS, the recommended way to decouple the producer and consumer is by hopping through event routers (SNS, EventBridge) and queues (AMQ, SQS, etc.).

Each such interaction or traffic flow needs active oversight, and developers spend cycles on:

• Building and maintaining bespoke integrations, e.g.:
  • Events originating from an HTTP endpoint feeding through EventBridge into Amazon SQS to another HTTP endpoint.
  • A developer would love to standardize on SNS streaming events into an SQS FIFO queue for in-order delivery, but that combination is not supported.
• Building new message pathways with a different logic for each use case, e.g.:
  • Neither SQS nor SNS supports request-reply natively and only two distinct pathways can deliver a command and control type use case. So, a command sent through SNS to IoT endpoints to stop all conveyor belts in a factory will not get acknowledgement on the same channel.

Such complexities can be counterproductive and distract developers from building based on business logic to not having to spend time to learn and then design around oddities that should not exist in the first place.

What will be ideal for developers and architects is a simple way to tie multiple ends of their distributed architecture together – while not being excessively concerned with managing messaging patterns or having to build parallel pathways for transactions that are naturally bi-directional.

PubSub+ is made precisely to meet these needs in a distributed environment.

Developers benefit from:

• A simple, single API that ties brokers across clouds and platforms.
  • Unlike SQS – which polls to get data from a queue – PubSub+ will push data into any and all environments to an interested subscriber.
• Eliminated complexity by streamlining traffic flows through dynamic topics.
  • The source can produce traffic at virtually any rate and the streams can be abstracted and distributed based on dynamic and hierarchical topics on the platform.

Contain In-Motion Data Within Your Governance Perimeter

SQS, SNS, and EventBridge cannot be deployed inside a VPC in the AWS cloud, so these “public” services might put your data beyond your governance perimeter.

AWS proposes that an enterprise interface with these services using a VPC endpoint and a secure link, but SQS, SNS, and EventBridge services are “public” by design. That said, when messages leave your VPC into the AWS public services realm, you lose direct control over the location of your messages because AWS will replicate data across zones/regions to deliver their promised availability.

In contrast, PubSub+ Event Broker: Software can be deployed on the AWS cloud instance of choice, either inside a VPC (BYOL) or through a cloud formation template and peered with AWS services.

It also gives you control over how you move your data out of the AWS cloud, lets you glean relevant events/messages, and lets you push data to relevant AWS services.

These options help contain in-motion data within your operations and governance perimeter.

Examples Where PubSub+ Augments the Power of AWS

Global Airline

This airline needed a solution to distribute flight events across their various applications and lines of business in real-time, and it had to align with its cloud-first strategy. The main objective was to put batch data from Amadeus (a leading booking and operations platform) into motion and use services in the AWS cloud to action on key events.

PubSub+ Cloud interfaces with the customer’s VPC in AWS cloud and streams events and data between their IBM mainframe-based applications and AWS services.

Smart City Project in Asia

The IoT sensors on a city bus publish MQTT messages to a topic that contains the bus’s current coordinates.

SNS requires pre-determined topics configured and coded into the publishing/subscribing applications. Without wildcard character support, a restriction on the number of topics per account and a limit of 200 filter policies per region, SNS cannot support a use case where each new latitude/longitude pair is the “topic” on which a city bus transmits data. On the consumer end, systems can filter down on wildcards and not be restricted by a limitation on a number of filter policies per region.

With 5,500 buses constantly sending messages with changing topic destinations, it was impractical to build an efficient solution for this using AWS SQS, SNS, or EventBridge.

With PubSub+ our client can gather and filter down actionable data from all the buses in real-time. They filter relevant data and process it on-premises and also in the AWS cloud to make real-time decisions on their network.

To Recap: Enhance the Value of Your Systems with PubSub+

By making cloud-native services pervasive, AWS has enabled mature enterprise customers to build complex workloads with high agility and with great impact. Customers that are working to strengthen their hybrid cloud architecture (while building a path to multi-cloud) will benefit from a high performance, standardized, and governance-ready data movement layer.

PubSub+ is field-proven in enhancing the value of our customer’s investments in AWS and will help extract value from your entire application ecosystem, including microservices, SaaS, cloud services, legacy apps, mobile devices, and the IoT.

Try out Solace PubSub+ for free on AWS Marketplace.