AWS gives you unparalleled flexibility and scalability in managing and processing your data. It can help you extend your reach, build innovative solutions, and reduce your time to market. However, when you look at getting your AWS data into motion and into other public clouds, their complex array of messaging and queueing tools can limit your scalability, introduce security gaps and risks, impact your time-to-market, and build-up avoidable expenses.
AWS offers a few options for sending persistent, time-sensitive messages and for supporting event-driven architecture. At first glance, it may seem like Simple Notification Service (SNS) and Simple Queue Service (SQS) have all your bases covered for fanout, guaranteed delivery, reliability, scalability, and ordering – but there are limitations you should be aware of that could impact your business.
You may already know that SNS only guarantees delivery if the endpoint is accessible, with a limited number of retries before being lost. SNS and SQS are commonly used together to overcome the challenges of asynchronous messaging – including guaranteed delivery – but, unfortunately cannot tackle the challenge of message ordering.
When working with data that is real-time and needs to be delivered in order – in a market data scenario, for example – the AWS messaging services will typically give you a choice between SNS (to publish the values of a stock) and SQS (to accept and deliver the stock price). When the order of messaging is important, AWS offers SQS FIFO (first-in-first-out). This sounds like an ideal match for its distributed publish-subscribe system (SNS), especially since SQS FIFO supports exactly once delivery. Unfortunately, SNS and SQS FIFO are incompatible – SNS can only feed into standard queues (best-effort ordering, delivered at least once).
In an event-driven system, there are many cases where data is time-critical, has many consumers, and the order of delivery is important. A combination of these solutions or a few workarounds may get you by, but one needs to also consider the other limitations of a solution that relies only on AWS messaging.
The bigger challenge emerges when you are looking to use best-of-breed services from multiple platforms and clouds. The AWS messaging services cannot extend to other clouds, which forces you to egress all data at significant cost, compromises data lineage, and requires your developers to learn how to implement multiple APIs within the AWS cloud and in other platforms.
The bottom line is this: you need multiple AWS messaging/integration services to deliver common business use cases. Naturally, this makes design and management more difficult. Even then, it’s mostly restricted to inside the AWS cloud – which may not be easy to work with.
This diagram shows the utility of Solace PubSub+ vs. the various AWS tools at your disposal, in context of how easy it is to deploy them in AWS.
For use cases that span multiple platforms and require real-time data distribution, enterprises benefit from a loosely coupled, distributed system that uses the same API and consistent features to ingest and deliver traffic. Standardization of features, code, pricing, etc. would help developers focus on the use cases rather than the plumbing.
This can be accomplished by deploying code through the cloud, your virtual private cloud, or an on-premises appliance to gather and deliver events across a network of event brokers – or an event mesh.
PubSub+ is an event streaming and management platform that helps enterprises design, deploy, and manage event-driven applications across on-premises, hybrid cloud, multi-cloud, and IoT environments so they can be integrated and event-driven.
PubSub+ lets you:
AWS provides 5 different services to distribute information and events, each of which meets some data distribution requirements. Here’s a summary of what most customers need in this area, and how the AWS integration/messaging services (and PubSub+) can meet those needs.
Taken together, AWS services check all the boxes. But they don’t work on the same set of APIs, aren’t priced the same way, and can’t be deployed in any and all environments that you choose for your distributed enterprise. Relatively speaking, PubSub+ is a Swiss Army knife that meets all these needs in one platform and in a distributed environment.
A significant factor here is application portability.
When microservices interface with PubSub+ Event Broker, they can be ported as-is between data centers and cloud platforms. These services are coded not to interface with SQS/SNS/EventBridge APIs or Kinesis, but to interface over – for example – JMS with PubSub+.
AWS provides scalability, global reach, and unparalleled cost-efficiency across its wide spectrum of services.
Working in conjunction with AWS, PubSub+ can help move events and data in and out of the AWS cloud. It will also glean relevant events/messages from outside the AWS cloud and push them to relevant AWS services. A customer can architect and efficiently govern data mobility through topic filters.
In AWS, the recommended way to decouple the producer and consumer is by hopping through event routers (SNS, EventBridge) and queues (AMQ, SQS, etc.).
Each such interaction or traffic flow needs active oversight, and developers spend cycles on:
Such complexities can be counterproductive and distract developers from building based on business logic to not having to spend time to learn and then design around oddities that should not exist in the first place.
What will be ideal for developers and architects is a simple way to tie multiple ends of their distributed architecture together – while not being excessively concerned with managing messaging patterns or having to build parallel pathways for transactions that are naturally bi-directional.
PubSub+ is made precisely to meet these needs in a distributed environment.
Developers benefit from:
SQS, SNS, and EventBridge cannot be deployed inside a VPC in the AWS cloud, so these “public” services might put your data beyond your governance perimeter.
AWS proposes that an enterprise interface with these services using a VPC endpoint and a secure link, but SQS, SNS, and EventBridge services are “public” by design. That said, when messages leave your VPC into the AWS public services realm, you lose direct control over the location of your messages because AWS will replicate data across zones/regions to deliver their promised availability.
It also gives you control over how you move your data out of the AWS cloud, lets you glean relevant events/messages, and lets you push data to relevant AWS services.
These options help contain in-motion data within your operations and governance perimeter.
This airline needed a solution to distribute flight events across their various applications and lines of business in real-time, and it had to align with its cloud-first strategy. The main objective was to put batch data from Amadeus (a leading booking and operations platform) into motion and use services in the AWS cloud to action on key events.
PubSub+ Cloud interfaces with the customer’s VPC in AWS cloud and streams events and data between their IBM mainframe-based applications and AWS services.
The IoT sensors on a city bus publish MQTT messages to a topic that contains the bus’s current coordinates.
SNS requires pre-determined topics configured and coded into the publishing/subscribing applications. Without wildcard character support, a restriction on the number of topics per account and a limit of 200 filter policies per region, SNS cannot support a use case where each new latitude/longitude pair is the “topic” on which a city bus transmits data. On the consumer end, systems can filter down on wildcards and not be restricted by a limitation on a number of filter policies per region.
With 5,500 buses constantly sending messages with changing topic destinations, it was impractical to build an efficient solution for this using AWS SQS, SNS, or EventBridge.
With PubSub+ our client can gather and filter down actionable data from all the buses in real-time. They filter relevant data and process it on-premises and also in the AWS cloud to make real-time decisions on their network.
By making cloud-native services pervasive, AWS has enabled mature enterprise customers to build complex workloads with high agility and with great impact. Customers that are working to strengthen their hybrid cloud architecture (while building a path to multi-cloud) will benefit from a high performance, standardized, and governance-ready data movement layer.
PubSub+ is field-proven in enhancing the value of our customer’s investments in AWS and will help extract value from your entire application ecosystem, including microservices, SaaS, cloud services, legacy apps, mobile devices, and the IoT.
Try out Solace PubSub+ for free on AWS Marketplace.
An engineer by education and a marketer by profession, Gaurav was a senior manager in Product Marketing at Solace until 2021. He developed product strategies, campaigns and content that helped customers appreciate the differentiating functionality and business benefits of Solace’s event streaming and event management products.[position] => [url] => https://solace.com/blog/author/gauravsuman/ ) )