How to Achieve Cloud and On-Premises Integration

The Situation You Face with Cloud and On-Premises Integration

To create a data distribution fabric that allows you to deploy applications in any cloud, you need to cover three types of connectivity:

Within individual clouds
Hybrid cloud, i.e. integration between on-premises apps and your cloud-native apps and services
Multi-cloud, i.e. connectivity between different clouds

Steaming Events within Individual Clouds

Most enterprises have in place one or several on-premises message brokers like IBM MQ or JMS, which can be based on open source or provided by a vendor.

When it comes to deploying new applications in a public cloud, many enterprises use multiple regions of a given cloud provider’s network and/or have a strategy that involves more than one provider’s public cloud – either for commercial or technical reasons, or simply to avoid the rest of lockin. Many will also have a private cloud, which could be running Kubernetes (which of course you can run in the public cloud too), OpenShift, Tanzu, Rancher or other. In any event, you’re talking about an environment that’s just grown from a simple on-premises deployment to one that spans multiple cloud technologies and locations.

Since the applications you deploy into the cloud will likely be 12 factor applications, microservices, or Function as a Service (FaaS), you will need event streaming in each of these clouds to connect these components. The path of least resistance out of the gate as a point solution is to use a given cloud’s own messaging offering – SQS or SNS in AWS, Service Bus in Azure, Google Cloud Messaging in Google Cloud Platform, ActiveMQ in OpenShift, etc, but most people have decided (for good reasons) that they are not going to use their legacy messaging in the cloud – mostly because they don’t support modern application requirements.

In a three-cloud architecture, you would end up with a trio of new and different technologies, each with its own API which complicates development and prevents you from easily moving applications from one cloud to another. It can make governance and management pretty complicated, and even “as a service” offerings demand that you establish governance. But using the cloud native services is a fast way to get up and running, so let’s assume we go with that and see what happens next.

Streaming Events Between Clouds and Datacenters

Applications don’t exist in a vacuum – they consume and produce data that comes from or goes to other applications, most of which live on-premises since that is the bulk of the IT base for enterprises today. So you want to have a distributed application span your existing IT assets and your new multi-cloud environment. How do you connect these legacy on-premises applications to the cloud to create a hybrid cloud integration architecture?

If you have chosen 3 different event streaming technologies in your 3 different clouds, then you need 3 different clouds, then we need 3 different ways to move data from your on-premises environment to the cloud. Not only that, you also end up with different addressing (e.g. topics) for the same event in these different clouds, so how do you manage these events across your environments and know what data is carried in what events? Or how to secure it? Since the data you chose to move to the cloud is always changing, along with the cloud-based applications themselves, how do you meet the dynamic data movement needs of these disparate components in a coordinated way? If you currently have both JMS and MQ , for example, you’ll need six different interconnects (JMS times 3 clouds plus MQ times 3 clouds). Such integrations don’t come “out of the box” for the cloud-based services, so you’ll need to build each one, and make sure they are fast, manageable, secure, fault tolerant, etc. – all the properties you expect of data movement infrastructure.

This is starting to look complicated…

Streaming Events Between Multiple Clouds

If you run applications in multiple clouds, you will likely need to connect those applications so they can share data. Again, if you have 3 different cloud messaging technologies then you need 3 different ways to connect them – each of which you need to build, scale, secure, and administer.

This looks like the pairwise tightly-coupled integration of old that we replaced with (now old) message-oriented middleware like MQSeries and Rendezvous because they were too fragile and complex – only now instead of connecting applications we are connecting technologies. More complexity.

Problems with Today’s Techniques

In this approach, what other things are there to consider?

Application Portability. For many of you, the freedom and flexibility of running applications in the cloud is very important as you don’t want to get locked in to one cloud – you’ve been there before. Even if you aren’t worried about using a single cloud, the possibility of developing apps in the cloud and then deploying them on-premises remains attractive. In general, application portability is valuable in a changing world so difficult to predict. Most cloud services either offer proprietary APIs built on a RESTful protocol, or they offer a limited JMS API that also uses HTTP under the hood. That means in most cases, applications are developed to run in that cloud, and can’t be ported to another without significant effort.
Enterprise Grade Features. Many application developers today are used to sophisticated messaging functionality and exchange patterns, and they expect those features to exist in this new and improved cloud world. Features like pub/sub and queues from the same API, asynchronous I/O, wildcard subscriptions, session-based transactions, message priority, message selectors, dead message queues, and time-to-live… just to name a few. They expect messaging to support their chosen application runtimes such as JEE and more modern tools like Node.js, Spring Cloud Streams, and Spring Boot, and they’re disappointed to learn that such features and integrations are often not supported by cloud-based services.
Performance & Bandwidth Costs. Many cloud-specific messaging services use HTTP to send/receive messages. This doesn’t allow for streaming, so you have to send one message at a time and wait for a response before sending the next message. Some let you batch messages but your application needs to handle that, which means more complexity. In any event, message rate is limited by round trip time, which introduces high and inconsistent latency. Shared tenant messaging services in particular have extremely variable latency that can be unacceptable in a microservices architecture. Most public clouds also charge for outgoing bandwidth so intelligent message routing and filtering is needed to avoid surprisingly high monthly bandwidth charges. Capabilities like WAN optimization and compression are almost always missing from messaging services, too, even though they are critical in a hybrid/multi-cloud architecture.
Management & Monitoring. You will usually get some visibility into the performance of your applications from your data movement infrastructure, but in this case, you will have one such monitoring and management system for every cloud you are using. Also, in this type of a hybrid/multi-cloud environment, there are communication paths that fall outside the scope of any one cloud provider’s service (e.g. between clouds), so you’ll need to build custom monitoring solutions that will each need to be configured, monitored, engineered, and managed on an ongoing basis.
Simplicity & Robustness. The more heterogeneous an environment, the more complex and the more fragile it is. In this case, there are many different technologies with many different connectors that lead to ongoing operational challenges.
Custom development and integration. The cloud is supposed to make things easier, to let you spend less time and money on infrastructure thanks to more “flip the switch” capabilities. The use of multiple messaging solutions demands more of the custom development, integration, and operations you want to leave behind!

Let’s Talk About a Better Way: Event Mesh

Gartner has been talking a lot about an architectural layer called “event mesh”; a new kind of infrastructure that links event brokers to enable the event-driven, intelligent and efficient streaming of information between applications, devices and user interfaces across environments and around the world.

We happen to excel in this area, and have helped scores of enterprises implement event meshes, so I’d like to walk you through what it looks like in practice.

Streaming Events Within Individual Clouds

Solace technology can be “one click“ deployed as a service in all leading public clouds such as AWS, Azure and Google, leading Platform as a Service offerings such as Kubernetes, OpenShift, GKE, Tanzu, and more by means of the PubSub+ Platform. That means your applications can make use of the same eventing services and APIs no matter where they’re running, giving you a uniform event mesh that spans clouds.

It also means you can move your applications between any public and/or private clouds without any code changes, which means all your DevOps for provisioning, monitoring, alerting, and security are the same in each of your clouds. That way, if a policy changes to favor a new cloud, or you find benefit in going to some other cloud, we’ve got you covered.

Streaming Events Between Clouds and Datacenters

Solace PubSub+ Platform serves as the foundation of some of the world’s most demanding use cases and deployments for two decades, and of course all of our customers are migrating some information and/or workloads to the cloud.

We offer four out-of-the-box ways to extend your current applications to a multi-cloud architecture:

For Solace customers this multi-cloud fabric just “drops in”: your on-premises PubSub+ event brokers seamlessly connects to your PubSub+ cloud services to create your event mesh. The sophisticated message routing and integrated WAN optimization capabilities you are probably using today work the same way when extending to the cloud.
To access data flow without changing existing applications, our technology lets you tap into IBM MQ or JMS and transfer information to your new application(s) in the cloud(s) without any changes to your on-premises applications. We accomplish this with our wide range of connectors that interface to these legacy systems. Data produced by cloud applications can also be streamed to your existing ESB for delivery to existing applications. Streaming events over a secure connection ensures fast, reliable, WAN-optimized movement of your real-time data either to or from any cloud.
Existing enterprise applications or 3^rd-party applications that use JMS, JCA or runtimes such as application servers or Spring frameworks can connect directly to Solace event brokers on-premises to move their data to the cloud.
We provide a variety of open protocols for new on-premises applications to connect directly to a Solace event mesh if that’s the direction you want to go. We support REST, MQTT, and AMQP 1.0 as open interfaces and APIs in many languages. That way you can take advantage of new data movement technologies in your non-cloud apps to modernize and simplify your on-premises architecture.

Most enterprises have applications developed in many different languages and frameworks, and the cost and time to change them is massive. Being able to connect these applications to the cloud or migrate them to a simpler on-premises architecture to connect them to the cloud is a big win.

Streaming Events Between Multiple Clouds

Solace technology reliably and securely streams events between cloud providers, and between regions of the same cloud without needing any adapters and using a dynamic routing protocol so you don’t have to configure routing rules. We support non-persistent quality of service for use cases that demand high speed streaming and request/reply; as well persistent delivery to ensure that there is no message loss for your critical messages that need to be delivered between clouds regardless of WAN fragility, lack of bandwidth, etc. We also compress event streams by up to 80% and support de-duplication over these cloud links to slash your bandwidth usage. Since most public clouds charge you for the bandwidth that leaves the cloud, this data compression translates to serious savings every month – who wouldn’t like that?

Other Advantages of an Event Mesh

Here are some of the other advantages of a Solace event mesh.

Event and Stream Management. The PubSub+ Platform is the first platform that gives you the ability to manage the topics and schemas flowing through your hybrid/multi-cloud event mesh with PubSub+ Event Portal, which provides a single place to define, discover, govern and life-cycle manage your events across your entire enterprise. This is a critical component to promote reusability of events in your organization and to democratize access to realtime data.
Application Portability. Implementing an event mesh that spans your public and private clouds means the same event streaming services are uniformly available to your applications in all these clouds, so you can easily move applications from one cloud to another. Solace provides many open APIs (JMS, JCA, MQTT, REST and AMQP1.0) and integrates with many cloud runtimes (like Spring Cloud Streams, Spring Boot) to also avoid application lockin to our technology. We believe in open interfaces and freedom of choice.
Enterprise Grade Features. If you build your event mesh with our technology, you’ll benefit from all the key enterprise features you are used to in designing your applications, along with features that are specific to applications like price/odds distribution, asynchronous web push, IoT connectivity and more so you can build sophisticated applications with diverse communication needs all on a single platform.
Performance & Cost. The WAN optimization capabilities we support can dramatically reduce the cost of bandwidth leaving the public clouds, which is an important consideration as you watch your savings accrue month after month. This same transport compression is not only available for use between Solace event brokers, but if you use our APIs, you can also benefit from this bandwidth reduction for applications connected from outside the cloud.
Management & Monitoring. A sophisticated web-based admin tool called PubSub+ Insights lets you view real-time statistics and status about message rates, queue depths, buffer usage and lots more at the level of a virtual broker, specific queue and even individual connections. You can set thresholds so you’re alerted about conditions such as slow consumers or consumer apps you need to scale, slow links between clouds, and disks that are filling up. Our Insights tool enables integrated monitoring and alerting and can be integrated with your firm’s existing authentication and authorization systems.
Simplicity & Robustness. It has always been our goal to provide a single event streaming solution for all of your event distribution needs. That’s why we’ve integrated all the functionality for hybrid and multi-cloud deployments into a single technology stack, managed by a single pane of glass using Mission Control and Insights for ease of deployment and a simple, robust architecture with dynamic routing, fewer moving pieces, and fewer points of failure. Solace technology includes features like high availability, replication for disaster recovery, congestion control mechanisms and more – all to provide a robust event streaming and management platform you can rely on.
Out of the Box. The beauty is that all this hybrid and multi-cloud connectivity complete with “single pane of glass” management works right out of the box. No need for any custom development for integration or monitoring – it’s all ready for you to use.

Focus on the Big Picture When it Comes to Cloud and On-Premises Integration

Its tempting when moving to the cloud to take the path of least resistance and use technology that is native to that cloud – makes you feel like you are moving fast. But you could end up with buyer’s remorse because your system becomes very complex, fragile and restrictive over time, such that you are no longer moving fast. Getting the event streaming piece of the puzzle right from the get-go can save you the large hidden costs associated with heterogeneity and simultaneously enable future options for application evolution. Its important to focus on the big picture when it comes to cloud and on-premises integration rather than just the tactical – a better solution is readily available.

Providing this type of simple hybrid and multi cloud event distribution fabric is what we at Solace are committed to doing – so you can focus on and be efficient at developing and deploying your applications – wherever you want to develop or deploy them.

We are honored to be working with some of the most prestigious brands in the world to help them with their on-premises data movement needs and to now be helping them with their transition to the cloud. For one customer, we provide elastic capacity for odds distribution to mobile applications; for another we provide IoT connectivity to vehicles; for another we provide hybrid connectivity for end of day risk calculations. All of these are awesome applications that are changing their business.