Introducing Open Data Movement

Today Solace introduced something we’re calling Open Data Movement. I’m going to do just two things here: define Open Data Movement, and give an example in pictures.

Breaking Down the Term

Each of the words are important and we’ve chosen them carefully.

Open

We see this as meaning free from proprietary lock in. Solace’s approach to real-time data movement favors:

• Open protocols – the defined and de facto standards that have emerged for posting, messaging and streaming, including REST, JMS, MQTT, AMQP 1.0 and WebSockets.
• Open APIs – where possible, we encourage use of open standard APIs like JMS, Paho or Qpid giving developers and enterprises maximum choice.
• Open to all the popular cloud environments – Solace works the same in private or public IaaS and PaaS platforms, including Amazon Web Services, Microsoft Azure, OpenStack, VMWare, Cloud Foundry, OpenShift and more.
• Delivering all the message patterns and classes of service – best effort, guaranteed or transactions for request-reply, publish-subscribe, fanin/fanout, and more
• Openly integrated with popular DevOps tools – Modern products need to be fully integrated from getting started with shared sample code in GitHub, all the way to automating the test, stage and deployment process in tools like Puppet or Bosh (and everything in between).

Data

All of the information that’s cranked out, collected and consumed by applications, connected devices and even we the people as part of the increasingly information-centric world we live in: sensor data, streaming data, big data, real-time data, reference data, the works.

Movement

The many and varied ways all that information actually moves between applications. Often it is message queueing, sometimes it is streaming, sometimes WAN optimization is needed, sometimes it is a simple point-to-point post from one app to another. Many people conflate all this to messaging, but it really is much more, and we find our clients like to think about their bigger problem as reliably moving data in support of their workloads.

Ultimately, Open Data Movement is about choice. Solace lets you choose from among all the clouds, all open protocol, all open APIs, all message exchange pattern, and all classes of service. I could go on and on in words about what this means, but I think the story is better told in pictures.

Open Data Movement in Pictures

Let’s take a look at several architectures of increasing complexity. At the end, I’ll compare how it works in Solace with how it would work with other technologies.

Simple Message Bus

First, let’s start with a simple messaging bus. Here are six applications all sharing information through a messaging broker or bus. All applications use JMS to send and receive messages. Any given interaction could be point to point, publish subscribe, or a continuous stream of updates (like to a big data analytics engine). It’s pretty easy to conceptualize, and lots of commercial and open source brokers can do deliver this to some level of performance.

Unlike Protocols

Consider the same scenario, with all applications still in one datacenter, but now the many applications are part of different use cases, that have been implemented using different messaging or streaming protocols. A browser app is sending data via REST to be captured in a database connected via JMS. An IoT applications uses MQTT from the devices to a server running AMQP. The datacenter assets are connected using a free mix of JMS, AMQP and MQ.

Without Solace this would get messy, as each exchange of unlike protocol clients and servers would need gateways or application code as an intermediary to the server. With Solace it looks very much the same as the one protocol case. Solace uniquely supports all the popular open messaging protocols (MQTT, REST, JMS now, MQ through integration and AMQP 1.0 coming soon). The Solace broker can automatically terminate a connection from one type of client and forward the message via a different protocol to one or more servers.

Bring on the Cloud

Next, let’s say it makes sense to move the IoT data collection to Amazon Web Services. The rest of the assets will remain in the datacenter, but the MQTT connections will move to AWS. All you need to do is fire up the AWS native version of Solace and the application behaves as before.

The Cloud is Really Many Clouds

Many of our customers want to take advantage of cloud architectures, but aren’t ready to fork their crown jewels over to the public cloud providers. As a result, most are implementing a private cloud IaaS like OpenStack or VMWare and a PaaS like Cloud Foundry or OpenShift. Some of our large clients are also engaging with multiple public clouds so they are protected against an outage in any one of the clouds.

In our scenario, as we distribute the workloads from the legacy datacenter environment to a private cloud and multiple public clouds, you can see that establishing a broker in each environment is all that is needed. Solace takes care of all the routing between brokers and can easily be scaled independently within each environment.

Adding Workloads

Once we’ve established and connected each of the clouds, adding more workloads doesn’t complicate things any further. The message bus handles the additional complexity, and you can easily scale the capacity within each environment as needed.

The Same Scenario with a Broker per Protocol

Contrast that will how this would work without Solace, if you used just one technology for each of these other protocols. Development would need to build bridges between each of the protocols and operations would also have to scale all the bridges to allow information to flow between the islands of protocols.

Now imagine scaling that environment? And making it fault tolerant? Seriously, this gets complicated fast! Many of our clients have literally thousands of applications, making the above look like child’s play. Taking out dimensions of complexity is a big part of what Open Data Movement is about.

Join the Movement!

So that, in pictures, is what we mean when we talk about Open Data Movement. Any applications, all your workloads, across and within all the clouds, using any protocols and APIs, to do any messaging patterns with any class of service. All with greater scale and reliability than any other commercial or open source option that gives you with just ONE of those protocols.