Entries by Ken Barr

Bridging Hybrid Cloud and Multi-Cloud Environments with Solace

With this blog post I will describe and demystify the prominent orange lines in this “marketecture” diagram, i.e. the means by which you can link or “bridge” multiple clouds with Solace open data movement technology. I will walk you through the general use cases and drill down to the actual implementation details including example configurations.

Topologies

In most cases, these inter-cloud links will be implement with Solace’s bridging feature, which provides robust fault tolerant links that support message filtering, compression and transport level security. Generally speaking, a Solace bridge is an embedded specialized client that can transfer messages from one message router or HA cluster to another, either one way or bi-directionally.  Solace bridges can be established between any combination or number of virtual message routers and hardware appliances.

Use Cases

Let’s look at some use cases and map them to commonly understood message exchange patterns, then see how each is implemented.… Read the rest

Solace in Azure Quickstart

Earlier this month AWS announced the release of an AWS Quick Start for the Solace VMR.  Now there is also a quickstart on the Azure Quick Start page.

The VMR is a software message router that is fully compatible with the Solace message appliance and Solace SDKs, as well as open source AMQP, JMS, MQTT, REST and WebSocket.   The VMR provides message-oriented middleware with class-leading reliability, performance, scale and robustness.

This Azure Quick Start allows you to select between “Single Node” or “High Availability” deployment models.  The single node deployment model can use the community edition of the Solace VMR and provides a very simple way to evaluate all the functionality a non-redundant instance of the VMR has to offer. The high availability deployment model requires a time based evaluation instance of the VMR and adds the robustness of Solaces high availability solution.

Both of these deployment models utilize the simple template driven Azure Resource Manager (ARM) interface to quickly and easily deploy complete solutions.… Read the rest

Solace Available via AWS Quick Start

This week AWS announced the release of an AWS Quick Start for the Solace VMR.   The VMR is a software message router that is fully compatible with the Solace message appliance and Solace SDKs, as well as open source AMQP, JMS, MQTT, REST and WebSocket.   The VMR provides message-oriented middleware with class-leading reliability, performance, scale and robustness.

This AWS Quick Start is one of their “automated gold-standard deployments” which shows how to create a fully integrated production deployment of a Solace VMR high availability cluster.  Solace has had multi-cloud offerings for some time, but this Quick Start allows you to more quickly and easily deploy a fully configured HA cluster. We worked with AWS architects to develop and validate this deployment in an optimized configuration that eliminates the need to worry about the details and mechanics of configuring the cluster yourself.

Solace will continue to work with cloud providers to provide simple but complete deployments to make our multi-cloud offerings even easier to deploy and start working with in minutes. … Read the rest

Replay Services in Enterprise Messaging

The ability to replay previously-published data to some set of consuming applications is a common requirement of many enterprise messaging and data movement backbones. Often people think about replay in these narrow terms.  They look at product capabilities and fit their needs into the functionality offered, instead of thinking holistically about the actual, frequently multiple, replay requirements of their business.

For example, I have heard discussions around using the Kafka broker as an enterprise message replay server because of its ability to replay messages. Apache Kafka is a write ahead commit log server that has been successful in aggregating typically low value data for ingestion into Big Data solutions. While Kafka technically supports replay, its capabilities are only a subset of the typical needs and you need to build tooling around it to make it useful.

In this blog I will discuss the many use cases and requirements for replay in an enterprise and compare Apache Kafka’s capabilities to those of a purpose built enterprise replay server, specifically Tradeweb’s Replay for Messaging for which Solace has a tight integration in the Replay for Solace edition.… Read the rest

What’s New with the Neon Release of MQTT Paho

In June 2016 Eclipse announced the Neon, (1.2.0), release of the MQTT Paho Java API. The new functionality provided is automatic reconnect and offline buffering (allowing publish to be called when the application is not connected) for the C, Java, JavaScript and Android clients.  As well as WebSockets support for Java, Python.

Solace continues to support open standards and open APIs by investing engineering effort to ensure these APIs interop in the most efficient manor with the Solace message router. In this blog I will look at the new auto-reconnect, high availability and publish offline features and show how they interop with Solace message broker to provide an end-to-end highly available solution.

Click these links to learn more about Solace’s general support for MQTT or high availability features, or to obtain a full working example on Github.… Read the rest

Technical Look at Using Solace as a Channel for Apache Flume

In a previous blog I explained what the Solace channel for Apache Flume was and how it could be used to streamline data workflow from enterprise message bus to big data infrastructure and vice-versa.  In this blog I will describe how the Solace channel works and what are the technical advantages compared to memory or file channels.

Solace Flume channel receiving Flume Events from Source.

solace-flume-channel-post_1The Flume Source interacts with the Flume channel with 4 methods calls:createTransaction(), doPut(Event) and finally doCommit() or doRollback().

The Flume Channel uses the Solace message router session based transactions to implement the channel transactions.  For more information on Solace session based transactions see Using Local Transactions

If the channel receives doPut(Flume.Event) it takes the Event and transposes it into a Solace message.  This is done by taking the Event header name value pairs and placing them into a Solace message header map, then taking the Events body and writing it as a byte buffer into the body of the Solace message. … Read the rest

Solace Integration with Apache Spark MQTT Utilities

apache-spark-mqttPreviously I had published an integration guide to integrate Spark Streaming into the Solace message bus via JMS.  This method of integration provides a plush set of integration options including Topic and Queue messaging patterns as well as enhanced security with TLS.  Integration with Spark Streaming via JMS does require extending the Spark custom receiver to support JMS.

If you are looking for a simple integration without having to write any code, then MQTT might be a better fit for your needs, as the Apache Spark that is distributed with Hortonworks, Cloudera and other Hadoop releases comes with a streaming MQTT interface built in.  As of SolOS 7.1.1, Solace makes it easy to integrate MQTT clients into the Solace message bus.   Spark provides the MQTTUtils Class that wraps up and provides a simple interface to receive messages from an MQTT broker and maps received messages into a Spark context.  The MQTTWordCount example that comes with Spark 1.4 and above shows how this is done.… Read the rest

Solace as a Channel for Apache Flume

As per the flume.apache.org: “Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.”

So why would you need anything other than the already available components of Flume to transfer data into your Big Data infrastructure?  Why not just use existing Flume Source and Flume Sinks to connect Flume to your enterprise messaging solutions?

Well the next picture starts to show the problem.  To scale Flume you need to stitch together several Flume Agents in varying hierarchical ways with point-to-point TCP links.  If your data has high value, next you will need to add fault-tolerance and high availability meaning you will need to add disk access and redundancy at each level. … Read the rest

Solace Achieves Cloudera Certification

Solace Systems Message Routers have been deployed by innovative companies of all sizes as the messaging foundation for their enterprise services bus. Our technology ties together applications across datacenters via event driven data flows, so their most mission-critical high volume data is already flowing through Solace Message Routers. As a result, when these organizations embrace new technologies they call upon us to help integrate them into their ESB.  It is as part of these efforts that we have been working for some time with Cloudera as an enterprise grade big data solution. Using Solace to feed Cloudera data workflows has given joint customers superior insight into any data, from any app, in real-time, with an enterprise mindset.

cloudera-certified-logoBased on these successful deployments we decided to formalize our integration with Cloudera to make the integration easily repeatable for our joint customers, and we are thrilled that we recently achieved certification with Cloudera’s Enterprise Hadoop solution (CDH).… Read the rest