Posts

Larry Neumann

Globally Syncing the Popular Hazelcast IMDG

/in /by

This morning we announced a joint solution with Hazelcast, providers of the leading open source in-memory data grid (IMDG). IMDGs are an essential element of next-generation application architecture, and over 3, 000 companies — including several dozen Solace customers — use Hazelcast to keep data current across geographies.

hazelcast-solace-diagramThe new joint solution enables the replication of Hazelcast clusters across multi-cloud and hybrid cloud environments. That means as data changes in one Hazelcast node, Solace automatically disseminates that update to other nodes using compression, TCP optimization and route optimization for high throughput and low latency over the WAN. The solution is ideal for companies who need real-time data to be current across diverse geographical areas as data values change, such as financial institutions who need to sync real-time risk data across global financial centers and mobile carriers who want to give customers fast access to customer and operational data culled from distributed datacenters.… Read the rest

PR

Hazelcast and Solace Partner to Provide Applications with Real-time Data Across Diverse Geographies

/in /by

Most powerful WAN-accelerated messaging solution combines with fastest in-memory data grid to drive efficiencies in the processing of global data workloads

Palo Alto, Calif., October 25, 2016 – Hazelcast, the leading open source in-memory data grid (IMDG) with hundreds of thousands of installed clusters and over 16 million server starts per month, and Solace, the leading provider of open data movement technology, today announced a new joint solution which enables multi-cloud and hybrid cloud replication of Hazelcast clusters for worldwide operation. It enables enterprise grade reliability, WAN scale-out and low latency cluster replication.

Hazelcast is frequently deployed in architectures where caching, scalability, performance, data sharing, redundancy and fault tolerance are essential. By combining Solace’s store and forward WAN messaging solution with Hazelcast, organizations can reduce WAN bandwidth costs, minimise data update propagation time and insulate their infrastructure from network or data center outages. The solution is ideal for companies who need real-time data to be current across diverse geographical areas as data values change.

… Read the rest
Tom Fairbairn

What a Load of JSON: XML and other Text Data in The Solace APIs

/in /by

I recently wrote a series of blog posts that introduced the elements of Solace messages including message header properties and how to encode data in the payload.  Now I’d like to dig a little deeper and talk about using text messages in Solace. Text messages offer more than just simple string transfer: common formats such as JSON and XML allow the transfer of complex data as strings with readily available libraries to write, read and parse the data for you.… Read the rest

Tom Fairbairn

Inside a Solace Message, Part 3: Payload Data

/in /by

solace-message_payloadIn the first post of this series I introduced the structure of Solace messages, and in part two I talked about how Application Header and User Defined Properties can be used to decorate messages to aid processing the message data.  Now it’s time to talk about how the data itself is structured – how to put the payload in the message.

Don’t I Just Add the Data?

At first glance this question seems trivial: don’t I just put the data in the message?  That’s the point of a data agnostic messaging system, surely? Well, yes… and no.  I hope you weren’t expecting a different answer!

Solace Message Routers will transport whatever data you choose to put in your message (provided it fits within the maximum message sizes, 64MB for direct and 30MB for guaranteed)  But there are other considerations:

  • Is your network heterogeneous? For instance, are mobile devices part of your messaging infrastructure? 
… Read the rest
Larry Neumann

Don’t Be Embarrassed by Shrinkage (in your Market Data)

/in /by

The financial industry has a love-hate relationship with market data. It’s the lifeblood of their industry, but trading firms view dealing with it like grocery stores view constantly swapping the price tags on shelves – the cost is pure overhead and it doesn’t differentiate them from the competition. It has to be done and done well, but they want it done as cheaply as possible.

For decades, trading firms had two choices: get market data directly from tens to hundreds of constantly changing sources and figure out how to distribute it where it’s needed, or get it from consolidators that had them by the watermelons (to extend the grocery metaphor) in terms of cost and ability to switch vendors.

OpenMAMA is an open source initiative that addresses this problem by unbundling market data from its delivery. OpenMAMA does the dirty work of consolidating direct feeds and abstracts the delivery layer so you can consume it however you like.… Read the rest

DIshant Langayan

Understanding Guaranteed Message Publish Window Sizes and Acknowledgments

/in /by

One of the main advantages of Guaranteed Messaging (a.k.a.  Persistent Messaging or store-and-forward messaging), is that a receiver does not have to be available for the publisher to send messages as the Solace Message Router will persist the messages until the receiver becomes available again. The publisher is guaranteed that a message is successfully persisted by the Solace Message Router once it receives an acknowledgement for the message it sent. Persistence increases reliability but does add some overhead compared to Solace’s Direct Messaging, which provides no acknowledgments to the publisher.

Despite this overhead and the round trip times of acknowledgements, Solace Message Routers provide a high throughput rate for Guaranteed Messaging publishers. This high performance is enabled in the SMF protocol through the use of a Publish Window. In this post, I will introduce you to the concept of Publish Windows and their sizes, and how acknowledgements affect these windows. We will also look at when and how to adjust the sizes of the Publish Window using Solace’s Java API.… Read the rest

Ken Barr

Technical Look at Using Solace as a Channel for Apache Flume

/in /by

In a previous blog I explained what the Solace channel for Apache Flume was and how it could be used to streamline data workflow from enterprise message bus to big data infrastructure and vice-versa.  In this blog I will describe how the Solace channel works and what are the technical advantages compared to memory or file channels.

Solace Flume channel receiving Flume Events from Source.

solace-flume-channel-post_1The Flume Source interacts with the Flume channel with 4 methods calls:createTransaction(), doPut(Event) and finally doCommit() or doRollback().

The Flume Channel uses the Solace message router session based transactions to implement the channel transactions.  For more information on Solace session based transactions see Using Local Transactions

If the channel receives doPut(Flume.Event) it takes the Event and transposes it into a Solace message.  This is done by taking the Event header name value pairs and placing them into a Solace message header map, then taking the Events body and writing it as a byte buffer into the body of the Solace message. … Read the rest

DIshant Langayan

Blocking vs Non-Blocking Publishing with Solace Messaging

/in /by

In this post, I will introduce you to the concepts of blocking vs non-blocking publish available with some of Solace’s APIs. You will also learn when and how to use these two modes in your applications. This feature is supported in only the Java RTO, C, and .NET Solace APIs. The Java API only supports a blocking publish model. For the purpose of this post we will be using Solace’s C API to demonstrate code samples and concepts, with references to documentation on how the same can be achieved using other available Solace APIs.

The post also assumes the following:

Blocking and non-blocking publish can be used with both Solace’s Direct Messaging Delivery mode as well as Guaranteed (a.k.a Persistent) Messaging Delivery mode. In this post we use Direct Messaging Delivery mode and any differences with Persistent Delivery mode will be highlighted when necessary.… Read the rest

Ken Barr

Solace as a Channel for Apache Flume

/in /by

As per the flume.apache.org: “Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.”

So why would you need anything other than the already available components of Flume to transfer data into your Big Data infrastructure?  Why not just use existing Flume Source and Flume Sinks to connect Flume to your enterprise messaging solutions?

Well the next picture starts to show the problem.  To scale Flume you need to stitch together several Flume Agents in varying hierarchical ways with point-to-point TCP links.  If your data has high value, next you will need to add fault-tolerance and high availability meaning you will need to add disk access and redundancy at each level. … Read the rest

Ken Barr

Solace Achieves Cloudera Certification

/in /by

Solace Systems Message Routers have been deployed by innovative companies of all sizes as the messaging foundation for their enterprise services bus. Our technology ties together applications across datacenters via event driven data flows, so their most mission-critical high volume data is already flowing through Solace Message Routers. As a result, when these organizations embrace new technologies they call upon us to help integrate them into their ESB.  It is as part of these efforts that we have been working for some time with Cloudera as an enterprise grade big data solution. Using Solace to feed Cloudera data workflows has given joint customers superior insight into any data, from any app, in real-time, with an enterprise mindset.

cloudera-certified-logoBased on these successful deployments we decided to formalize our integration with Cloudera to make the integration easily repeatable for our joint customers, and we are thrilled that we recently achieved certification with Cloudera’s Enterprise Hadoop solution (CDH).… Read the rest