Scaling Data Movement when Planning for Big Data

Wikipedia defines big data as “a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools.” In the majority of cases, that same big data storage requirement comes with a big data movement challenge. Just as companies turn to new data management approaches to capture, store and analyze their very largest data sets, they need to look at new technologies for moving data at big data scale. The prior generation of enterprise messaging middleware such as MQ or JMS simply can’t keep up with the volumes that come with big data requirements.

Solace and Big Data

That requirement is at the core of Solace’s founding vision. More than 10 years ago, we saw that three long term trends  would lead to the need to move and manage an increasingly massive volume of data:

• Globalization of business, driven by the internet
• Rapid expansion of computing endpoints, driven by mobile and machine-to-machine (M2M) connections
• Exponential growth of data volumes driven by business process automation, complex event processing and decision making tools.

We chose to tackle the problem with lightning fast hardware technology instead of the “more clever software” approach that had been the hallmark of the incremental advances made in the middleware market over the last 30 years. We’d seen this same approach successfully eliminate high-volume computing choke-points for decades, most notably in internet switches and routers, and more recently in load balancers and security devices. Middleware is the foundation of every distributed application architecture, and we felt it needed the same treatment.

We delivered our first hardware product in 2005, and since then we’ve given a broad swath of companies the extreme capacity and rock-solid reliability they need to tackle the size and scale of big data applications which, by the way, will be tomorrow’s mainstream applications.