Bringing Big Data Architects some Solace

Most people (correctly) think of big data as being about Hadoop – large scale clustered data storage and BI. As a company focused on messaging and other kinds of real-time data movement, we’ve gotten used to the question: “So … what’s your play in big data?” This InfoWorld article about the drive toward real-time business intelligence hits on the answer.

Big data is currently in its early innings, with most deployments roughly analogous to where batch-loaded IMS databases were before relational databases were broadly adopted for large-scale OLTP. That is, most people are loading log files or other data sets at intervals “after the fact” thus limiting use cases to ones that get value out of looking at historical information to draw conclusions about the business.The leading edge of big data architects are striving to keep their Hadoop stores more current by aggregating and analyzing real-time data as things happen. This unlocks many more use cases based on opportunity-driven decision making.

While there are plenty of ETL and open source tools you can use to bulk load blocks of past information into big data stores, tackling real-time big data requires different technology. The article explains:

“This requires that jobs used to collect and transform data from the sources be modified to support faster data velocity. If for your business, the right-time upload of customer transactions means they must be processed every five minutes, you should be able to use traditional high-performance batch ETL technology with jobs running frequently on smaller data sets. If however you must collect geolocalization information with a 10-second maximum lag, then streaming or messaging technology will become a requirement.”

And that, in a nutshell, is our play in big data. We are the world’s highest capacity way to get real-time data into big data clusters using streaming or high-speed messaging techniques. You can learn more about how leading companies are integrating real-time big data into their overall big data architecture through our big data web page.