Explaining “Big Data on the Move”

At the recent QCon conference in San Francisco, I gave a talk called “Big Data on the Move” in which I introduced the audience to some of the unique challenges, techniques and technologies associated with enabling the distribution of “big data” volumes of information. The room was full and my talk seemed to hold the attention of the notoriously tough QCon audience.

Special thanks to the fine folks at QCon for letting me share video of this talk. I hope you find it informative. I’ve included an index of some key points below the video.

(YouTube blocked at work? Watch here instead.)

00:52   What’s missing from the current big data landscape
05:16   The three key types of “Big Data in Motion”
06:25   Big data capture and collection
09:22   Scaling horizontally vs vertically
12:56   Dealing with traffic volume spikes
14:56   Dealing with the “long fat pipe”
17:25   Alerting and streaming events
20:22   Fanning out data to a huge audience
22:35   Dealing with slow subscribers
26:12   Real-time data analytics
27:30   The role of in memory data grids or CEP engines
28:30   Real world use cases
28:55   Transportation sensor network
32:27   Homeland security
35:09   Trade automation
36:30   Telco customer service
38:25   Open Q&A