Is the real-time web streaming data in the wrong direction?

Examples of web streaming have become rather predictable and yawn-worthy. It’s always some variation of streaming real-time stock market data, news and status updates from the cloud to your browser, tablet or phone – classic filtered fan-out data distribution. Sure, there are a few upstream bits like the character inputs used for real-time keyword search completion, or chat applications, but the upstream data is a trickle compared to the fire hose coming downstream. However this model is beginning to flip directions and applications are more frequently streaming large volumes of upstream data with a downstream trickle.

Consider how most Big Data is being collected at the server side today. Click streams, log data, activity streams, search queries – they are all pouring into Kafka, Scribe, or Flume and ending up in a variety of big data repositories. As users increasingly run thicker smartphone, tablet or desktop apps the view from the web server becomes less and less complete. Take for example a shopping site that monitors user behavior to optimize user experience or increase click-to-sales ratios. Before rich apps, it was the clickstream that contained all the needed information to know who looked at an item, who put the item in their shopping cart, and who actually purchased or abandoned the cart. With a rich app on your phone or tablet, commerce sites can now track new events like:

• who swiped which part of the screen?
• who likes to contemplate the purchase button, but never clicks?
• who is scrolling up/down versus right/left?
• which color, position or shape of button attracts more activity?

It’s now possible to put these apps in verbose mode and stream huge amounts on interaction data upstream for real-time and historical analysis. Imagine real-time heat-maps of every user interface across all client devices at your fingertips. All this data improves the small but important trickle downstream – perhaps recommended specials or dynamically generated menus that perfectly target each user.

Of course, it isn’t all about shopping. There are endless other telemetry-style data collection use cases. The client app can just as easily be a vending machine, railroad switch sensor, electricity smart meter, or your car. Shouldn’t the HD back-up camera in your new car take snapshots every 30 seconds and publish them to a private real-time stream that you can monitor and archive? It’s sitting in your driveway, why not put it to work as a home security webcam?

Hit and run drivers and burglars better beware the upstream real-time web.