Does Your Data Have a “Best Before” Date?

Today I presented as part of a webinar about real-time streaming analytics alongside Forrester analyst Mike Gualtieri. He described how streaming analytics differs from traditional analytics where data is collected, loaded into an analytics engine and analyzed after some period of time.

I was intrigued by Mike’s characterization of the “perishability” of information, that is, how much opportunity for insight is lost as different kinds of events age. Equating this observation to everyday life, I bet you don’t usually check the expiration date on a bag of chips. But if you’re like me you do glance at the stamp on milk, and if the date is coming up you’ll give that gallon a sniff before pouring it all over your Lucky Charms.

Which events in your business have long shelf lives, and which ones become useless before you can identify what happened or have a chance to act?

Everything depends on context, of course, and streaming analytics introduces many new opportunities for insight and action. For example, consider a credit card customer named Martin making a purchase. That single purchase has opportunities for many types of insights, each with a different degree of perishability.

The credit card company can capture Martin’s purchase event for long term storage, and in context of all his purchases develop a model of his buying patterns and consumer behavior. That model could be combined with his credit score to establish a risk profile, which might lead to an offer to increase the Martin’s credit limit the next time he is within 20% of his cap. They could also sell Martin’s profile information to other businesses looking for specific consumer personas, which might get him a great deal on a trip to Cuba. All of this is traditional after-the-fact analytics, and that specific purchase event has a very long shelf-life as a piece of the puzzle that makes up that consumer’s behavior.

That same purchase event presents even more insights and opportunities for action as it is being captured. For example, let’s say Martin jumps on that offer of a discounted trip and goes to book a flight.

• If Martin has never traveled outside the country (as far as the credit card company can tell from their own analytics and profile of him, anyway) they could call or text him before approving the transaction to confirm that it’s really him making the purchase.
• The credit card company knows Martin only buys trip insurance within half an hour of making travel plans, so they send him an email seconds after he books his flight.
• Shortly after he books his trip, Martin calls a cigar shop in Cuba to order cigars for delivery to his hotel. When the trip is booked, they could add Cuba to his fraud white list so purchases in Havana don’t also trigger fraud.

These opportunities for insight and action would have expired, or at least become far less valuable, minutes or hours later when the data found its way into long term storage.

I found this chart that Mike used during the webinar to communicate very helpful in understanding the varying perishability of events.

Source: Mike Gualtieri, Forrester Research

The perishability of an event is directly related to the amount of time you have to act on the insight derived. With traditional “capture, load and analyze later” analytics, you miss out on the freshest aspects of your events, so you’re limited to reporting on insights that are relevant for a longer period of time.

We should all be thinking about our analytics planning in terms of this perishability to ensure we’re getting the most out of our data, and that Martin won’t miss out on visiting old Havana.

If you want to catch a replay of the webinar, you can find it here.