Business Brains

Information overload: when is too much data too much?

Posting in Cities

Is the process of ingesting petabytes of data about every online transaction, every tweet, and every 'like' button clicked making companies any smarter about the people they seek to serve?

Readers of this blogsite as well as the rest of SmartPlanet will know that we have been tireless proponents of analytics to solve all kinds of problems -- understanding customers, markets, cities, health issues, transportation issues, and even searching for extraterrestrial life, to name a few.

But, particularly for businesses, is the process of ingesting petabytes of data about every online transaction, every tweet, and every "like" button clicked making companies any smarter about the people they seek to serve?

Two points here: First, as discussed before on these pages, there's a school of thought that suggests that spontaneous decisions often are better than decisions arrived at after hours or days of intense analysis.

Then, there's that old maxim of quality over quantity. Do attempts to collect and analyze piles and piles of data about customers really lead to a full understanding of customers? Or just piles and piles of data, some of it potentially intrusive?

A new report in Knowledge@Wharton talks about the privacy issues in tracking customers so closely, but also brings out another conclusion: that while technology has made it possible to collect massive amounts of data on customer habits, it also has resulted in "sloppiness" in the way the data is handled, rendering it  meaningless in many situations.

As Wharton marketing professor Peter Fader expressed it:

"In the past, data was scarce, so companies carefully analyzed what they had, Fader notes. Firms like Pillsbury and Procter & Gamble used in-house experts to track repeat purchases, augmenting what they found with demographic data and results from expensive surveys. Direct marketers compiled lists that they tested and re-tested. Insurance companies learned how to classify their customers into high and low-risk groups. 'Companies were very shrewd with their data because it was so hard to get,' says Fader. 'Today, it's drinking from a fire hose. We have everything, a million variables ... and the data miners are trying to use all of it.' In the rush to collect and connect, he adds, the data itself has become the focus -- not the customer."

Fader points out that many online purchases are too random to draw meaningful conclusions. "How granular should our messaging get? The one-to-one is going too far. Just because we have two or three data points on a customer doesn't mean we've nailed them down." Often, he adds, "taking the data too literally draws misleading conclusions and puts out messages that are not only ineffective but almost creepy."

The takeaway here is that all the information in the world is available to businesses that want it; but the ability to be selective, careful and smart with data will separate a company from others that are buried in piles of bytes.

Share this

Joe McKendrick

Contributing Editor

Joe McKendrick is an independent analyst who tracks the impact of information technology on management and markets. He is a co-author of the SOA Manifesto and has written for Forbes, ZDNet and Database Trends & Applications. He holds a degree from Temple University. He is based in Pennsylvania. Follow him on Twitter. Disclosure