Big data is not working. The industry today is worth in excess of $122 billion, yet Gartner predicts that 2017 will see 60% of all big data projects fail. In research conducted by Capgemini and Informatica, meanwhile, just 27% of organizations said that their big data projects are profitable, 45% said their efforts are breaking even, 11% that they are losing money, and 12% that it was too soon to judge. Ultimately, what we are seeing is that it is easy to invest in big data, but using it is a different proposition entirely, and more and more data is being sucked into a money pit from which it will never return.
The reasons for these failures are varied, with everything from a lack of technical expertise to ineffective leadership being blamed. One major contributing factor, however, may be the obsession with quantitive data at the expense of qualitative data. Organizations are essentially marginalizing the human element and context that must accompany numbers for them to be worth anything.
Big data is generated by the millions of touchpoints that organizations have with customers. It provides quantitative information from which analysis can reveal insights that lead to new efficiencies and improved processes. For example, traffic data and predictive models have enabled airlines to precisely match aircraft capacity with seat demand while slashing their service on routes that rarely reach capacity. Planes now average about 83% full as a result. This is fairly easy to do simply by looking at historical patterns of behavior, yet while quantitive analysis can achieve results when it comes to these examples, if you are trying to use data to better understand customer behavior and trends for marketing particularly, as many are, more information is needed. You need to understand the human emotion behind the numbers. This is perhaps best evidenced by polling errors in last year's EU referendum and US presidential elections, where the absence of public sentiment in the majority of statistical models meant that the majority got it dramatically wrong.
Thick data/qualitative data bridges these knowledge gaps. It provides the context that enables you to understand why something is the way it is. Ultimately, a relationship between a stakeholder and a brand is based on emotion. It is not rational. You have to understand the quirks of human behavior to predict how an individual’s relationship with your service or product will evolve over time. Without this understanding, patterns you may uncover that suggest people will behave in a certain way could be based in a world that no longer exists. The trends you've identified may be a flash in the pan, and as a result your campaigns will neither appeal to those you may have thought to have been loyal customers, nor potential new ones.
This is exceptionally complicated, and it is still too early to leave it to machine learning, as you often can with big data. People's actions often make little sense, even to other humans, much less machines. Fear, greed, selfishness, love, desire... they are too hard for an algorithm alone to understand. Thick data is generated through primary and secondary research in the form of surveys, focus groups, interviews, questionnaires. It comes from ethnographers, anthropologists, and others who are highly experienced and trained at observing and analyzing human behavior and its underlying motivations. This enables them to reveal people’s emotions, stories, and models of their work and therefore which products they are most likely to buy, what price they will pay, and so forth.
Lego is one example of thick data at work. The toy giant was on the brink of financial collapse in the early 2000s when it decided to engage in a major qualitative research project. Their aim was to understand why the emotional needs of children' at play weren’t being met by Lego’s current offerings, with children across five major global cities studied to get an understanding of their emotional needs in relation to Lego. Hours of video recordings of children playing with the bricks were studied, eventually revealing a pattern of behavior. It turned out children were passionate about the play experience, the creating and using their imagination, rather than the instant gratification of toys like action figures. Lego needed to abandon its strategy of releasing new action figures and toys and revert back to basics, marketing instead its traditional building blocks.
When we rely solely on big data, we end up with a warped sense of the world in which human beings are simply numbers to be fed into an algorithm. This is not to say that it is useless, nor that in many cases it can be used alone. It is still a powerful and helpful tool that companies should invest in. However, companies should also invest in gathering and analyzing thick data to uncover the deeper, more human meaning behind big data. They need to work in tandem alongside one another, ethnographers and data scientists working side by side to make sure that big data has its groundings in actual human behavior, not just what a machine thinks it should be in perpetuity.