The Problems With Big Data

Big Data is brilliant, but what problems does it currently have?


We are all aware that Big Data is now an important part of businesses across the world. From the ways that they are making decisions to the split second choices that machines are making through the sensors attached to them.

However, we need to remember that Big Data, unlike many other business related trends, is still in its infancy, our generation are the founders of the movement. This means that it is not yet fully grown and the results of its use are not always perfect.

So what currently makes Big Data imperfect?

Big Data Sets Does Not Mean Big Information

We need to look at data sets in a scientific way. They are not just the pools from which information can be gathered, because much like fishing in a lake, if there are several different types of fish in there, you could catch any one of them.

The same could be true with data, with several uncontrollable variables often included within the data that is being analyzed.

One of the most common ways to get rid of these is through filtering the data, which in itself can skew the analysis. When removing some of the variables, you may be taking out the most important one or if you are trying to make the sample strictly controlled it could make the sample so small that the relevance of using Big Data becomes pointless as the results seem relatively obvious.

Data Is About Patterns

Finding correlations in data is about finding patterns within it. This by its very nature means that there is going to be a certain degree of bias within the system as the ability to find patterns is inherently biased because these are patterns perceptible to the human brain.

With complex data systems it is possible to find some of these without this bias affecting it too much, but in reality we see that many patterns are found through visualization, something that certainly requires mainly human interaction with the images to find a pattern. This may mean that several important patterns could be missed literally through human error.

You Need To Know About More Than The Numbers

Knowing that seeing a particular pattern and causal trends across a few different strands is the basic way that patterns are formed. If you can see that a certain object is being sold at a certain time of year or that people in a certain area are more likely to use a particular website is great, but in order to make any impact from this information, it is vital that the underlying causes of this pattern are known.

Why are people buying this? What makes people from that area visit that website?

Simply knowing that they are doing this may provide insight, but this is very shallow and does not allow a company to make the most of the opportunities presented to them through this information. If you don’t know why the data shows what it does, how are you going to make changes based on it?

There needs to be an understanding of the analysis beyond seeing that it exists, which at the moment is often something that data alone cannot do.

Despite these, it is worth mentioning again that Big Data is in its infancy. As it is not fully established, the chances are that many of these issues will be resolved in the years to come. We are not yet at the top of the curve of Big Data, in fact despite the massive strides we have taken in the last few years, we are still near the bottom. 


Read next:

Maximizing the Organization: Turning Data Into a Competitive Advantage