Ever since its inception, at least as a quantifiable and named ‘thing’, big data has been surrounded by a huge amount of hype. We have seen hundreds of articles discussing how it will make everybody’s life better/worse and how companies are using it for good/evil. Once these initiatives have come to pass though, the actual impacts haven’t been the world changing events that many believed.
Instead, many have become developments not immediately recognizable as driven by data - like the spread of AI, the IoT, and In-Memory databases — and even these have made incremental progress. None have really made front page news, despite them having a huge impact behind the scenes. According to the Gartner Hype Cycle, big data was entering the Trough of Disillusionment in 2014, then had completely disappeared from the cycle by 2015. So does this suggest that big data hype is over?
Not even close.
The hype around big data has instead changed from the potential from data itself through to the technology that big data supports. Self driving cars? Data-driven. AI? Data-driven. Virtual Personal Assistants? You get the idea. Data is the single biggest driver of innovation today and this means that the hype has moved away from the potential of predictive analytics through to the ability to turn on your office lights from the other side of the world by talking to a speaker.
It is not only in new products and technologies where the hype exists either, it is in the implementation of data within specific areas. For instance, when asked on Quora about this hype, DJ Patil responded with ‘I’m seeing an incredible growth of people using data across the country. It turns out there are more than 25 leaders at the top across major government agencies who are focused on data.’ It is difficult to therefore argue that the hype surrounding data has ended, however, this hype is both a good and bad thing.
One of the really positive elements has been that the hype has sped up the spread of data and accelerated its development in many companies. In 2015, for instance, the average spend on data-related initiatives was estimated to be $7.4m, with 80% of enterprises having already deployed or seeking to deploy big data projects across their organization. This has led to a jump in the number of data scientists and even led to Glassdoor naming data scientist roles as the best jobs in the world, given the number of open vacancies combined with a median base salary of $116,480. This has led to an increase in the number of people aiming to become data scientists, which has therefore increased the talent in the area and sped up the development of data led technologies.
As more companies have seen the benefits of even comparatively basic big data, like Netflix’s use of suggestion engines to help customers choose content they will like from the 8000+ TV shows and films or Tesco’s Clubcard program tracking purchase history, more and more are looking to adopt it. As more money is invested in data, including external vendors and cloud based technologies, these can then accelerate development and benefit a wider audience, further accelerating development.
However, the hype is not all good and can have negative consequences, as many companies have found. Hortonworks have been a victim of this hype, with the data software provider’s shares peaking at $27.65 in June 2015 and trading at $9.10 today. Much of this initial value was down to hype, having been the first ‘big data company’ to go public and the market reacting excitedly to the potential. Consistent quarterly operating losses and dilutive secondary stock offering, however, has seen their value plummet.
We have also seen public trust in data quickly dissipate as what were presumed to be accurate analysis of things like polling data and financial results have been wrong, with both sides using the data to beat the other. Take the Brexit campaign where it was suggested that the UK would have significant financial struggles if they left Europe. The same data is being used to show that they won’t - because there has still been growth - and they will - because the growth slowed dramatically despite still being in Europe. The importance of data should be sacrosanct, but when its manipulated so heavily it is difficult for people to believe the hype as both sides will ultimately see it as being wrong.
So the hype certainly hasn’t gone from big data, which is both a good and bad thing. We are going to see the adoption of more data-focussed technologies and approaches in the coming years, whether this quells or increases the hype around it remains to be seen.