Big Data Top Trends 2017 - How Are We Doing?

Half way through the year, we check back to see how we're doing


In November 2016 we made our predictions about what the market would look like in the coming 12 months. We put our collective heads together, spoke to our network of data wonks, and put our ideas down on paper. As we are now halfway through 2017 we wanted to revisit our predictions and see how clear our glance into the crystal ball was.

Prediction: Machine Learning, AI, And IoT To Become Common

Reality: The pace of change within machine learning, AI, and the IoT has been high in 2017, with huge leaps forward in their use across the board, both from a consumer and business perspective.

IBM’s Watson has become practically a household name, the ‘threat’ of AI has seen the term become common on front pages across the world, and it is being touted as a way of protecting the world against the increasing number of cyber attacks.

Equally we have seen the IoT make huge strides, with an increasing number of people bringing connected devices into their homes and companies making significant gains in IoT. Some of the success stories so far include EPAM Systems, who have seen their shares increase by 36% in the first six months of 2017 and Skyworks who have seen theirs increase by 47% in the same time. The success of these stocks signifies that the market is buoyant and some of the key companies in the sector are performing well, which is a clear indication that the IoT is becoming increasingly common, both in people’s homes and perception.

Accuracy: Correct

Prediction: Increased AI Accuracy

Reality: AI has taken centre stage in 2017, moving from a relatively indistinct concept to reality, and alongside this, there has been an increased accuracy. One of the key manifestations of this has been within the medical field, with treatments like Mammograms, heart attacks, and strokes predicted considerably more accurately thanks to AI usage. In tests conducted against doctor-led diagnosis, AI scored between 0.745 and 0.764 (with 1 being 100% accuracy) compared to 0.728 from doctors. It can also predict autism in young children, before symptoms manifest, something that was impossible before.

It has also had more negative consequences, with AI at heart of the darker elements of recent elections. Its accuracy use has been credited with suppressing the votes of certain parts of the population. This stems from companies like Cambridge Analytica who have been accused of underhand methods using precise AI combined with psychological warfare techniques ( and stemmed headlines like ‘Donald Trump, Our A.I. President’ (

Accuracy: Correct

Prediction: Companies Will Need To Prepare To Operate At Speed

Reality: This is a prediction that is difficult to gauge given that we predicted that rather than actively adopting in-memory and quantum computing techniques, companies would instead be preparing for their increased presence. It is therefore hard to say whether or not companies are actively preparing, but there have certainly been moves from companies who provide many of the data platforms used that suggests they are increasingly moving towards this kind of service. Large players like Hortonworks, IBM, and SAP all now offer some kind of in-memory service that is likely to be adopted by companies in the future.

Accuracy: This is probably wrong, if platforms are preparing to offer it as part of their package, there will be little need for companies to do the same.

Prediction: Less Industry Specialization

Reality: This is again a prediction that is hard to get an accurate read on half way through the prediction period because there aren’t really any objective and quantifiable ways to find solid numbers on the subject. What is certain is that we have seen anecdotal evidence of this, with several of our speakers from the past 6-12 months working across many different industries throughout their career to date.

It means that they have experience related to the use of data rather than the use of data within a specific industry, which has typically been the case in the past, especially when they were working as individuals rather than in teams. As more data scientists have begun to take a collaborative approach, there has been less of an onus on understanding of the business and more on actually using the data effectively.

Accuracy: TBC

Prediction: Government Scrutiny On Data

Reality: There is little doubt that data has taken center stage in terms of government attention, especially in the US. We have seen it become the biggest talking point in politics over the last year, from Hillary Clinton not having good enough data security, Russian hackers stealing data from the DNC, and recently a leak from Deep Root Analytics which released details from 200 million users.

However, despite it becoming a huge issue for governments across the world, there is little additional legislation that has been passed this year that is likely to have an impact on companies. Considerable work being done in the area though, primarily as a result of the GDPR (General Data Protection Regulation), an EU legislation that requires companies and governments who hold data on EU nationals to protect that data in robust ways. The approaching 2018 deadline for this has meant that despite a lack of original legislation in the area, it is clear that a considerable amount is being done in terms of preparation for it.

Accuracy: Correct

Looking small

Read next:

Expert Insight: 'An Effective Visualization Results From A Great Deal Of Curiosity And Exploration'