Big Data Top Trends 2018

We take a look at what will happen to big data in the next 12 months


2017 has been a year where data has become mainstream, where it moved from being something that those who worked in business saw as a buzzword to something that everybody was talking about. Whether that was through the narrative surrounding some of the big data hacks or the scary stories around AI stealing people's jobs. Suddenly there is a genuine data narrative in the public consciousness and those who are not working with data in the way they should be are being rightly chastised as a result. However, the next 12 months is likely to see this data enlightenment increase even further and the speed of change within the market is going to accelerate.

It is this highly unpredictable and dynamic landscape that we have tried to piece together our annual predictions for the next 12 months:

Increased Move To The Cloud

It will come as a surprise to nobody that companies have seen the light and moved to the crowd in growing numbers. The initial worries about the security of data held in the cloud has been replaced with the realization that the cloud is considerably safer than anything most companies can offer when holding anything onsite.

There has also been an increasing number of employees working remotely, meaning that there has been a need to give secure access to data and analytical tools from wherever they are in the world, making Big Data as a Service (BDaaS) an increasingly important tool. When you add to this the scalability, speed, convenience, and cost it should come as no surprise that Forrester predicts that 50% will embrace a cloud-first policy in the next year.

Real-Time To Continue Growth

According to the "Streaming Analytics Market by Verticals - Worldwide Market Forecast & Analysis (2015 - 2020)" report, real-time analytics were predicted to see an average annual growth of 31.3% between 2015 and 2020, which seems like it is still reigning true in the market today.

What was once seen as a practice for only the biggest and richest companies is now becoming increasingly common amongst SMEs thanks to the prevalence of the technology and decreasing costs. More companies will be looking to in-memory and in-chip approaches to capture and analyze data as quickly as possible and once a competitor starts to use this kind of technology and can make decisions that much faster, that market will be forced to also adopt it.

At present we are currently just moving from early adopters to early majority in the Rogers Adoption Curve, but 2018 is the time when we begin to move into the early majority stage. This move is being driven by the increasing necessity for companies of all sizes to collect and analyze data to remain competitive in their markets, so it is not longer the multinational conglomerates alone who need to analyze in real-time, it's everybody.

Big Data To Be Buried

The term 'big data' will quietly go the way of IT, something that still has a broad meaning, but by itself is almost meaningless. Like IT, big data is a term that encompasses far too much in the minds of people to really have a specific meaning beyond saying to people who know very little to say 'I work in big data'.

We now have so many different elements that fall under big data, from machine learning and data collection, to analytics and data security, none of which really have too much to do with one another, but are still classified as 'big data'. We are seeing an increasing awareness of all of these areas amongst even those who have no real interest in big data thanks to things like hacks, robotics, self-driving cars, and myriad other data-driven technologies.

It means that although the use of the term big data will not die out, the use of it as a descriptor beyond the holistic look at data will. Big data is dead, long live big data.

Increased Media Scrutiny

The number of huge hacks in the past year has meant that the public is now fully aware of how much data companies hold on them and the huge damage that the loss of that data can do. We have also seen from the huge media coverage of the alleged Russian influence on the US election exactly how much data companies hold on each person, whether that’s Facebook or more secretive polling companies.

We have also seen a huge amount of interest on parts of the data landscape like AI and self driving cars, that have created major headlines around the world. Especially in the case of AI, there is a considerable amount of scaremongering, making it a hot topic to be covered by many news organizations.

This awakening has meant that rather than the data being a techie area, it is instead a mainstream subject covered by the popular press as much as any other mainstream subject. It means that we are more likely to see scrutiny of these areas simply because we now have a more informed public.

Whether this will be a positive or negative for the industry are yet to be seen. On the one hand it will direct people towards areas that have been historically ignored by the general public, but on the other hand it could bring a certain amount of scaremongering, similar to what we’ve seen around AI ‘taking people’s jobs’.

Quantum Computing To Become A More Concrete Thing

At present quantum computing is little more than a concept with a few impressive numbers behind it. Google, the Turing Institute, Microsoft, Intel, and several other companies have already had several impressive tests with the quantum computers, but the reality is that we are still some way off having a real, useable quantum computer.

However, as we move through 2018 we are likely to see quantum computers to become more of a 'thing' as more tests are conducted across the spectrum. According to a paper published by Google in July 2016 the company believed that it would have a 49 qubit quantum computer working by the end of 2017, which would mean it could outclass any supercomputer that currently exists. We have heard that Google have revealed 50 qubits as a proof-of-principle, which represents 10,000,000,000,000,000 numbers at once, something that regular computers would need memory power well beyond anything we can currently produce.

2018 will certainly not be the year that we suddenly start seeing quantum become the norm or even available to the world's largest companies, but what we've seen so far is 'It’s absolutely progress to building a fully scalable machine,' according to Ian Walmsley, Hooke Professor of Experimental Physics & PVC(R) at Oxford University.

Hacks To Get Bigger

Benjamin Franklin famously said 'In this world nothing can be said to be certain, except death and taxes', but in 2018 we can add 'and huge hacks will impact companies'.

8 of the top 10 hacks of all time have taken place in the last 3 years, which isn't coincidence. Data security may be getting better but the improvement in data security is running almost totally parallel with the increased skills of those looking to steal it. With the huge amount of data now collected the number of huge hacks is only going to increase, as we write this the true scale of the Uber hack and its coverup ar e being uncovered, but its clear that something like 57 million accounts were compromised and the company paid $100,000 to keep the hackers quiet.

This is something that is only going to become worse as more data is collected and companies pay hackers to either unlock their data or stop it from being shared. According to research from Symnatec, there has been a 266% spike in the amount paid by companies to the hackers who infected them with ransomware compared to 2016. At the same time there are very few people arrested for hacking. If there is this increase in reward for hacking and only a slim chance of being caught, the reality is that people are going to see it as an opportunity and the activity will increase.

AI To Continue Its March

AI is having massive impacts on the world and given the huge strides forward we've seen in 2017, this will only increase in the next 12 months. More and more companies are adopting AI technologies for any number of actions, from relatively basic roles like warehouse management and within chatbots, through to more complex elements like accounting and data science, but this is likely to go even further in the next 12 months.

One of the biggest elements of this will be the increased capabilities of self-driving cars, which are going to spread in both use and performance in more geographies. For instance the UK Chancellor announced that self-driving cars will be allowed on the roads from 2024, where at the moment they are allowed only with a human behind the wheel. We will also see a big increase in the amount of experimentation with AI for considerably more complex problems, with DeepMind at the forefront of this, even if at the moment this only takes the form of games like GO and chess. However, 2018 may well be the point at which we see these kind of elements becoming the foundation of even further AI development.

We are also seeing AI move into the home too, with huge growth in products like Google Home, Amazon Alexa, and Apple's HomePod bringing more and more devices into people's houses. This isn't simply a fad either, according to research by RBC Capital Markets in March, 'voice-activated internet' devices could be a $10 billion market by 2020, showing huge growth in that time. Amazon currently controls around 70% of the market with their Alexa devices saw a 25% increase in sales in the second quarter of 2017, showing that the market isn still in ascendancy. 

Would you like more information on data? Join the Big Data Innovation Summit in San Francisco
Looking small

Read next:

Expert Insight: 'An Effective Visualization Results From A Great Deal Of Curiosity And Exploration'