Will Big Data Close Its Own Skills Gap?

Will Big Data As A Service close the skills gap or just paper over the problem?


Big data has had a huge impact on almost every area of business. It has changed the way people market themselves, how they sell to customers, and even how they set strategies moving forward. Companies now regularly collect petabytes of data to aid their decision making and ability to react to changing landscapes, but the reality is that there is still some way to go.

One of the biggest struggles that companies have had in this regard is that there is a dearth of truly top quality data talent currently available. It is well known that the data skills gap is having a big impact on the effectiveness of data programs across the world, with Sanjay Brahmawar, global head and managing partner strategic business development for IBM’s Watson Internet of Things business saying in March 2016 that ‘One of the biggest issues is going to be the gap in skills. Getting the skills required to analyse and manage all of this data is going to be difficult.’ The potential numbers involved in this is also shocking, with Sanjay also noting that ‘By 2020 we will have one million unfilled jobs in the IT sector. Primarily because the skills we have today aren’t the right skills for the future. The future is more about the business understanding and the data understanding.’

One of the ways that many companies have tried to get around this issue is through Big Data As a Service (BDAAS) platforms, which are predicted to be worth around $30 billion by 2021. Companies like IBM offer elements of this, which in essence allow regular people to analyze data in a similar way to data scientists. They take many forms, from services that basically provide pre-analyzed data from specific areas, through to simple to use dashboards where you can put raw data in one end and have analyzed data to come out at the other.

However, are these platforms simply a Band-Aid to cover the wound or are they genuinely viable long-term solutions to the data skills gap?

Answering this question is actually surprisingly difficult, because although they by no means solve the skills gap, they certainly allow companies to use data who would otherwise be unable to. However, they can only offer a fraction of what a fully qualified and passionate data scientist can, due to the fact that they’re essentially there to process and present data, something that is only a small part of what a data scientist actually does.

These platforms cannot look through business problems and identify the potential data sets that could help to shine a light on elements that could be improved. For instance, if somebody with little-to-no training was tasked with finding out why sales of a particular product were higher during 2 weeks of every year, would they have the ability to fully investigate it? They would be able to pick out the headline data, but the ability to really get to the bottom of an issue is something that is only possible through identifying and analyzing what data is needed, something that BDAAS platforms will never be able to do.

However, that isn’t to say that they won’t have a profound impact on data science. The ability that they give to untrained employees is huge, giving them the opportunity to analyze data, even if it doesn’t get to the same level as they would if they were using a data scientist. It means that the data scientists do not need to waste time doing relatively simple things, giving them the opportunity to work on problems that could have a more profound impact on the company.

Ultimately BDAAS systems are never going to fill the skills gap because they lack the human element of data scientists that makes them great at their jobs. However, they will allow companies without access to data scientists to undertake at least basic forms of data analysis, and allow those who do have access to utilize their data scientists to the most pressing and important problems. 


Read next:

Why Blockchain Hype Must End