The Big Data Skills Gap

Is it a thing? If it is, what can companies do to jump it?


One of the most discussed issues in the data community is the skills gap or lack thereof.

Some claim that it doesn’t exist and is more about scaring companies into paying data analysts more. Other claim that it is a very real issue and that many companies are struggling to find the required skills.

So which is it?

I think that the latter argument has far more compelling evidence, as surveys have shown that companies are struggling to find the talent required to overcome their data science shortfalls. This is unsurprising given that according to HBR, 85% of companies are planning on implementing a data science programme in the next 5 years. This demand will naturally outstrip supply, and 72% of those asked in a different survey claimed that sourcing analytical skills was challenging or very difficult. In this same survey 4% claimed that it was actually impossible.

The truth is that those who claim that the skills gap is a myth, may be the ones working at the companies with the best pedigree and who offer the most money. Sourcing the best candidates for those kinds of companies is probably not going to be as challenging as for smaller or less established companies.

So what can be done to solve this skills gap for the companies who are struggling to find the necessary talent?

The easiest answer is simply to pay more than others, but this approach not only affects your ROI, but will also mean that the employees you get will likely only be working for you for money, rather than for any particular affinity for your company.

So the best way is to either sponsor a student early, with a guaranteed tenure after graduation, or train pre-existing staff with the skills needed to perform the necessary role. These both have potential negative impacts though. A student who shows early promise may not fulfil this at the end of a course and existing employees may not have the necessary initial skills required to take on what could be a complex role.

There is no simple answer to how this can be solved. It is something that is only going to become more complex with time as companies realise that they need to adopt this data driven approach is vital to keep pace with competitors. We are seeing that from the results of companies who have successfully adopted data driven approaches that they are performing better than their competitors.

Universities are ramping up their efforts to increase the numbers of graduates from related subjects such as statistics or programming. It will certainly not be a short term solution, but should help to create a good number of graduates in the coming years. This will take time, but is there a short term way to fix this?

At the moment, it is possible to utilise outside companies to analyse some data. Companies who hold particularly sensitive data may not want to go down this route (after all Robert Snowden was a contractor at the NSA), but for less sensitive data this could be the answer. It is a good way to enter the world of data analysis without the risk of having a full time member of staff and investing in the often expensive technology yourself. 


Read next:

Groupon: What Advice Would You Give To Someone Starting A Career In Big Data?