There Is No Data Talent Gap

It's a much discussed problem, but is it real?


There is plenty of supposed evidence that a substantial gap exists between the number of data scientists needed and those available. A study by McKinsey, for one, projects that ‘by 2018, the US alone may face a 50% to 60% gap between supply and requisite demand of deep analytic talent.’ Indeed, the 2015 MIT Sloan Management Review found that 40% of companies are already struggling to find candidates to fill their data analytics roles, and if the number of speakers at tech conferences complaining about a shortage is anything to go by, the situation is not improving. 

This alleged shortage of candidates is baffling. Just a few years ago, Harvard Business Review called Data Scientist the sexiest job of the 21st century, while the role also topped Glassdoor’s list of ’25 Best Jobs in America 2016’. The job is rare in that it is both intellectually fulfilling and also highly lucrative, with salaries averaging around the $120k mark. Even a summer internship will often pay somewhere in the region of $6,000 to $10,000 a month. It seems to tick all the boxes, so why is there a lack of suitable applicants?

Many reasons have been given for the alleged shortfall. There is a common misconception that maths and science subjects at school are only for the ‘ultra bright’ - those getting straight As throughout their school life. Some argue that this can be off-putting to those considering a data-related degree. Evidence for this is thin on the ground. While ten years ago, just a handful of colleges in the US offered Big Data/analytics degree programs, now almost 100 schools have data-related undergraduate and graduate degrees, as well as certificates for working professional or graduate students wanting to augment other degrees. Universities do not just set up courses for no reason, they do it because the demand is there, and these courses are largely full, suggesting a healthy pipeline of job candidates. Of course, these students take time to filter through into the jobs market, but the growth in the number of courses started long ago, and there is no reason we should not be seeing the results now.

There is also an argument to be made that a skills gap exists because data science suffers from the same issue all STEM industries suffer from: A lack of women. However, while only around 18% of computer science degrees go to women, they are achieving more than 40% of statistics degrees. Carla Gentry, a successful data scientist and founder of Analytical Solution, explains that, ‘More women are becoming interested in the big data field because it's an interesting subject, filled with lots of potential. I think 'we' see the whole picture of these possibilities because as wives, mothers, etc. we have to see the macro view all the time. Therefore seeing the big picture comes naturally, in my opinion.’ Gentry does, however, continue to say that, ‘But, we do have an uphill battle to gain a foothold in this field, as I am constantly reminded even after 17 years in data analytics. Until our own field (tech/data science/analytics) recognizes us for our talent, how do we think others will? There are too few truly talented, experienced people in Big Data to silence the share women have attained. It's time to start highlighting talent and not gender. We need all hands on deck if we plan to take Big Data analytics to the next level.’

Gentry may well be right, and it could be that managers are still reluctant to hire female data scientists because of some prejudice. If this really is the problem, it’s one that is easily solved simply by hiring more women. This is a failure of management, and it could be that similar failures are leading to the perception that a gap exists that’s not really there. Managers are often not ‘data people’, and many simply do not know what they’re looking for in a Data Scientist. The term itself encompasses a variety of things, but ask a dozen hiring managers what these are and you’re unlikely to get the same answer twice. A statement made by Tom Pohlmann, Head of Values and Strategy at Mu Sigma, is especially telling. He argued that ’it can be difficult to find candidates with the creativity and experimental mind-set to truly revolutionize the handling of big data to truly transform a business.’ In how many other roles is it expected that someone will come in and ‘revolutionize’ the business? Most Data Scientists are just smart people looking to do their job well and add value. If every small business is holding out for the next Sergey Brin to come in and drive their profit through the roof, they’ll likely be waiting a long time.

The talent is clearly there. Maybe it is the case that companies are simply not looking hard enough, but it could also be that darker forces are at play. If we assume that companies are not so overcome with prejudice that they won’t look at women for data science roles, and that they are do not believe a data scientist has to be a genius, only one question remains: Is the whole thing a fiction?

Many studies suggest that the skills shortage in STEM in general is a myth. According to Hal Salzman, an expert on technology education at Rutgers, ‘the supply of graduates is substantially larger than the demand for them in industry.’ In fact, he found that only half of STEM college graduates each year get hired into STEM jobs. High-tech companies like Qualcomm are actually downsizing. At the same time, Qualcomm - along with firms like Google, Microsoft and Facebook - is simultaneously lobbying hard to help get the Senate: S. 153 - the Immigration and Innovation (I-Squared) Act - passed, which gives more latitude to companies for employing workers on H-1B visas. H-1B visas are granted to foreign workers who have a bachelor’s or higher degree in a wide range of areas. These are designed to serve high-skilled immigrants, but often enable the importing of Indian and Chinese guest workers to replace an older, more experienced, but more expensive domestic workforce with cheaper labor. S. 153 would increase the number of H-1B visas from 65,000 up to 245,000. Also included in the bill is an extra incentive for STEM workers like data scientists, with a provision to give international students a lifetime work visa for obtaining any advanced STEM degree.

Michael Teitelbaum, a demographer at Harvard Law School and author of the 2014 book ‘Falling Behind? Boom, Bust, and the Global Race for Scientific Talent’, argues that a skills shortage in STEM is a falsehood being propagated by companies wanting to flood the market with cheap labor. As previously mentioned, the average wage for a data scientist is $120k, and bringing this down has clear benefits. Teitelbaum notes that: ‘If you can make the case that our security and prosperity is under threat, it's an easy sell in Congress and the media.’ Rochester Institute of Technology public policy associate professor, Ron Hira, agrees, arguing that ‘many in the tech industry are using it (H1-B) for cheaper, indentured labor.’

Whether or not there is a real intent to exaggerate the scale of the skills gap, the simple truth is that the talent is there. If companies really are struggling to employ data scientists, it’s entirely their own fault.


Read next:

Why Blockchain Hype Must End