Experts Claim Machine Learning Talent Shortage Not A Problem

I spoke to a number machine learning experts about why they are feeling optimistic despite the apparent skill shortage


The risk that machine learning will render swathes of jobs redundant increases by the day. Ironically, however, as this trend intensifies, so too does the search for data scientists and machine learning experts. Simply speaking, supply is not keeping up with demand. According to Tech Pro, as few as 28% of companies have any experience in the field. In a recent survey by O’Reilly, 20% of respondents reported a lack of skilled people as a bottleneck. Go to any data event and you will find "we're hiring" banners at the end of presentations commonplace - and those are companies which have machine learning experts!

Despite this apparently obvious deficit in the field, machine learning experts seem to err on the side of optimism when discussing any shortage. In anticipation of the Machine Learning Summit, taking place in San Francisco on May 9 and 10, I have been speaking to some of these professionals. I wanted to understand why they aren't as worried as we think they should be and what they think it all means for the future of machine learning.

Sam Zimmerman - Freebird Co-Founder and CTO

I do not think the evolution of machine learning will be hampered by the lack of research talent in the field. The machine learning field moves incredibly quickly thanks to open source norms, large amounts of attention by a variety of thinkers, and capital from VC’s and large companies. That being said, I do think machine learning is currently limited by two things;

  1. A lack of people who are incentivized to validate an incredible amount of fantastic research that is being published
  2. A lack of talent to apply the aforementioned research to messy real-world problems

In order to solve the validation issues, we need to find ways to incentivize talented individuals to replicate and generalize research that has already been completed. This can be done by rewarding the quality and extensibility of research, over speed and novelty. To solve the second problem, we need to develop more specialized skill sets and job descriptions as an industry. For example, we need engineers who can handle the idiosyncratic issues involved with deploying and maintaining machine learning models, as well as something like 'Data Science Product Managers' in charge of owning the product the model is powering. As a field, we’ve focused on too much on research. We need to develop other important roles in the data science ecosystem like 'data science product' and 'data science strategy' in order for data science to have maximum impact.

Hugh Williams - Machine Learning Data Science Manager at Uber

There’s simply a supply/demand imbalance right now and it’ll get better over time. What you’re seeing is a market correction, where tons of talent from other fields are crossing over to ML both because it’s valuable to companies but also because there’s a lot of ways to get into it.

Let me give you an example of why I love the current environment. Here is a sample of the fields that my team got their degrees in:

  • Transportation Engineering
  • Industrial Engineering
  • Thermal Science
  • Quantum Physics
  • Computer Science
  • Statistics
  • Theoretical Chemistry

Before machine learning, there’s no way all of us would have ended up in the same room - let alone be collaborating side-by-side to solve some of the coolest, most challenging problems at Uber. Yes, it’s tough finding great talent. But that difficulty pushes folks like myself to find brilliance in a wider spectrum of places.

Tom Andriola - Chief Information Officer with the University of California

There is absolutely a mismatch between demand and supply of people in machine learning fields. In industry, however, you're seeing the more progressive companies build their own internal engines to train people up on these models. The goal is to get them to convert from traditional computer programming and analytics and be able to take advantage of these new techniques like machine learning and AI. However, they are based on some different principles and fundamentals.

Let's say the average programmer has to have the basics in statistics. As with machine learning, there are a lot of different statistical models and techniques that are utilized. Most computer science majors from 15 - 20 years ago didn't get training in recurrent neural networks, which is one of the main techniques used now when we talk about deep learning. Therefore, what we are increasingly seeing is organizations, like one I work closely within the healthcare field called Optim, developing something akin to an internal university. They're essentially trying to convert a lot of their traditional programming and analytics people into data scientists.

Likewise, all of our universities have moved pretty aggressively over the last three years to build up a capability to train up more people in these fields. Seeing the demand in the marketplace, some have developed straight data science programs while others have built more of a center. Here, they pull from expertise across their campuses and expose students from many different backgrounds into building a core competency of understanding data science. They then allow them to take that back into their individual fields - whether this be in computer science, engineering, sociology, statistics etc.

So we're seeing some different models across our campuses, but they are all moving very aggressively. They understand that the broader economy and almost every industry out there is looking for these people - people who understand the science of data and the models that are now being used in the machine learning space.


Read next:

Why We Need Data Visualization To Understand Unstructured Data