The Best Data Scientists Are Failures

In order to succeed in data science, you must be the best at failing


There are some inbuilt human traits that are huge faults.

Think about anxiety, it does considerably more harm than good, turning the strong into a quivering mess, making logical people irrational, and even increasing the chances of having a heart attack. One of the biggest single causes of anxiety is the potential for failure. It could have been an important algorithm that has a flaw, an analysis that may have used inaccurate data, or even that you left the front door open.

However, it is this inherent anxiety and fear of failure that holds people back, stifles innovation and ultimately costs companies money. This is the case in data science departments more than almost any other part of the business.

When we think about one of the most basic forms of data science - the A/B test - it is the epitome of why the drive to succeed and the fear of failure draws people to make poor decisions. In an A/B test there are two changes that could make a difference and the one that works the best is the one that is used or moved forward in the process. What often happens is that the ‘winner’ becomes the way things are done because it ‘won’ the A/B test. However an A/B test does not show you the best option, it gives you the least bad option. Unless you then continue to test the infinite other variables, you will never know what the ‘best’ option actually is.

This requires a huge amount of failure. You have to come up with the strangest ideas to test, completely new ways of thinking and look for seemingly unconnected correlations. This means that in order to succeed it is imperative to fail considerably more than you succeed, because it is impossible to know if you have the best solution. If you think you do, you then need to constantly test against it to try and prove yourself right.

Thomas Edison, one of the most famous scientists of all time famously said ‘I have not failed. I’ve just found 10,000 ways that won’t work’ and this is the approach that all data scientists need to take.

However, this need to experiment and necessity to fail is often directly at odds with what companies need, which creates a dichotomy for both data scientists and their employers. Companies that spend hundreds of thousands on a data science department want to see ROI on their investment in as short a time as possible, which demands success rather than experimentation. People are then often put in a situation where if they admit that something they have done could be better, the company is likely to have a negative view of them and their performance.

Embracing failure as a data scientist is not simply about saying to yourself that you should be failing more than you’re succeeding, there is an issue within society and the education system that chastises failure and promotes instant success. It is something that Tony Little, former head teacher at Eton College in the UK (a school that has produced 19 of the UK’s 53 Prime Ministers and more CEOs than any other secondary school in Britain) has discussed. Little believes that children need to be taught how to fail and ‘Not just have the experience of failure, but of course within a supportive context, to learn from that experience of failure.’

Our current system with grading and testing means that many children are coached to succeed rather than experiment. It creates a situation where those who go to the best universities and go on to get the highest paying jobs are more often the ones who know how to succeed, whilst though who tend to experiment and accept failure are deemed as failures. This essentially perpetuates the problem as those who try things outside of the curriculum are not seen as innovators or free thinkers, the success of their education is instead constrained to grades on a report card.

In order to get the most out of our data scientists, we need to make sure that they aren’t simply getting the correct grades, but they are being taught to constantly experiment and constantly fail. Companies ultimately pay the big money for the guys who can see opportunities for business expansion or process optimization through data - this doesn’t come from getting everything right, it comes from failing enough to know what’s wrong. 

Big data hype small

Read next:

Is Big Data Still Overhyped?