In theory a company could start a data programme by simply downloading open source software and running data through algorithms. However, this is unlikely to be the case as in order to make this work companies need to have the necessary skills available to them internally.
Therefore, companies are finding that they are needing to invest heavily in data in order to create decent insights. In fact a recent IDG Enterprise study found that companies are spending an average of $7.4m on data related initiatives in 2015. With enterprises investing $13.8m and SMEs $1.6m, it shows the significant numbers behind the Big Data industry.
Much of this money will be spent on what are essentially free-to-use platforms, such as Hadoop, with companies like Cloudera and Hortonworks building either more UX friendly or complex systems on an open-source foundation.
These have seen significant results for these companies, with Hortonworks going public and increasing steadily in value and Cloudera being number 36 in the 500 fastest growing companies in the world in 2014.
Behind these numbers and increasing spend is the basic idea that Big Data is spreading in scale and expectation across many businesses. Numbers back up this idea, with the IDG survey showing that 70% of organizations asked had either deployed or are planning on deploying in the next 12 months.
With these numbers is not surprising that spend is increasing and are likely to increase further in the next few years.
Simple systems that may have been adequate for the start of data initiatives need to be upgraded in order to process more data in less time. As the use of data increases this is going to increasingly be the case, but as the numbers of adopters increases, the chances are that there is going to be a plateau in spend as less companies take the initial jump into data initiatives.
Aside from the actual technology surrounding data initiatives, one of the key outlays is on the renumeration of data scientists and those with data science skills. Diginomica claim that the average wage for a data scientist is currently $118,000, meaning that if there was a team of 10 data scientists the outlay alone would be close to $1.2m.
As the supply of qualified data scientists increases in future, the chances are that this number will decrease and therefore the overall spend on employee compensation will fall. However, the KD Nuggets payment studies of the last two years have shown consistent growth in salaries, meaning that we are not close to this point yet.
At present, the money spent on data initiatives is a strong indicator of the health of the business function, but as we move into a world where adoption becomes more widespread, the size of the increases in spend may decrease as improvements will be tweaks rather than full implementations.
However, at the moment we are seeing that spend on data initiatives is increasing at an impressive rate and this is likely to continue for at least the next 2 years, which is definitely a boon for the big data companies at the moment.