Data analytics has been around since the invention of abacus, improving dramatically as centuries passed by. It has had to. The population of the planet and the complexity of the problem has increased exponentially. Solving a problem in batches and deciding on the corrective action is often not enough.
Enter real time analytics. The main goal for us has been to solve problems quickly as they happen, or even better, before they happen. Today we all want to use real time data analytics to foresee and solve problems for our business. But how exactly is the current industry faring?
Real time data analytics have come a long way over the last decade. Especially with the advent of big data and the general industry inclination towards data driven decision-making. While there has been ongoing debate about instinct vs insight in business, the importance of data has risen exponentially with the introduction of big data.
The best part is that technologies and tools are easily available today. Tools have real time data analytics capability for any business is right at our fingertips. Starting from open source solutions, which are absolutely free (combination of Kafka-Druid etc.) to enterprise grade solutions (New Relic, Splunk), business with patience and/or the funding to afford it are not short of choice.
Real time analytics is yet to be efficiently utilized to its full potential. We often see businesses using real time analytics tools to barely monitor a data source such as data center or websites. While the idea of using real time analytics fascinates business decision makers, often the technical team of an organization does not align with the business teams to fulfil the promise of real time data.
Often, the technology stack used (open source components to build real time analytics tools are often not very convenient for rapid development) or the tools that are used (charges for enterprise grade tools are out of reach for small and medium business) provide the challenge.
The Hard Place
Stuck between a rock and hard place. That's how small and medium businesses often end up working with tools with limited features. Eventually they lose to bigger organizations in terms of agility. Building a custom open source based analytical tool requires money, time, effort and the know-how. On the other hand often tools like Splunk (lowest price $900/GB) or New Relic (separate pricing for data and hosting) puts these tools out of the reach of these organizations.
The good news is that several companies in recent times have been trying to bring affordable solution to this space. The likes of IQLECT (approx. $0.93/GB), DataDog ($15/host/month) and Scout ($10/host/month) are making a decent effort to provide an alternative to small and medium businesses.