Companies have really embraced Big Data over the last few years. They’ve had to, or they’ve fallen by the wayside. However, while it’s clearly still useful, it is no longer enough simply for Big Data to be ‘Big’. Data quality and real-time insight are equally - if not more - important.
Big Data is often created by tremendous amounts of data produced at tremendous speed. Financial ticker data is one example, as is sensor data. The Internet of Things is also set to increase volume, variety and velocity of data in the future, as it becomes a more and more prominent feature across society. Events in these data streams often occur thousands to tens of thousands of times per second, requiring what has become known as ‘Fast Data’.
To be of real benefit to organizations, Big Data has to be processed and actionable insights garnered in real time. This has been enabled by huge leaps in ‘stream processing’. Up until just a few years ago, building a stream processing system was too complex for most businesses to tackle, but thanks to innovations by firms such as Typeface, it is steadily becoming a ubiquitous tool for companies that employ Big Data.
Stream processing solutions are designed to handle high volumes of data in real time thanks to a scalable, highly available and fault tolerant architecture. Live data is captured and processed as it flows into applications, in contrast to traditional database models in which it has to be stored and indexed before being processed by queries. The solutions can then power real-time dashboards and on-the-fly analytics. Stream processing also connects to external data sources, which enables applications to incorporate selected data into the application flow, or to update an external database using processed information.
These solutions analyze and act on real-time streaming data using what is know as ’continuous queries’. These are SQL-type queries that operate over time and buffer windows, allowing users to receive new results as and when they become available.
A recent development in the stream processing industry is the invention of the ‘live data mart’. The live data mart gives end-users ad-hoc continuous query access to streaming data that’s aggregated in memory. For business user-oriented analytics tools, this means that they can then go into the data mart and see a continuously live view of streaming data.
The applications for Fast Data are many, basically aiding any business that requires data in real time. Fraud detection and cyber security is one area that it can be especially useful. Banks need to monitor machine-driven algorithms, and look for suspicious patterns. Whereas before, the data that was needed for finding these patterns was loaded into a DWH and reports checked daily, the stream processing implementation now intercepts the data before it hits the DWH by connecting directly to the source of trading. This also helps firms abide by new regulations in the capital markets, which require firms to understand trading patterns in real time.