Big data and the Communications Data Bill

The ignored storage challenge


The Communications Data Bill - known more colloquially as the ‘Snooper’s Charter’, is tabled to come into effect in the Autumn in the UK. This new act will require internet service providers (ISPs) to store data of users’ online activities for 12 months in anticipation of it being requested by government agencies. First proposed in 2013 during the coalition government, where it was voted against, the Conservatives vowed to put it in place following the 2015 General Election win. The ethics of the bill continue to be debated by ministers and liberalt groups alike, and, most recently, it was the subject of an independent report carried out by David Andersen QC. Up until now, however, the practical challenge involved, and that faced the IT industry, has yet to be addressed properly – how will the bill add to the already unsolved challenge of the big data explosion?

Innovations, such as big data and the internet of things (IoT), have resulted in businesses collecting more information than ever. Indeed, Google states that it collects 100 petabytes (PB) of data a year from YouTube alone. This mass of seemingly innocuous data is being analyzed, giving firms a far more comprehensive view of our buying and viewing behaviour which, in turn, is being used to create real actionable business intelligence. However, while the benefits of big data are well documented, its widespread adoption means we already have too much data to cope with. Businesses are struggling to see the wood for the trees and a bill forcing ISPs to collect more is only going to exacerbate the issue.

Of course, the industry has gone into overdrive in a bid to solve the growing data challenge and unlock the treasure trove of information that big data provides. With huge investment into mass data analytic tools, including both hardware and software, the industry has devoted its time to finding new solutions which can crunch massive quantities of data within tolerable times. Indeed, innovation has spanned the spectrum of advanced data analytics and search-based applications to parallel-processing databases and data-mining grids. As such, many companies believe that they have the foundations needed to drive the most from their big data venture, but there’s one vital cog in the machine that has been continuously overlooked – storage.

With the huge increase in data, understanding where and how that data is actually being stored is essential to the efficiency of the entire process. However, many businesses continue to rely on legacy storage devices which were never designed to cope with the fluctuating pressure of big data. Their rigid nature means that business environments are not scalable and cannot easily adapt to meet changing demands, instead, often choking when faced with an avalanche of information. Therefore, it needs to be asked, if businesses are struggling to cope with the demands of big data, how are ISPs going to cope with a bill that forces them to collect and store even more information?

Thankfully, there is another way – Software Defined Storage (SDS). Developed to give storage the same operational efficiency as other aspect of the big data process, such a model ensures firms can extract actionable business intelligence from their growing data stores. By providing a flexible and scalable architecture, businesses are empowered with environments that are adaptable to big data’s fluctuating demands – ensuring consistent high performance.

SDS also presents a sustainable and cost effective IT investment due to its open-sourced design (OpenSDS). For example, when there is a need to expand capacity, companies are able to cluster SANs instead of having to buy an entirely new solution. This means OpenSDS can be used to support start-ups as well as multi-national corporations, with the only difference being the number of clustered SANs. Furthermore, as SDS solutions are hardware-agnostic, they can fully integrate with existing data centre hardware, empowering businesses to build bespoke setups from the ground up.

Ultimately, the need for all companies to upgrade their storage is apparent, but for ISPs still relying on legacy setups, the new bill represents a looming problem. Being forced to collect and store users’ online activities for government agencies will place already creaking infrastructures under even more pressure. For the rest of the business community, however, whilst the issue isn’t as pressing, it’s not going to ease any time soon. With the amount of data being collected continuing to increase exponentially, day by day, businesses should see the ISPs’ plight as the perfect opportunity to upgrade. Indeed, by choosing to leave their rigid legacy setups behind, businesses can turn to SDS solutions and achieve the most from this data driven world. 


Read next:

Working At The Boundaries Of Aesthetics And Inference