Whats The Big Deal With Big Data? Is It Worth It?

It doesn't have to be so complicated


Data, data, data. Big data, analytics data, business intelligence, charts, graphs, lines, powerpoints and the infamous spreadsheet.  If you are in the business or marketing world, or if you're just making business decisions in general, chances are you are dealing with data.

If you type in big data, or business intelligence or any other data based buzzword into Google there is an endless amount of content surrounding it. You could argue I am adding one more piece of content to the heap. Which I am. But does data really have so much value that entire business are popping up as middle men offering ways to integrate different data analytics platforms? Some focus on getting OLAP to talk nice to Hadoop, others build their own proprietary systems like Domo. You might be asking "what is OLAP or Hadoop?" and that's my point exactly. There are so many different niches in the data world, and so many people offering little niches within those niches it makes you wonder how there is money to support all of this?

Why does it have to be so complicated? How do I know if any of these data mining, data visualization products along with all the middle men up and down the food chain are worth my time?

I have been working at different agencies and companies since my early twenties, and I have had the privilege (or curse depending on how you look at it) with having to work with about a million different data backends and different ways to export the traditional data cube.

The problem

The problem is we have so many data scraping tools to deal with. We have Google analytics, Adwords, Site catalyst, a ton of SEO analytic packages, different user behavior tools to track interactions on the site. Then we get into the Big Data world of server warehouses, massive spread sheets,  thousands of nodes, tracking cubes with endless amounts of data.

Fancy words like Hadoop, avro, chukwa, hive, BI, Pig, Spark, Tez and more.

How much time do we make decisions on this data, vs how much time do we spend organizing and tracking the data? How much time and money does it take to implement a new tool, or integrate another tool to play nice with an existing one?

The Solution

You are going to hate my answer. Because it is simple. Literally. The solution is to simplify. This could be extrapolated outside of the data world but one thing I have learned over the years is the more people it takes to run your "data" the less valuable it is. Whether it be simple web traffic and SEO data all the way up to a massive building full of servers handling astronomical amounts of data, the more people, the less valuable that data is. 

Why? Because getting that data costs more and more, and at what point does that data cost more than it provides? That is a really hard threshold to pinpoint down in a small business, let alone a bigger one.

So as a general rule of thumb, if a task needs to go through 3 or more people then you need to re-work something.

Whatever you use, take note on how many people have to communicate for things to work. Does your server guy have to talk to the guy at some 3rd party company offering integration who also has to talk to the girl back in your business about the contract but she doesn't talk to your server guy who is also talking to your analysts who don't know about the new integration so they cant implement the right changes for the integration which the original server guy doesn't notice got sent out wrong because he's busy tracking down this tangled game of "telephone"

Confused? So are your employees.

Or maybe your marketing guy needs something as simple as adding a new code to track a new page in site catalysts. By the time he goes through all the channels to make this happen he has spent more time trying to get data rather than analyzing it.

 So back to my super simple, anti-climatic solution.

Keep it simple.

There are a million tools out there that can help with this. For the marketing side of data (the less technical side) I use an nifty marketing dashboard that I can dynamically pull data from a myriad of sources in one place independent of relying on anyone else. Tools like Hubspot, Eventbrite, GA, Salesforce, stripe and even SQL tables. I don't have to go through anyone else or rely on anyone else to get the data I need.

Any type of tool that consolidates things, takes an employee out of the picture (I know that sounds harsh but it is true) or cuts out other tools is something you should look into.

I know that I didn't give to many technical answers because I don't know the answers that lie in technicality. There are plenty of great powerful tools out there for any type of data, just Google it. Whatever you do, no matter how fancy things may sound, keep it simple. Don't fall into the trap spending more on your data then you get from it.


Read next:

Working At The Boundaries Of Aesthetics And Inference