After years of hype, Big Data now appears to be “crossing the chasm”; moving on from only being available to large enterprises with big deployments and becoming accessible to most businesses. The latest development is news of Vault, a startup with an analytics engine that can read film scripts and recommend changes as well as highlighting the best actors, studios and locations for the film.
The concept of the “chasm” was popularised by Geoffrey Moore in his book “Crossing the Chasm”. Drawing parallels between the technology adoption lifecycle and the marketing shift needed in order to move from appealing to early adopters to the pragmatists of the mainstream market (the chasm lies in between the two), crossing the chasm has become the key test of disruptive innovation.
The signs have been visible for a while that Big Data is crossing the chasm. Netflix used Big Data to select House of Cards and its actors; and Share Dimension in the Netherlands is recommending where and when films should be shown to maximize revenues. Meanwhile Apple’s Researchkit correlates anonymized data to let anyone enroll in medical research; and IBM’s Watson is now helping you create recipes.
The big question around analytics technology like this is whether it will be accepted by the industry. Despite widespread (and well-founded) concerns that studio executives will reject computer-generated advice, I wonder whether the film industry is desperate enough to accept it regardless.
There’s certainly plenty of evidence to suggest that a change is needed:
- Studios are increasingly reliant on their biggest titles, which have migrated from being “summer blockbusters” to a year-round carnival
- Industry experts such as Spielberg have predicted that studios will not be able to survive a few of these films failing
- Audiences are becoming more choosy about which films they see, as ticket prices rise and alternatives become more compelling
- The industry has been casting around for “gimmicks” such as 3D and now VR which audiences have not accepted
- Films increasingly need to be globally successful and need to somehow appeal to ever-broader audiences
To me, this looks like an industry ready for disruption, where executives are used to gambling and where the opportunity cost of refusing the new technology (while their rivals adopt it) is likely to outweigh reluctance to embrace algorithms. It looks a lot like the financial services industry in the 1980’s.
The examples above show that Big Data isn’t only becoming available to smaller businesses, but that it’s affecting industries that were considered very difficult to disrupt. If highly creative film and TV industries; strongly regulated and private medical research industries; and the subjective and human-taste world of chefs are being disrupted by Big Data, it’s unlikely that any industry is immune.
With Big Data crossing the chasm, within a few years most industries will have been thoroughly disrupted and creating significant new innovations will be difficult. Today, businesses have the opportunity to get ahead of their competitors by hiring data science teams and embracing Big Data. This is an opportunity to carve out a competitive advantage that could solidify if your competitors aren’t quick enough.
This is also the moment where Big Data entrepreneurs will be able to turn a vision into reality and change the world, by bringing the benefits of Big Data to the masses. Big Data, analytics, and tools are changing the world, but just because they can read scripts and correlate data sets doesn’t mean that human intelligence no longer has a role. In fact, I would argue that these algorithms will free human data scientists from the activities they (objectively) aren’t so good at, giving them better information with which to make the creative decisions the computers can’t.
There is a lot more innovation like this still to come, especially in transportation, logistics and healthcare. Using Big Data to determine when to empty recycling receptacles, to calculate a bill for borrowing a bicycle, and for tracking irregular heartbeats is only the beginning.
If data scientists want to be part of this trend and apply their skills to solve some of the biggest challenges, they will need to think outside the box and tackle big problems that have big rewards. Every type of process including agricultural yields, manufacturing efficiency, and even the process of teaching a class or writing a novel can be optimized by collecting and analysing big data. It’s never too soon to start, those who keep delaying run the risk of being left behind.