The difference between the way we use data today and how it was used 20 years ago, is that we no longer purely use data to deduce meaning or explanation. In the past, the key to data use in a business capacity was to ascertain whether strategy implementation had successful via measures such as return on investment. Today, data concerns translating those insights into a market advantage. Within the past couple of decades, business has churned out more data than they know what to do with. As technology evolves, permitting the accumulation of ever more complex and inexplicable data, the limited mental capacity of mankind poses difficulties in deciphering the unknown environment. This has presented a turning-point in scientific method, as data scientists use data to frame theories, opposed to fitting data to preconceived marketplace theory. Such scientific inquiry has less dependence on existing information, and instead focuses on developing what is unknown.
Through storing incomprehensible amounts of data, and successfully mining it, firms are now able to find correlations that would otherwise may have been overlooked. Product development is and always has been a risky business. But now, companies are using big data to better their odds, by more accurately defining the consumer’s needs and tailoring a product accordingly. Thus, as businesses increase their use of big data, they often have to reevaluate the fundamental approach to product development. Previously, companies would create goods and use mass consumer feedback to adapt the product for the next generation, after the product had reached market. Today, firms are able to preempt consumers needs whilst in development. More impressively, due to the revelation of real-time analytics, some products are now able to be updated live. Thus not only increasing the product lifecycle, but revitalizing the product before decline in popularity or sales, facilitating the continuation of growth.
Beyond the theoretical framework complications, another significant limitation to applying big data to product innovation is internal coordination. The managerial implications for successfully mining data in today’s hyper-competitive marketplace demonstrate that a highly-skilled and effectively integrated data-team is a requisite. Despite the significant potential of big data transforming marketplace activities and product development, a study in 2013 determined more than half big data projects are unable to accomplish their objective.
The projected continuation of growth and understanding within the big data domain, permits data scientists and product developers alike to utilize the information to make better and more informed decisions. Although, with the increase in data comes more noise, and thus it is down to the data teams to be adept in identifying the relevant signals to make the right decisions. Until then, the unknown will simply remain, unknown.
For more information check out my Data Product Innovation Summit: