When people think of big data and analytics, it tends to be exclusively in terms of utilizing data they have collected themselves, such as customer journey, demographics, and so forth. The problem with this though, is that it doesn’t give you the full picture. In order to use data to its full potential, companies need to be collecting external data too, and incorporating it into their models for analysis.
External data is any data generated from outside an organization. It can come from a variety of sources, and open data initiatives are now plentiful, making available a wealth of external data for analysis. The US government alone makes available more that 131,000 datasets on the federally-run website data.gov. One example of external data is weather, which is particularly useful in supply chain management. UK based supermarket chain Tesco, for example, is renowned for their use of weather data to drive richer insights that help them to predict sales and stock requirements. They reported in 2013 that they had managed to save £6m ($7.5m) per year and reduced out-of-stock by 30% on special offers. In fact, in a recent survey of supply chain professionals by the UK Met Office, 47% cited weather as one of the top three factors external to their business that drives consumer demand. Of these, 57% said they had better sales forecast accuracy, 51% that they had better on-shelf availability, and 43% that they had reduced waste.
However, it appears they are - or at least were - in the minority. In 2012, Forbes writer Dan Woods argued that companies were suffering from he called ‘data not invented here syndrome’, and were failing in their data initiatives before they focused solely on using data created inside the four walls of a business. But is this still the case, or are organizations finally starting to realize the benefits of external data? Just 16% of respondents to the Met Office’s survey said they use commercial weather data, but is it being realized in other areas? We asked seven of the world’s leading data experts from best-in-class organizations whether they felt companies were looking at external data as much as they should be.
Vishal Singh, Country Head of BI & Data Analytics at Moët Hennessy USA
Most companies today have some way to go before they can comprehensively leverage external data. Today most companies focus on external data in silos, leading to a strategy disconnect. Strong data management and data governance strategies are key to deriving full benefits of external data. Most organizations have multiple siloed sources of data, vendor driven portals which have market intelligence and competitive insights, and internal systems have internal data. Unless organizations have full control on the data, having a holistic view of the business is not possible. In today’s fast changing business landscape where vendors and technologies keep changing, organizations would greatly benefit if they were to elevate the capabilities of their internal teams to manage their data, both internal and external.
Dr. Robyn Rap, Business Intelligence Analyst at Indeed.com
I think that depends greatly on the company. Here at Indeed we have a research arm called Hiring Lab, which provides economic insights from our data with support from Business Intelligence. We found, for instance, that the number of jobseekers looking for work outside of Great Britain jumped after Brexit. I'd be willing to bet that most companies have stories in their data. It just takes resources and creativity to find them and highlight what's interesting.
Saurabh Bhatnagar, Senior Data Scientist at Rent The Runway
No. External data is hard to come by and even harder to debug/clean. Things like US Census data, Maxmind or Nielson are extremely useful but those examples are few and far between. Companies tend to trust internal data for a reason. Sometimes that reason is simply not knowing of the existence of a useful external dataset.
Walter Storm, Chief Data Scientist at Lockheed Martin
Aside from obvious macroeconomic and specific market data, there are very few external indicators that I’ve found to have real predictive power for the business. The case for retail and ad agencies is obvious as you look at the clickbait we’re bombarded with on a daily basis, but I find social media to be misleading and of questionable value for my particular industry.
Matt Kautz, VP of Business Intelligence at Machinima
Some companies focus too much on external data, some focus too much on internal data. Identifying and integrating new data sources into the decision-making process is the primary job of the BI leader. BI must drive action, so it’s up to us to be sure that every data source being used or that could be used to make decisions is included in BI reporting.
Kevin Harrison, Assistant CDO of the State of Illinois
In general there is not enough focus on external data. Partially because it is extra work, and partially because the end-users do not necessarily see the value of it, until there is an actual example of how it helps the business. However, it appears to me, that once external data is brought into an environment, the shift is dramatic and occurs quickly,
Mario Trescone, Senior Director of Business Intelligence and Data Analytics at YMCA of the USA
Today, I believe most companies are aware of the importance of external data to stay informed and competitive, especially the large to mid-size organizations, however, they may not be structured or disciplined enough to know how to capture or apply external data within their Business Intelligence function. It really depends on how the organization is structured, its data culture, and how they ultimately are defining the Business Intelligence role. Throughout my career, the positions I have held in the realm of data analytics and research, connecting external data with operational data was/is at the heart of almost everything I do. How else do you put context around the trends you are seeing related to the current and forecasted demand of your products and services. Having a thorough understanding of your market, using both primary and secondary data, allows you to maintain a pulse on the market, ensuring relevancy of your organization today and into the future. So for anyone in the Business Intelligence role, if you are not connecting your operational information to external data sources you need to start.