How Big Data Is Used To Help The CIA And To Detect Bombs In Afghanistan

An Extract From Bernard Marr's new book


This article is an extract from Bernard Marr's new book ' Big Data In Practice: How 45 Successful companies used big data analytics to deliver extraordinary results'


How Big Data Is Used To Help The CIA And To Detect Bombs In Afghanistan 


Palantir, named after the magical stones in e Lord of e Rings used for spying, have made a name for themselves using Big Data to solve security problems ranging from fraud to terrorism. Their systems were developed with funding from the CIA and are widely used by the US Government and their security agencies. Their annual revenue is reported to be in the region of $500 million and they are forecasted to grow even larger – at the time of writing (January 2016) the company are tipped to go public with an IPO and are currently valued at $20 billion.

What Problem Is Big Data Helping To Solve?

Initially working on tools to spot fraudulent transactions made with credit cards, Palantir soon realized the same pattern-analysis meth- ods could work for disrupting all forms of criminal activity, from terrorism to the international drug trade. Now, their sophisticated Big Data analytics technology is being used to crack down on crime and terrorism.

How Is Big Data Used In Practice?

Palantir build platforms that integrate and manage huge datasets, which can then be analysed by their wide range of clients – including government agencies and the financial and pharmaceutical indus- tries.

Much of their work is naturally veiled in secrecy, but it is widely known that their routines for spotting patterns and anomalies in data which indicate suspicious or fraudulent activity are derived from technology developed by PayPal (Peter Thiel, who also co-founded the online payment service, is a Palantir co-founder).

They have been credited with revealing trends that have helped deal with the threat of IEDs (improvised explosive devices), suicide bombers in Syria and Pakistan and even infiltration of allied governments by spies. The US Government are Palantir’s biggest customer, and their software has become one of the most effective weapons in the digital front of the “war on terror”. Marines, for example, have used Palantir tools to analyse roadside bombs in Afghanistan and predict attacks and the placement of bombs.

The data needed to support Marines in Afghanistan was often spread across many sources without one single interface to access and analyse the data. Therefore, the United States Marine Corps (USMC) charged Palantir with developing a system that could integrate these sources quickly. The aim was to improve overall intelligence and reduce the amount of time spent looking for information. As units are often working in areas with low bandwidth or with no bandwidth at all, the system had to work without being connected to base stations. The Palantir Forward system provided the answer to this problem, as it automatically synchronized data whenever the connection to base stations was restored. USMC analysts were able to use Palantir’s data integration, search, discovery and analytic technology to fuse the data and provide greater intelligence to Marines on the frontline.

A key philosophy of the company is that human intervention is still needed to get the most from data analysis – particularly when you have to think one step ahead of an enemy. To this end, they provide handpicked expert consultants to work in the field alongside their clients on data projects.

What Were The Results?

Using Palantir’s system, USMC analysts were able to detect correlations between weather data and IED attacks, and linked biometric data collected from IEDs to specific individuals and networks. None of this would have been possible without having all the data integrated and synchronized in one place.

Palantir have now raised $1.5 billion in venture capital funding, indicating an enormous level of confidence in their technology. And the power of their platforms is being recognized beyond the realm of law enforcement and defence; the company are attracting many corporate clients, such as Hershey’s, who are collaborating with Palantir on a data-sharing group.

What Data Was Used?

In the Afghanistan example, the data used included a wide range of structured and unstructured data: DNA databases, surveillance records showing movements, social media data, tip-offs from informants, sensor data, geographical data, weather data and biometric data from IEDs. A big part of Palantir’s success lies in pulling such massive data sets together effectively.

What Are The Technical Details?

Palantir are understandably secretive about technical details, which means I am unable to share details on how data is stored or analysed.

Any Challenges That Had To Be Overcome?

Privacy is a murky area in the Big Data world, and for companies such as Palantir that gather enormous amounts of data public perceptions surrounding their use of that data is bound to be a concern. The company were implicated in the WikiLeaks scandal, when they were named as one of three tech firms approached by lawyers on behalf of Bank of America seeking proposals for dealing with an expected release of sensitive information. After their name was linked to the scandal, Palantir issued an apology for their involvement.

Concerns are growing about government use of individuals’ data, particularly in the US and the UK, in the wake of the Edward Snowden NSA leaks. As such, Palantir need to tread a fine line between gather- ing the data necessary for the job at hand and avoiding mass invasion of privacy. It’s an issue that founder Alex Karp doesn’t shy away from. Speaking to Forbes a couple of years ago, he said: “I didn’t sign up for the government to know when I smoke a joint or have an affair.” And in a company address he stated: “We have to find places that we protect away from government so that we can all be the unique and interesting and, in my case, somewhat deviant people we’d like to be.”1 With the company’s reported IPO coming up, public perception is likely to be as important as ever and it’ll be interesting to see how they manage this.

What Are The Key Learning Points And Takeaways?

One of the key points that Palantir make is that human interaction with data is just as valuable as the data itself. This is true whether you’re fighting a war or trying to attract new customers to your product or service. There is a danger that we place too much blind faith in data itself, when, in fact, how we work with that data and make decisions based on it is the key.

Palantir also provide an excellent example of how data can be especially powerful when more than one dataset is combined. Working with just one dataset can provide a very one-sided view – often it’s the correlations and interactions between different types of data that provide the real insight gems.



1. Greenberg, A. (2013) How a “deviant” philosopher built Palantir: A Cia-funded data-mining juggernaut, greenberg/2013/08/14/agent-of-intelligence-how-a-deviant-philosopher-built-palantir-a-cia-funded-data-mining-juggernaut/, accessed 5 January 2016.

You can read more about Palantir at: Impact-Study-Fielding-an-Advanced-Analytic-Capability-in-a-War- Zone.pdf chunk-of-projected-400m-round-as-market-asks-who/ approaches-heres-what-investors-need-to-know/ 78423362

Bean small

Read next:

City of Chicago: An Analytics-Driven City