Palantir Connects the Dots With Big Data

A startup famous for its antiterrorism work is partnering with corporate and humanitarian clients.
March 9, 2016, 11:30 AM UTC
Palantir, Alex Karp
CEO Alex Karp and philanthropy team leader Karin Knox with colleagues (from left) Josh Harris, Nick Zamiska, and Ryan Taylor. Photo credit: Brad Wenner

Palantir Connects the Dots With Big Data

With a growing book of corporate clients and a high-profile Syrian relief project, the data-analysis startup is branching out beyond its roots in the war on terror.

When you live and work in Silicon Valley, you grow accustomed to a kind of semantic saturation from overused buzzwords. Terms such as “disruptive,” “innovative,” and “mission driven” come to mind—all favorites among Valley startups, whether they’re building operating systems for robotic arms or phone-based bowling games.

So it captures your attention when the CEO of one of the most buzzed-about of those startups sidesteps that kind of language and instead explains his company’s decision-making process as “80% Piaget and 20% Hobbes.”

The CEO is Alex Karp of Palantir Technologies, the Palo Alto–based data analytics company that may or may not have helped track down Osama bin Laden. Karp holds a Ph.D. in social theory, which explains why that Piaget-Hobbes formula (more on that in a moment) figures in his view of how to manage and give purpose to a business. And like the formula, Palantir’s version of the tech industry’s “change the world” ethos becomes more distinctive, and more of a departure from the Silicon Valley norm, as you dig into it more deeply.

Palantir, currently valued at about $20 billion, is one of tech’s biggest “unicorns.” Its two main products, Gotham and Metropolis, serve the same basic purpose—bringing together massive, disparate data sources and scouring them for connections and patterns that aren’t obvious to the human eye. And yes, those software suites are named after the home cities of Batman and Superman, respectively. There’s nothing ironic about the superhero references: Having deployed their products to address crises like cybercrime, natural disasters, and the ugly byproducts of civil war, Palantir’s employees firmly believe that they’re members of the Justice League.

Palantir Karin Knox, a leader of Palantir’s “philanthropic engineering” team.Photograph by Brad Wenner for Fortune

When business leaders talk about the promise of big data, they often point to Palantir, even though the company and its clients are generally tight-lipped about exactly what the company does and how. This much we do know: Commercial customers rely on Palantir to detect fraud, study consumer behavior, and search for a competitive edge. Clients range from J.P. Morgan Chase (JPM) to chocolatier Hershey (HSY) to Bridgewater Associates, a global investment management firm. Private customers now account for about 75% of Palantir’s revenue; the company said it had $1.7 billion worth of commercial bookings, or commitments to spend, in 2015. (For more, read “Big Data Projects With Big Potential.”)

But as recently as six years ago the company sold exclusively to the government sector. And that’s why Palantir’s mystique carries a unique tinge. The company has its roots in, and is still best known for, antiterrorism and spycraft. Intelligence and national security agencies use its tools to flag suspicious activities, tracking the movement of money, contraband, and shady operators. Put simply, Palantir’s technology can enable governments and companies to monitor people. And that means Palantir faces the same theoretical and practical quagmire that other technology companies are now grappling with—the challenge of balancing safety and privacy.

Apple (AAPL), of course, has drawn a line in the sand on this issue, diving into a legal battle with the Justice Department over iPhone security. Facebook (FB), Google (GOOG), Microsoft (MSFT), and other tech giants have lined up behind it. But Palantir’s orientation differs—and that’s where Piaget and Hobbes come in. Jean Piaget was the psychologist who espoused the merits of hierarchy-free play for children; Thomas Hobbes was the 17th-century philosopher who believed that a strong central authority was the only option for maintaining peace. Palantir hasn’t taken a public stance on the iPhone dispute. But it operates under the assumption that small Hobbesian incursions on privacy are necessary to preserve our Piaget-style freedom of action, including in business. Palantir co-founder Peter Thiel argues that smart intrusion ultimately means less intrusion. “The government was collecting a lot of data [in the war on terrorism], more than they could analyze,” says Thiel. “If we could help them make sense of data, they could end indiscriminate surveillance.”

“This is protecting civil liberties through the use of software,” says Karp, referring to the company’s partnerships with governments, “and therefore protecting societies from hitting the button against all civil liberties.”

For more on Apple’s privacy case, watch this Fortune video:

[fortune-brightcove videoid=4786638641001 height=”350″]

It’s a tricky balancing act, but it’s at the core of Palantir’s operations—and one that will guide the company as it takes on serious business challenges. Launched in 2004 by Karp and a group of techies led by PayPal (PYPL) co-founder Thiel, Palantir has raised about $2 billion in funding over the years from investors like hedge fund Tiger Global, Yelp (YELP) co-founder Jeremy Stoppelman, and Thiel’s venture firm, Founders Fund. The company will have to return some of that investment at some point, possibly through a public offering. As unicorn euphoria fades, Palantir may face increasing pressure to do so; indeed, in late February, Morgan Stanley (MS) marked down the value of its stake in Palantir by almost a third.

Palantir expects to reach profitability by 2017. It doesn’t report current revenue, but the company told Fortune that it brought in more than $1 billion in 2014 and that its 2015 commercial bookings nearly doubled from the previous year; that figure suggests that growth is steady if not explosive. To maintain momentum the company will need to keep expanding its commercial services—a challenging task when your corporate raison d’être prevents you from saying much about what you do. “We’ve always been bad at marketing and sales,” admits Karp.

But in interviews this winter, Karp and his colleagues gave Fortune a rare, close-up look at how they operate. They also took Fortune inside some of their humanitarian projects—most notably, an ambitious refugee-relief effort in Syria—which aim to show a skeptical public that Palantir’s data mining can be an unequivocal force for good.

It was a humanitarian project that first introduced Palantir to the U.K. division of Spain’s Santander Bank. In 2015, both companies signed on to a charity-funded initiative to look into connections between illicit financial networks and global human-trafficking rings. The initiative didn’t materialize—British restrictions on information sharing among banks got in the way—but Palantir’s software impressed Santander chief operating officer Juan Maria Olaizola. “You can say that we went from a blind date to marriage in less than six months,” says Olaizola.

Today Santander is using Palantir’s tools to figure out whether their customers are up to anything nefarious. Palantir’s analytics software can harness the bank’s proprietary data and match it against public information to check whether customers have been connected to bad acts, past or present. That “public information” typically includes criminal databases, huge streams of data from social media, and a variety of other sources. (These sources are the stuff that privacy advocates’ nightmares are made of, and Palantir and its clients uniformly decline to talk about them with specificity.) In theory, Santander or any other customer could perform this checking itself, but it would need to find engineers with the right expertise and spend a long time building code—a potentially ineffective and time-consuming process. Thanks to the effort Palantir put into Gotham and Metropolis, it has the talent and data ready to roll.

Tracing Patterns: Engineers in work pods at Palantir’s Palo Alto offices.Photograph by Brad Wenner for Fortune

A typical deployment starts with Palantir employees embedding with a customer, poring over potential data sources, and identifying problems. Olaizola describes Santander’s relationship with Palantir as “a joint effort of exploration and discovery.” That exploratory approach is central to Palantir’s process, a contrast to the classic fee-based relationship with a software provider. Typical commercial contracts with customers last an extraordinarily lengthy five to 10 years, though Karp says clients aren’t locked in if they’re unsatisfied: “We believe that a good partnership is one where a client can leave.”

Because the relationship may last for years, Palantir’s embeds can keep folding in new kinds of data and applying its tools to different problems as a client’s needs change. Palantir isn’t the only game in town: Competing vendors include big-data player Splunk (SPLK) and IBM (IBM), with its i2 software. But while some have complained about Palantir’s costs, its technology and people seem to command respect, even among companies that didn’t hire it.

Palantir vets its clients as tightly as they vet Palantir. The data challenge that a company’s case presents must be compelling, and the information that will be analyzed must be made freely and totally available to Palantir’s researchers, no strings attached. Those standards apply to humanitarian projects too: One key to Palantir’s new Syria initiative, for example, is a huge database assembled by enterprising researchers at the nonprofit Carter Center (for more, read “Palantir Turns Its Software Toward Syria”). The client’s and Palantir’s ideologies must align too, which sometimes means walking away from big contracts. When a tobacco company wanted to use Palantir tools, Palantir turned down the partnership, Karp says, for fear the company would harness the data to pinpoint vulnerable communities to sell cigarettes to.

Among clients who do make the cut, in most cases the “Why Palantir?” rationale is the same. The client has data, and plenty of it, and buying Palantir’s formidable analytics expertise is more cost-effective than developing its own. Zurich Insurance has worked with Palantir to develop data-mining software to help it price policies more accurately—an arena where tiny changes in risk assessment can create a big competitive edge. In 2015, TurboTax maker Intuit (INTU) used Palantir’s tech to investigate efforts by identity thieves to fraudulently file for tax refunds.

Palantir’s consumer expertise is growing too. Hershey uses Palantir technology to better understand and anticipate purchasing trends. (Fun fact: Palantir’s tools helped discover that when Hershey’s chocolate is placed next to marshmallows, sales go up.) The company has also partnered with payment processor First Data (FDC) on a product called Insightics, which utilizes First Data’s voluminous credit card records to provide advice to small businesses—helping decide, for example, whether an ice-cream store’s customers are dispersed enough geographically to justify opening a second store.

CEO Karp leads colleagues in a tai chi class at headquarters.Photographs by Brad Wenner for Fortune

On a Monday afternoon, Karp leads a tai chi class in a courtyard at Palo Alto headquarters, standing in front of a phalanx of 30 or so Palantirians. The rank and file closely follow their leader’s slow and subtle movements. He occasionally stops to adjust the position of an employee’s hands and posture. It’s guidance, not correction: As he later tells a reporter, “We are an antiauthoritarian culture.”

In other words, the culture is more Piaget than Hobbes. But Palantir’s roots are intensely Hobbesian. It’s impossible to write about the company without writing about the Central Intelligence Agency. While Palantir’s technology first took shape as a homegrown antifraud algorithm at PayPal, the CIA’s venture capital arm, In-Q-Tel, was its first outside investor, and until 2010, its only customers were in intelligence, law enforcement, and defense. (Such users now include the Federal Bureau of Investigation and the Navy Seals, among many others.)

In-Q-Tel has invested in more than 100 startups, but only one has been associated with the 2011 killing of Osama bin Laden. For years, tech and security insiders have theorized that Palantir’s analytics helped the government pick up bin Laden’s scent. It’s a link Palantir has never confirmed or denied. “I can’t comment on our specific national security successes,” Karp says when asked by Fortune. “Maybe a different way of answering is that not everybody likes our affiliation with national security, but we’re very proud of it … That also involves finding terrorists and sometimes taking them out.”

Whether or not those terrorists include bin Laden, its suspected role has made Palantir an object of intrigue, as have the eccentricities of its corporate culture. Karp’s wild hair and typical office garb (brightly colored tracksuits) may not stand out in Silicon Valley, but his doctorate in social theory from Frankfurt University does. References to the Lord of the Rings fantasy trilogy permeate the company. Its name stems from the palantiri, crystal-ball-like objects whose users can see events all over the world. And a sign at headquarters exhorts the staff to “Save the Shire”—a rallying cry about protecting innocents in a violent world.

Palantir’s public safety mission, expressed in a slogan referencing the Lord of the Rings trilogy.Photograph by Brad Wenner for Fortune

As Karp tells it, work for private-sector customers was almost an afterthought to saving the Shire. But other founders have said they always had the private sector in mind. In a speech to a Stanford University class in 2013, Palantir co-founder Stephen Cohen told students that even as they worked on antiterrorism projects, the founders also thought, “Let’s hope big enterprises use a Silicon Valley approach to understanding their data.” As Thiel puts it, “It turned out that effective analysis is very valuable in a lot of different markets.”

Not everyone is so sure that the connection between national security work and for-profit customers is a benign one. “Alex Karp regards the work of Palantir as sanctified,” writes Alec Ross, former senior adviser for innovation at the State Department and author of the book The Industries of the Future. “But I don’t believe that these kinds of capabilities will stay bottled up and only be put to work in the common interest. Perhaps we can trust Palantir not to engage in dodgy projects, but it won’t have a monopoly on this kind of intelligence capability.”

If there’s one place on the planet that can one-up Silicon Valley’s do-gooding lingo, it’s the village of Davos, where leaders who spend the year worrying about shareholder returns take a weekend off to worry about improving the world. This January at the World Economic Forum, the usual crowd of snow-boot-clad CEOs and politicos coalesced at the village’s conference center. But outside, in a downtown storefront, Palantir established a sobering but memorable public presence.

As heads of states and CEOs tromped through Palantir’s sleek, nearly all-glass entryway, they were met with bold-lettered words proclaiming: war in Syria, a growing international humanitarian crisis. Inside the pavilion was an interactive digital map of the Syrian region, which has been embroiled in war for five years and counting. Small blue dots marked areas controlled by opposition forces, red stood for Syria’s government, black was ISIS factions, and orange marked Kurdish territory. When visitors swiped along a timeline at the bottom of the large touchscreen, the colored dots shifted around, representing the changing spheres of control in the country.

Courtesy of Palantir/Carter Center

The display was powered by Palantir’s software. At the back end, thousands of data points—accounts from social media, NGO field reports, news reports—were updated in real time and converted immediately into a visualization of the complexity of the power struggle. (A new data point that will soon be added to the platform: the price of bread in different towns, since rising prices are often a sign that aid is needed.) Another display showed a large image of a mobile app, an early-warning system Palantir hopes will help aid organizations anticipate the location of the next big wave of violence, enabling them to provide aid to civilians even before the wave hits.

Even onlookers who hadn’t heard of Palantir—and there were many—were intrigued, and the buzz among influencers was notable, according to several Davos attendees. “What we wanted to do at Davos with our booth is say, ‘Look, this is a very difficult situation,’ ” says Karp. “ ‘But we can help every country spend less on resources and have a much bigger impact on mitigating the situation.’ ”

If it seems like a stretch to use the same software to sell chocolates, catch crooks, and provide tents and soap more quickly to refugees—well, it is and it isn’t. The Syrian civil war isn’t just one of our era’s most consequential humanitarian crises; it’s also the first such large-scale conflict to amass massive, messy mounds of digital data. (An example: There are more than 3,000 videos of testimony from Syrian defectors on YouTube.)

Palantir’s humanitarian efforts leverage its fine-grained ability to piece together critical patterns from those mountains of information. That, of course, is the very strength that prompts companies to hire Palantir. And projects like the Syria effort display those strengths in a very public way, without an obscuring scrim of confidentiality.

Just as important, doing good reinforces Palantir’s sense of mission, both within its own ranks and to potential clients. It’s the kind of opportunity that makes Karp wax almost poetic. “We are conviction addicts,” he says. “We really want to believe in what we’re doing.” You can tell people this in a marketing pitch, he adds, but there’s no substitute for the work itself. “When you work with them, they can see that you have conviction.”

A version of this article appears in the March 1, 2016 issue of Fortune with the headline “Palantir Connects the Dots.”