By Stephen Brobst, chief technology officer, Teradata Corporation

When President Barack Obama or then-national security advisor Condoleezza Rice observed that intelligence agencies failed “to connect the dots” for either the botched Christmas bombing of Northwest flight 253 or the tragedy of 9/11, they evoked a simple children’s game of drawing lines on a page to complete a picture. Connecting dots about known terrorists’ plans, then, sounds simple.

Let’s be realistic. It’s not.

The biggest challenge our security organizations face involve overcoming cultural clashes and power struggles within and between them – and only strong leadership in Washington can address those issues.  There also are complex technology issues involved in deploying a comprehensive, responsive, and collaborative analytics system across multiple agencies.  This may be the easier problem to address.

After all, the commercial sector has been successfully collaborating across technology organizations (think suppliers and purchasers or merged companies) for years.  Intelligence agencies are awash in data from various legacy systems running multiple databases that are seen as barriers to implementing a practical, centralized repository of data.However, I know of one major healthcare provider that was able to consolidate operational data from hundreds of facilities into a single data warehouse. The company’s supply chain operation alone depends on 16 discrete databases, which only can retain data for 30 days. Now, all those databases feed into the central data warehouse and overall historical trends are visible to procurement analysts, resulting in savings amounting to millions of dollars through such things as tighter inventory management and improved contract negotiations.

Although saving lives rather than money is the primary goal of intelligence analysts, among the benefits prized most by the healthcare provider I mentioned is the “ability to analyze, understand, and respond effectively to complex market developments” as they are happening. Swap “terrorists” for “market” and you have exactly the kind of system that will help connect dots.

Data volume is not the problem

Some people expressed concern because Umar Farouk Abdulmutallab’s name was in the sprawling Terrorist Identities Datamart Environment or TIDE database of 550,000 people suspected of having connections to terrorist groups. It’s been said that the size of TIDE was the problem, allowing the failed Christmas bomber to board his flight.

That’s nonsense. For a well-designed data warehouse, getting actionable information from data on that many individuals is easy.

Many thousands of companies have hundreds of terabytes of information in their data warehouses. And there are increasing numbers of companies with petabytes.

There’s a major U.S. Internet site, for example, that maintains many petabytes of information on its 89 million active users worldwide and lists more than 200 million items online. Daunting, yes, but with the right tools, an information treasure chest. For example, the company not only knows that in 2009 the site moved more than a half million items related to the Twilight series, but also that 31,871 of them pertained to teenage heartthrob Robert Pattinson, while only 4,183 were about his co-star Kristen Stewart. This kind of granular detail is critical for detecting and reacting to market trends.

Modern data warehouse and business intelligence technologies are definitely up to the task to deliver real-time information to cross-agency intelligence analysts. These tools can be installed and would begin providing information within months of deployment. So, what is standing in the way?

Culture is the weak link

Historically, an enterprise’s culture is the culprit. Organizations are slow to change. Studies by The Data Warehousing Institute (TDWI) and others have shown that data warehouse deployments fail primarily for one or more reasons involving weak management support, lack of user involvement, staff turnover, organizational politics, or an inadequate grasp or ever-changing nature of the project’s scope. Low on the list of stumbling blocks is technology.

In discussing the intelligence failures of 9/11 in 2005, Sunil Desai wrote in Policy Review that “the essence of the problem is that the entire interagency community is dominated by individual agency cultures rather than a common interagency culture.” As evidenced by the recent Christmas bombing attempt, the interagency culture problem persists.

The Obama Administration is trying to encourage greater collaboration among intelligence agencies. While it is relatively easy to integrate data from disparate sources into a single repository, it is much more difficult to achieve interagency collaboration. The Project on National Security Reform has identified numerous structural problems among the various intelligence agencies. Managers and staff, for instance, are not evaluated on how well they collaborate outside their agency. Also, there are few, if any, mechanisms to work horizontally with other agencies. The group’s 2009 report repeated Desai’s insight that no interagency culture exists.

Intelligence professionals need to understand that their careers and people’s lives depend on interagency collaboration. Leadership from the highest levels must instill that change into their organizations and create mechanisms, including interagency data warehouses, for an interagency culture to flourish. When that happens, and with the right analytical tools in hand, intelligence analysts will have a fuller picture of terrorists’ plots to work with, not just the dots.

Brobst is chief technology officer for
Teradata
(
TDC
), a Dayton, Ohio-based company focused on
data warehousing
and
enterprise analytics
.