Estes started the company, which is headquartered in Franklin, Tenn., to make what is called cognitive computing a commercial reality. If you watched the star turn of IBM’s Watson on Jeopardy! in 2011, you’re already familiar with the concept of a computer that can learn by experience and respond to queries that don’t neatly fit its programming. With this kind of artificial intelligence, a computer can begin to understand intent, so that when a terrorist sends a message to another that says, “The baby is in the carriage,” a computer can mark the seemingly benign exchange as suspicious. Estes started his company at the turn of the millennium because he thought that all software would eventually teach itself. Three years later the Pentagon came knocking on his door with several small projects in hand and the Sept. 11, 2001, terrorist attacks freshly seared into its institutional memory.

“September 11 showed us the consequences of not being able to act on information,” says Estes, who serves as the company’s chief executive. “The desire to make software that could learn was the impetus for the company, but after 9/11 it became a national impetus.”

Digital Reasoning moved quickly to seize the opportunity. In 2004 it landed a prime contract with the U.S. Army’s National Ground Intelligence Center. In 2010 it received an injection of cash from In-Q-Tel, the Central Intelligence Agency’s venture capital fund. Working for the U.S. national security apparatus—its sole customer—allowed the company to turn its increasingly deep understanding of cognitive computing into high-powered analytics tools for the intelligence community. The Department of Defense deeply desired prescience—a way to hint at the course of global events before they actually happen. Digital Reasoning delivered that with software that could, in an instant, process large volumes of disparate, disorganized, and seemingly unrelated information and discern from it intangible connections between people, places, and events.

In 2012 the FinTech Innovation Lab, a mentorship program in New York City, invited the company to meet and exchange ideas with people in the financial services industry. After talking with representatives from Goldman Sachs, Credit Suisse, and J.P. Morgan, Estes decided it was time to bring his company’s technology to the private sector. Digital Reasoning had spent a decade behind the veil of government secrecy, he thought; now it was time to expand into an industry with an appetite for intelligence as voracious as any military’s: Wall Street.

Over the past 18 months a handful of the world’s largest financial institutions, including Goldman Sachs and Credit Suisse, have begun deploying the very same advanced analytics tools that Digital Reasoning first developed for national security purposes. Their goal, Estes says, is to analyze every shred of digital traffic coursing through their networks to hunt down and expose potential rogue traders, market manipulators, and SEC rule violators within their own ranks.

$121 MILLION Amount paid by Goldman Sachs and Bain Capital to settle a 2007 lawsuit accusing them of collusion.

Estes calls the practice “proactive compliance.” With smart software to maintain constant vigilance over every email, instant message, media report, and memo sent within a company, financial firms can pinpoint a person’s intentions to engage in prohibited or illegal activity before they develop into infractions and the subpoenas start flying. In military intelligence work, “You’re basically trying to figure out who talked to who, who else knew about it, and did they act upon it,” Estes says. “That kind of forensic stuff is very similar to the investigations you might have at a bank.”

Today financial firms manage compliance in a number of ways. Some hire teams of analysts to manually read through randomly selected emails, an expensive proposition, even for a financial institution. Some use sophisticated algorithms to scan trading logs and other structured data to look for suspicious trading patterns, yet with 70% to 80% of an organization’s data unstructured, a great deal of potential evidence lies untouched. The largest problem may be that both methods are fundamentally reactive: By the time analysts or algorithms detect potential evidence of wrongdoing, it usually has already happened.

$1 BILLION
Amount by which J.P. Morgan increased spending on “internal controls” in 2013

“There’s a realization that there has to be a more efficient way—not just from a cost perspective, but from a timing perspective and a quality perspective—to handle compliance,” says Brian Gudofsky, a managing director in Credit Suisse’s technology investment-banking group.

The solution is to detect intentions, not evidence, an approach that should give any management team pause. Conventional software methodically scans for specific keywords—“leverage,” “budget,” “broker,” “favor”—and offers up every piece of correspondence that includes those terms. Digital Reasoning’s system, which it calls Synthesys, constantly weighs each piece of correspondence against the next to determine probable context. It takes recent news headlines, such as an erupting financial scandal, into consideration. It takes note of how personal relationships evolve within the company and flags abnormalities, such as if people in disparate parts of an organization suddenly begin to email one another with alarming frequency. All suspicious activity is collected for review by human analysts, who are often employees of the company in question and therefore subject to scans. As it’s used, the system refines its results based on previous successes to better look for telltale signs of collusion, unreasonable trading, or information leaks from within.

Digital Reasoning’s early success in the private sector marks a much larger attitudinal shift among companies of all stripes that are looking at new technologies to help them keep employees on the right side of the law but, perhaps as important, prove to customers, clients, and regulators that they are actively doing so. “We see this enormous demand for what we call ‘policy analytics’ alongside data analytics,” says Danny Weitzner, a principal research scientist at MIT’s Computer Science and Artificial Intelligence Laboratory. “Organizations need to be able to not only comply but to show the public and regulators that they’re doing the right thing with data.”

Nine months ago, Weitzner co-launched TrustLayers, a startup based in Cambridge, Mass., that also helps companies catch rule breakers within their own organizations. The company’s technology focuses on preventing violations, often accidental, of corporate data-use policies—a growing priority in industries such as health care where access and use of certain types of data are tightly governed by law.

The notion of demonstrating compliance, rather than merely achieving it, has become increasingly important to companies, says Jim Hendler, a professor in the computer science department of Rensselaer Polytechnic Institute. Hendler, an adviser to TrustLayers, works with IBM on commercial applications for Watson. “We want to show that we’re using data lawfully,” he says, “not just show that we’ve locked it down the way regulators want us to lock it down.”

$14.4 BILLION Worldwide spending on business intelligence and analytics software in 2013

Digital Reasoning took on its first Wall Street customer in 2012. Today it counts a half-dozen large financial institutions as clients and has offices in London and New York City. The company’s next challenge is to expand the scope of its technology to develop analytics tools focused on market intelligence, says Marten den Haring, a senior executive. Such software would allow investment banks and hedge funds to use the same data processed for compliance purposes to map events that could influence trading decisions, giving them an edge in the market. Digital Reasoning’s engineers are also looking at applications in the health care and energy industries, den Haring adds, though he is reluctant to offer further detail.

“It’s a fun time to be in this space,” he says. “By no means do I think we’ll be the only company in this space 24 months from now.” When that competition arises, Digital Reasoning will undoubtedly be the first to detect it.

If you liked this story, check out “Big Data’s Dirty Problem” by Verne Kopytoff

This story is from the September 1, 2014 issue of Fortune.