“Competitive intelligence is gathering information, which is analyzed to the point where you can make a decision,” says Leonard Fuld, president of competitive intelligence and research firm Fuld & Company. This includes gathering information about competitor’s products, pricing, business culture, and investments, as well as external factors like market conditions and government regulations.

More than anything, CI aims to eliminate surprises. “Companies seem to have been caught off guard more by new and disruptive technologies in the last five years,” says Jan Herring, former director of intelligence at Motorola. “As a result, senior management has become more appreciative of gathering intelligence and we’re seeing expanded areas of application.”

MORE: The next Premier of China speaks to U.S. business leaders

Herring, a former CIA officer, was asked to bring his government intelligence experience to Motorola in the mid-80s. “I believed that, much like governments, multi-national companies were going to need their own intelligence departments to be able to make the right decisions,” says Herring.

In many ways, competitive intelligence is as old as business itself. In the late 1800s, the Rothschild family sent its bankers to France to observe banking techniques and adopt the best strategies. However, Herring says the modern incarnation of competitive intelligence took root in the 1980s, pointing to the publication of Harvard Business School professor Michael Porter’s Competitive Strategy: Techniques for Analyzing Industries and Competitors. “It was the seminal document that caused everybody to focus on intelligence gathering as a profession,” says Herring.

Recession-proof business service?

While businesses have been slashing budgets in the wake of the recession, expenditure on intelligence has actually edged up. A survey of 400 global companies by Fuld & Company reports that in the last five years the number of companies that spend more than $1 million on CI has increased from 5% to 10% of all companies with CI programs.

The pharmaceutical industry accounts for 27% of companies that spend more than $2 million on competitive intelligence, far more than any other sector. “Pharmaceutical companies are probably the best at doing what we do now,” says Herring. “Plus there is a lot of public information out there they can utilize.” Predictably, the financial services world has seen the largest decrease in spending while technology is the fastest growing sector for CI programs in the last five years.

Pharmaceutical companies (and the outside firms they hire) perform competitive intelligence in a variety of ways. CI practitioners like Fuld attend scientific conferences, often hosted by drug companies looking to generate buzz for their latest product. They keep their eyes and ears open for gossip, insider information, and questions asked by attendees that might lead to a better understanding of what’s actually going on in a given industry.

“A lot of times, companies will hire us when one of their drugs is about to be approved for the market and they want to know how the competition will respond,” says Fuld. Other points of interest include results from clinical trials, articles in medical journals, and “messaging,” the words and rationale companies use to market a drug. Fuld will keep a close watch for news about mergers and acquisitions, government regulations, and anything else that could affect his clients’ prospects.

Another strategy is to monitor online voices, especially on blogs and social media. Fuld says Twitter is a great place to glean information about customers’ thoughts on a product and how the competition responds to them. “Patients will talk about their experiences, their illnesses, and their needs,” says Fuld. “They’ll be the ones who bring the drug to market.” Companies also use social media channels to send out alerts for their activities, such as notices for participation in clinical trials.

MORE: Marijuana’s new market

However, the CI industry is still figuring out how to use data from social media. Michel Bernaiche, CI practitioner and interim CEO of the Society of Competitive Intelligence Practitioners, says the amount of information available on social media networks can be overwhelming and thus hard to process. “The amount of information can be cumbersome, and the data analysis is not easily repeatable,” says Bernaiche. “It’s going to be a huge game-changer if we can figure out how to harness it, but I think we’re still a couple of years away from that.”

‘We’re not really spies, promise!’

Both Bernaiche and Fuld assert emphatically that gathering intelligence on the competition is not spying, nor is it unethical or illegal. Fuld says he does not uncover trade secrets, gather dirt, or mislead anyone about his identity when talking to people, so any information gleaned is fair game. He says he also makes sure to avoid the kinds of conflicts of interest that could arise from working for a client’s competitor.

A few years ago, corporate investigation and security firm Kroll Inc. came under fire for its work for Texas financier Allen Stanford. Stanford, best known as the mastermind of one of the largest and longest-running Ponzi schemes in U.S. history, hired Kroll to unearth embarrassing information on a senior State department official who was investigating him. Kroll obliged by reporting that the official’s wife was a lesbian who had left him for another woman, information that was patently untrue.

Bernaiche admits CI can involve gathering information and doing due diligence on some executives’ backgrounds, but not to uncover dirt. “We do it to find out what motivates their decision making and to possibly anticipate their actions,” says Bernaiche. “We look into their college background, who they hung out with, what they studied, and their previous decisions, you can often see a pattern.”

Fuld says he has received requests to dig up dirt but not in the way one would think. “I had one company who couldn’t believe their competitor could be such a low-cost operator,” says Fuld. “They seriously asked,  ‘We think that company is a money laundering front for the mafia, can you please check it out?’” Fuld says that while the mafia didn’t own it, he did find out why the company was able to keep costs low.

MORE: After Newtown: The business world’s way forward

For Bernaiche, sometimes gathering “intelligence” comes down to making good old-fashioned phone calls. “Marketing departments are great places to get information from,” says Bernaiche. “And I’ve found that sales people love to talk, they love to correct you.”

Bernaiche says he will sometimes make an intentionally false statement so that a competitor’s salesperson will correct him and give out information in the process. “Sometimes they’ll slip up and reveal information, but just because they give you information you’re not supposed to have, doesn’t mean you can’t use it.”