Colleagues ethically challenged? How to handle it

September 7, 2012, 2:13 PM UTC

FORTUNE — Dear Annie: I’ve never seen this problem addressed in your column before, but I’m hoping you and your readers can give me some pointers. I’ve been in my current job as a regional department head at a financial services company for about eight months now, and during that time I’ve repeatedly witnessed instances of less-than-ethical behavior on the part of both my immediate boss (who has been here for about 20 years) and a couple of the people under me.

It seems that some of these practices are just part of the corporate culture here, since no one but me seems to have any objection to them. So far, I’ve been “going along to get along,” basically turning a blind eye, but it’s making me uncomfortable. I’d like to take action, but I’m not sure what to do. Reporting the misconduct to higher-ups seems politically unwise, especially since I’m still relatively new here and probably viewed as highly replaceable. Is there any way to blow the whistle without also having to look for another job? — Uneasy

Dear Uneasy: You’ve picked an interesting moment to ask. As you may know, the Securities and Exchange Commission announced a couple of weeks ago that it will pay out its first-ever bounty of $50,000 — 30% of the amount collected in an enforcement action, which is the maximum allowed by law — to an anonymous whistleblower who reported financial wrongdoing. This has raised concerns among employers (maybe even yours) that the prospect of a monetary reward will prompt people to report wrongdoing directly to the SEC, or another government agency like OSHA, without first alerting their bosses.

MORE: Want to achieve something great? Stop working.

Moreover, according to employment law firm Seyfarth Shaw, the number of whistleblower complaints to regulators has been climbing, up about 20% since 2008 — but the number of cases that have been resolved has stayed flat, rising barely 0.5% over the same period. “Having so many open cases hanging around creates a really awkward situation for employers and employees alike,” notes James Curtis, a Seyfarth Shaw partner in Chicago. “The statutes prohibiting retaliation against whistleblowers carry heavy penalties, so companies have to tread very carefully to avoid even the appearance of punishing an employee who has reported misconduct.”

You don’t mention whether the behavior you’ve witnessed is illegal, or whether it’s merely unsavory. If it’s the latter, here’s a bit of background you might find intriguing: More than half (52%) of employees in companies with revenues of $5 billion or more say they have observed unethical (but usually not illegal) behavior by colleagues over the past 12 months, according to a study by nonprofit research group Ethics Resource Center. At companies with sales of less than $5 billion, the percentage is 45%.

That doesn’t necessarily mean big-company employees are less ethical. The same study noted that, the bigger the company, the more likely it is to conduct formal ethics training, which makes people more aware of standards and expectations in this area, hence perhaps more likely to speak up about infractions.

Overall, the five most commonly cited forms of bad behavior were conducting personal business with company resources, spotted by 29% of employees polled; abusive behavior toward coworkers or subordinates (22%); lying to employees (21%); illegal discrimination (18%), and Internet abuse (17%).

MORE: Telecommuting: Lower pay, but fewer annoying colleagues

Most alarming, the study says that retaliation against people who report wrongdoing to higher-ups is rising. More than one in five (22%) experienced some form of revenge in 2011, versus 12% in 2007, and 15% in 2009. The most common punishments: Being passed over for a raise or a promotion, being relocated or reassigned, or getting demoted, although “physical attacks against the reporter’s property” jumped from 4% in 2009 to a startling 31% last year.

So clearly, if you’re going to inform on your colleagues, you need to tread carefully. Peter Handal, CEO of Dale Carnegie Training, suggests approaching your response in three stages. “First, with the people under you, if the misconduct is relatively minor, why not just have a conversation about it?” he says. “Let your direct reports know where you stand.”

Take, for instance, using the Internet on company time. “If someone is doing a little last-minute shopping the week before Christmas, it’s probably best to just overlook it,” Handal says. “By contrast, if someone is looking at porn online in the office, you have a responsibility to put your foot down, in part because that could turn into a legally actionable situation, and in part because it’s just totally inappropriate.” In other words, as department head, you have the authority to make rules about what your team can or can’t do on your watch — so use it.

With your boss, the situation is a bit more delicate. “Before you go to higher-ups to report anything, make sure you’re right,” Handal says. “Ask this person, ‘Did I see you do what I thought I saw?’ Phrase it as a question, and give the boss a chance to explain — or, if the situation arose from a mistake, to correct it.”

If that has no effect, proceed to Stage Three. “Report the misconduct to someone higher up in the company, and ask to remain anonymous and keep this conversation strictly between the two of you,” Handal advises. “And again, phrase it as a question rather than an accusation. Say something like, ‘Here’s what I’ve noticed is going on — how would the company handle that? What are the next steps?’”

At that point, Handal says, “the company’s formal procedure for dealing with ethical lapses — and most big companies do have one — kicks in. The ball is in their court now, so step back and let the process take over.” If you’ve done that and nothing changes, he adds, “I’d think twice about continuing to work there.”

MORE: Leadership lessons from Nick Saban

If the behavior that’s bothering you is illegal, the sooner you speak up, the better. Senior management may not regard you as a hero, but you’ll have earned at least some grudging thanks. Notes attorney James Curtis, “It’s in companies’ own best interest to nip illegalities in the bud. Too often, the CEO never hears about wrongdoing until the regulators are already involved, and then it’s too late.”

One more word about the Ethics Resource Center study: The most-observed shady behavior — using company resources for personal purposes — was also the least reported, red-flagged by only 38% of those who witnessed it. (Bribes to public officials, by contrast, got reported 77% of the time, and bribes to clients 79%.) That means people in your position are making plenty of judgment calls about what’s worth reporting and what to let slide. The trouble with letting too much slide, of course, is that one day you wake up and you’re Enron. Good luck.

Talkback: Have you witnessed unethical or even illegal behavior at work? Did you report it? If so, what was the result? Leave a comment below.