FORTUNE — For a trader playing the market, the temptation to “go rogue” is huge. Authorized trading is like a complicated, legal slot machine and requires the assumption of major risk on a regular basis. So, then, how do you fill your company with people who have the confidence to play the market but can also fight the temptation to throw good money after a bad trade?

The latest example of this dilemma comes from UBS trader Kweku Adoboli, who was arrested on September 15 and is now facing charges of fraud and false accounting. UBS (UBS) says he covered up unauthorized trades over the past three months that lost the company $2.3 billion.

This is particularly awkward for UBS as the bank recently overhauled its risk management system. Granted, preventing rogue trading is no easy task, “Nobody who’s in this game would be foolish enough to believe that it couldn’t happen to them,” says David Johnson, managing director in consultancy firm Protiviti’s Houston office.

Without knowing the details of UBS’ security measures, there are a couple of best practices banks should follow. Companies need to have security systems that flag suspicious trades, and on top of that, managers need to review trading activity. Every year or so, the company needs to do thorough internal audits to see which parts of their business may be vulnerable to unauthorized trading. Companies can even record conversations that traders have from their desk phones.

But beyond operational security, preventing rogue trading is a question of hiring and corporate culture. Managers must create an environment that counteracts the powerful psychological forces that encourage smart traders to make bad decisions that put the company at risk.

First off, the job selects for employees who believe they can beat the market. To be a good trader, you almost have to create a kind of mythology around your ability, says Johnson, “People who are in that business tend to be willing to place large bets, they’re very comfortable with that,” he says. The environment is high-pressure and competitive.

Besides, it’s a fair bet that managers do not root out unauthorized trades unless they lose the company money. “You haven’t heard of financial scandals where a rogue trader has earned $2 billion extra for the company,” says Barry Staw, a professor of leadership and communication at the University of California, Berkeley. Traders who have bet correctly on unauthorized trades are inadvertently rewarded, making the behavior tough to break.

But perhaps the biggest mental glitch at work is the temptation to make a series of irrational decisions after losing money. Almost everyone falls prey to this phenomenon, according to Staw’s research; it’s called “escalation of commitment.”

In trading, this means that even though trades are discrete events, it is near impossible for people not to factor a previous loss into their current decisions. The result is that when we want to dig ourselves out of a hole, we make decisions that a third-party observer could clearly identify as bad ones.

At a certain point, people hit a threshold where irrational choices actually become rational. Say a trader is losing money, and he begins to make riskier and riskier bets. He realizes that his career is in jeopardy and he might even be arrested. The penalty may not change much, for the trader, whether he has lost $1 billion or $2 billion. If that’s the case, once he crosses that threshold, it is rational to go double or nothing in an attempt to get out of the hole. Although there’s still plenty of money at stake for the company, the trader likely has nothing more to lose at this stage. And people operating in crisis mode tend to look out for their own skin rather than the common good.

It’s still unclear what happened with Adoboli, but in general, it is difficult to reverse this pattern of risky betting once a person feels trapped. One of the best ways to prevent this behavior doesn’t rely on security software but rather the culture in an office. Even though traders operate in such a high-pressure environment, they need to feel like they can talk to their managers about problems, insecurities about trades they might have done wrong, and near misses.

One of the risks of raising the penalty for mistakes is that traders may feel less inclined to come forward with small errors that could be learning opportunities, Staw says. Then the company only sees the huge ones that are too big to hide.

Investing is so prone to this kind of behavior that employers need to put all systems of checks and balances in place: extremely thorough hiring practices, an open culture of communication, strong technology to catch suspicious trades, and hyper vigilant human oversight.

As one of David Johnson’s clients once told him, “You have to hope that your traders are the finest moral people around. Then, you set up your policies and your rules as if they’re all lying, cheating crooks.”