The government said Wednesday it arrested Lee Farkas, the former chief of wholesale mortgage lender Taylor Bean & Whitaker, on charges he “operated a sophisticated shell game” that sought to prop up his failing enterprise at the expense of investors and taxpayers.

Farkas was indicted on charges of conspiracy and bank, wire and securities fraud. He ripped off the Federal Housing Administration and investors in at least two financial firms, and tried to make off with funds from the Troubled Asset Relief Program, officials said.

Under federal guidelines, he could face life in prison if convicted. 

“The gravity of this fraud is really quite amazing,” said Assistant Attorney General Lanny A. Breuer (right). He said losses from the scheme will be measured “in the billions.”

Among other things, authorities said, Farkas filed false information to get TARP loans via an investment in TBW’s biggest lender, Colonial Bank of Alabama. Colonial failed last August in the third-biggest bank failure of the year.

Taylor Bean also was a huge lender under Federal Housing Administration programs. Officials said the FHA and Ginnie Mae, another government mortgage lender, lost some $3 billion in the Taylor Bean fraud. It is the FHA program’s biggest-ever loss, they said.

The Federal Deposit Insurance Corp.’s inspector general said in April that Colonial’s loss had increased by $1 billion since the bank’s August collapse.

Colonial losses on the Taylor Bean fraud were around $400 million, officials said Wednesday.

The mortgage firm was under financial stress for years before its failure, the Securities and Exchange Commission alleged Wednesday in a civil complaint against Farkas. The SEC said Farkas began overdrawing his accounts at Colonial in 2002, when Taylor Bean’s mortgage servicing income started running short of its outlays.

He then began stuffing Colonial with loans he delicately described as “crap,” the SEC alleges. He had managed to send the bank $1.5 billion worth of these loans by the time Colonial and TBW went belly-up last summer.

But before it collapsed, TBW briefly had a deal in place to inject $300 million into struggling Alabama-based Colonial. The deal was intended to help Colonial secure $550 million in TARP funding.

Officials in the Justice Department and the Securities and Exchange Commission on Wednesday called the Colonial injection a “sham” that was part of a wide-ranging fraud scheme to keep Taylor Bean in operation despite its own losses.

Neil Barofsky, the special inspector general of TARP, called Colonial’s late 2008 announcement that it had met the requirements for TARP funding “suspicious” and said the unusual claim helped investigators unravel the case. He said this spring investigators were looking at Colonial and TBW.

The announcement comes three months after the government arrested its first TARP fraud suspect.

The Feds said in March they arrested Charles Antonucci, former CEO of the closely held Park Avenue Bank, with self-dealing, bank bribery, embezzlement of bank funds and fraud, among other charges. Antonucci is accused of trying to defraud the TARP program of more than $11 million in taxpayer money.

Update 4:40 p.m.: A reader points out that I mangled an earlier explanation of the role of warehouse lending in the Colonial-TBW fiasco. I have removed that section because on reflection it was beside the point even when stated correctly.