But the Congressional Oversight Panel also warned that the toll could be far higher – and that as a result the U.S. financial system is still in a “precarious place.”

The panel’s estimate of the losses that could be facing Bank of America , JPMorgan Chase , Wells Fargo and Citigroup is in line with those produced on Wall Street. Indeed, the panel’s approach involves picking some of those reports and averaging out the results.

The oversight panel estimates the four biggest banks will lose $36 billion on loans that were bundled into securities and sold to Fannie Mae and Freddie Mac, the government-backed mortgage investors, and $16 billion on those sold to private investors.

The big four banks have taken losses or reserved for future losses on $21 billion of that sum, meaning they face some $31 billion of possible losses in coming years, the panel said. It notes Wall Street expects these losses to hit earnings but not leave the banks short of capital.

That said, the panel – which was put in place to oversee the Troubled Asset Relief Program, the bank bailout enacted two years ago and now in runoff – warned that there’s much we don’t know about the mortgage mess.

[cnnmoney-video vid=/video/news/2010/10/14/n_tarp_kaufman.cnnmoney/]

Even as it accepts Wall Street’s assumption that legal action against the banks won’t likely lead to huge losses, it points to one high-profile dispute as an example of the risks facing large firms.

To put in perspective the potential problem, one investor action alone could seek to force Bank of America to repurchase and absorb partial losses on up to $47 billion in troubled loans due to alleged misrepresentations of loan quality. Bank of America currently has $230 billion in shareholders’ equity, so if several similar-sized actions – whether motivated by concerns about underwriting or loan ownership – were to succeed, the company could suffer disabling damage to its regulatory capital.

So what to do when facing potentially large but unquantifiable risks? Why, run more stress tests:

It is possible that widespread challenges along these lines could pose risks to the very financial stability that the Troubled Asset Relief Program was designed to protect. Treasury has claimed that based on evidence to date mortgage-related problems currently pose no danger to the financial system, but in light of the extensive uncertainties in the market today, Treasury’s assertions appear premature. Treasury should explain why it sees no danger. Bank regulators should also conduct new stress tests on Wall Street banks to measure their ability to deal with a potential crisis.

As it happens, bank regulators are doing just this, with the Fed having said Friday it would force banks to pass new, self-administered stress tests before they can hope to raise their dividends.

But the oversight panel, formerly run by Harvard’s Elizabeth Warren and now in the hands of departing Sen. Ted Kaufman, D-Del., still sees considerable cause for concern.

The American financial system is in a precarious place. Treasury’s authority to support the financial system through the Troubled Asset Relief Program has expired, and the resolution authority created by the Dodd-Frank Wall Street Reform and Consumer Protection Act of 2010 remains untested. The 2009 stress tests that evaluated the health of the financial system looked only to the end of 2010, providing little assurance that banks could withstand sharp losses in the years to come. The housing market and the broader economy remain troubled and thus vulnerable to future shocks. In short, even as the government’s response to the financial crisis is drawing to a close, severe threats remain that have the potential to damage financial stability.