The New York Times reports today that the Federal Housing Finance Agency, which oversees mortgage giants Fannie Mae and Freddie Mac, is planning to argue in a lawsuit that America’s biggest banks failed to check the health of mortgages before they securitized them and sold them to investors at the height of the housing bubble. The suit, which officials expect to file in the coming days, is targeted at Bank of America (BAC), JPMorgan Chase (JPM), Goldman Sachs (GS) and Deutsche Bank (DB) and others.

But don’t expect the banker outrage to last too long. This suit follows several others being pursued in what’s become the “Giant case against evil banks.” Everyone’s joining the party — but the piling on is just resulting in white noise. Meanwhile, other parties who certainly deserve blame for the financial implosion that occurred after the housing bust remain virtually untouched.

Last month, American International Group (AIG) sought more than $10 billion in losses in a suit filed against Bank of America, accusing the nation’s biggest bank and its Merrill Lynch and Countrywide Financial units of misrepresenting the quality of mortgages bundled into securities and sold off to investors. In May, the US Justice Department filed a civil suit against Deutsche Bank. The 50 attorney generals are in the final stages of negotiating a settlement addressing abuses by the largest mortgage servicers, including Bank of America, JP Morgan and Citigroup (C). And countless other suits by homeowners and investors are helping to keep the lawyers swimming in money.

As The Times notes, the forthcoming FHFA lawsuit accuses the big banks of failing to perform due diligence required under securities law – for instance, missing evidence that borrowers’ incomes were either inflated or falsified. Securities backed by the mortgages quickly sunk in value when borrowers found themselves unable to make their mortgage payments. Fannie and Freddie, which the U.S. government rescued in 2008, lost more than $30 billion partly as a result of the deals.

The mortgage giants and the FHFA might be blaming the banks for the ills the agencies have encountered, but it’s hard not to wonder if so many others should be named in the suit – from the rating agencies that labeled these investments safe to the borrowers who lied about their incomes and bought way too much house for their own good.

And it will be difficult for the mortgage companies to prove that they were completely unaware that they were making very risky, if not bad, investments. In 2006, regulators warned Freddie that its purchases of subprime securities were becoming too risky to manage. And yet, the company continued gobbling up what turned out to be bad debt.

What’s more, the case against the big banks will likely be hard to prove. Financial service industry executives could argue that losses on mortgage-backed securities had much more to do with the downturn of the broader economy and housing market than how such securities were packaged and sold.

Certainly the outrage is warranted. But should we stop with the big banks?