CryptocurrencyInvestingBanksReal Estate

Can Wall Street thrive again?

December 12, 2011, 3:00 PM UTC

The financial industry is besieged by protestors. It’s also facing a slow-growth world and a wave of new regulation. In order to flourish again, the big firms must first change in painful ways.

FORTUNE — The brighter side of financial cataclysm wasn’t easy to see in late 2008 — the crisis was at its most acute, and no one knew if Armageddon lay ahead — but Barney Frank was upbeat. He told a consumer lobbying group, “Next year will be, I believe, the best year for public policy since the New Deal.”

For anyone on Wall Street, that cheery forecast from the proudly big-government chairman of the House Financial Services Committee was not good news.

Frank was wrong only on the timing: It took until 2010 to enact the Dodd-Frank law, the most sweeping regulation of Wall Street since the New Deal. (With his crowning achievement in place, Frank recently announced he won’t seek reelection next year.) The new law is so vast that it nearly equals all federal regulation of financial services from the previous 75 years.

That alone would have transformed the industry, but it’s only part one of a double whammy. The other element is an awful economic environment — slow growth in the U.S., slowing growth in Asia, and a European crisis so severe that, for all we know, Armageddon could be creeping up on us again.

Combine those forces, and Wall Street is a deeply different place from what it was three years ago. The changes are a mixed bag for investors and even for customers, who were supposed to benefit from the massive regulatory overhaul. Though the new rules are far from complete, Wall Street is already becoming smaller and less adventurous.

It’s also despised. The Occupiers may have begun to disperse, but the fury that fueled them hasn’t. The latest Trust Barometer compiled by Edelman, a communications firm, finds that the three least trusted industries in the world are insurance, banking, and financial services — Wall Street.

10 best stocks for 2012

The industry’s most immediate problem, worse even than its lousy reputation, is the terrible business climate. “The big firms are overextended, bloated in regions that are shrinking,” says Meredith Whitney, the analyst who forecast the subprime disaster in 2007. “In past years, 70% to 80% of Wall Street revenue has come from the U.S. and Europe. Both continents are in the process of multiyear deleveraging. The firms have gale-force headwinds against them.”

Today’s ultralow interest rates are another headache — a fact that surprises many people. Some think the Fed is keeping rates low in order to rescue the banks by enabling them to obtain funds at low cost. Trouble is, the rates at which banks lend those funds are also hitting record lows. The spread between rates produces what bankers call net interest income, and “it’s very hard to come by in this environment,” says a former top bank executive. That’s especially painful because “it goes straight to the bottom line.” Many Wall Streeters would actually love to see long-term rates rise.

Regulatory upheaval, meanwhile, is only getting started. Dodd-Frank requires hundreds of new rules to be written, and Washington is way behind schedule — partly because Wall Street is lobbying aggressively to shape those rules. Expect another two to five years before they’re finished. To see what’s taking so long, and why Wall Street is nervous about what’s coming, consider the new regulation with the highest profile of them all, the momentous Volcker Rule.

In concept it can be stated in one short sentence: Banks can’t trade for their own account. In practice, the current draft is 288 pages and includes over 1,000 questions to which banks and anyone else may respond. The Federal Deposit Insurance Corp. will announce a final rule sometime next year. Then the banks and the FDIC can start arguing over what it means.

As currently drafted, the Volcker Rule is “a complete game changer,” says Whitney. Beyond the ban on proprietary trading, for example, banks may no longer hold securities in inventory on the chance that a customer might want them; a customer must first state an intention to buy them. “I can’t have anything in the dairy case. When you order, I’ve got to go out and find the cow,” says Whitney. And that “slows the business down dramatically.”

So will other new rules, especially the higher capital requirements that regulators are imposing. The effects of entirely new regulatory bodies created by Dodd-Frank are still mostly unknown. The Financial Stability Oversight Council is just getting started. The Consumer Financial Protection Bureau doesn’t yet have a director. They, and the hundreds of new rules still to be written, will shorten Wall Street’s reach and hinder its speed. That’s what they’re meant to do.

The best and worst of Wall Street 2012

What is Wall Street’s business model in a world like that? The phrase you keep hearing is “back to the future” — making money on fees for underwriting, M&A advice, and investment management rather than on highly leveraged proprietary trading. Good news for high-net-worth individuals: You’ll be feeling lots more love. “Each of these firms is looking at wealth management,” says a former top executive at one of them. No wonder: It’s a high-return, low-volatility business. But building it is hard because those well-off clients are far more attached to their advisers than to the firms those advisers represent. Recruiting and developing an army of top-quality advisers take time.

A bigger challenge for Wall Street is that its turf is no longer the center of the financial universe. In 2005, five of the world’s 10 most valuable banks were American, including four of the top five, led by No. 1 Citigroup (C) and No. 2 Bank of America (BAC); none of the top 10 were Chinese. Today four of the top 10 are Chinese, led by No. 1 Industrial & Commercial Bank of China and No. 2 China Construction Bank. Only four are American, the most valuable of which, Wells Fargo (WFC), is No. 4. David Rubenstein, managing director of the giant Carlyle Group private equity firm, poses the key questions: “Is the U.S. still able to dominate global financial markets? We were 46% of the world’s GDP in 1960. Now we’re 21%. Can we still have virtually 100% of the world’s investment banks?”

The answer — no — is obvious. More broadly, Wall Street has to change in painful ways. The major firms, gloriously profitable just a few years ago, are not earning their cost of capital. They’re failing, and everyone seems to agree on their near-term future: lower returns and lower profits. The firms have to get smaller, cut expenses, live less large, pay people less. The glory days are over.

But hold on. Wall Street’s glory days are over every 10 years, like clockwork. They were over at the end of the ’70s, after a decade of market stagnation; again at the end of the ’80s, when takeovers and LBOs faded; at the end of the ’90s, with the dotcom bust; and now with the subprime disaster. Every time, Wall Street comes back in new ways that no one imagined.

That pattern is hopeful for the firms. For Barney Frank and the legions of new regulators he helped to create, it’s worrisome.

More Fortune Investor’s Guide 2012

10 best stocks for 2012

Expert roundtable

5 mutual fund all-stars

The triumph of Blackstone on Wall Street

New global hot spots for investors

Ramit Sethi: The new finance guru on the block

Where to put your money now

Investing after the Arab Spring

The best and worst of Wall Street 2011

Being a boring investor … and loving it

This article is from the December 26, 2011 issue of Fortune.