FORTUNE — The Federal Reserve’s annual stress tests turned out to be more unpredictable, and less favorable for the banks, than predicted. That’s good. Economic downturns are unpredictable, so the test should be too. Not that bank investors liked it. Even the shares of banks that passed the Fed’s tests dropped on Thursday.
That’s a reversal. In the past year, shares of financial firms have risen 26%, significantly more than the market as a whole. Much of that rise recently was based on the belief that banks would emerge from the stress tests with the okay from the Fed to return gobs of money to shareholders in the form of higher dividends and more share buybacks. Billionaire investor Warren Buffett has said that he finds the obsession with dividends and buybacks odd. Investors can get their money back whenever they want. All they have to do is sell.
But for the banks, not quite six years after the financial crisis, the ability to return capital to shareholders, rather than needing to hang onto it to protect their butts, was supposed to be another key sign that the nation’s financial firms were safe investments again. Analysts predicted that the 30 largest banks would return a collective $75 billion to shareholders in the next year.
MORE: Citi and four other banks stumble in stress tests
Instead, the stress test once again pointed out that the banks still aren’t as healthy as some think. Keefe Bruyette & Woods, an investment bank that specializes in following financial firms, called the results of the test “below expectations.”
The Fed approved just $55 billion in payouts. A good chunk of the miss was Citigroup (C), which was expected to hand out a little over $8 billion in dividends and buybacks. But Citi had its plan rejected. Two other banks, Bank of America (BAC) and Goldman Sachs (GS), only passed the Fed’s test after agreeing to cut back what they would pay out to shareholders. Even JPMorgan Chase (JPM), which passed both the first and the second parts of the stress test, ended up announcing it would devote nearly a billion less to dividends and share buybacks than most expected.
The Fed also predicted that the banks’ legal troubles were far from over. As part of the stress tests, the Fed estimated that the nation’s largest banks could pay another $150 billion covering legal fees and buying back soured mortgages over the next two years.
MORE: Banks may be lending too much
Then there are interest rates, which are likely to continue to rise in the next year. Higher interest rates could eventually be good news for the banks, but many predict initially they will cause trouble. Already, higher rates have significantly cut mortgage refinancing activity, which has been a big money maker for the banks in the past two years. Loan growth in general has remained slow. What’s more, the Volcker Rule, which restricts the banks’ trading operations, and other post-financial crisis regulations will officially go into effect later this year. That might further pinch profits.
At the same time, investors seem to be pricing a lot of growth into banks stocks. The shares of the nation’s six largest banks have an average price-to-earnings of 15. Those same banks have $6.7 trillion in assets on which they earned an average return of 1.2% last year, or around $80 billion. To get to $92 billion, or 15% more, banks would have to get their return up to 1.4%, higher than it was before the financial crisis. The other option is that banks would have to do a lot more lending. Given the headwinds neither higher profitability nor more lending looks all that likely. In fact, you don’t have to construct a stressful scenario for banks to disappoint investors in the next year. Analysts only expect earnings at the six biggest banks to reach $82 billion in 2014.
Bank stocks have been on a roll for a while. They may face a climb from here.