Welcome to the 4% return market

November 19, 2013, 3:00 PM UTC

FORTUNE — The stock market probably isn’t in a bubble, but there’s a bigger risk you need to consider: That buying now will mean much lower returns over the next few years.

The valuation on the S&P 500 is still reasonable enough – a P/E of 16.6, based on trailing earnings, which is only slightly higher than average.

That’s good news if you’re worried about a crash. But if you want 10% long-term returns? Or even 8%? You’re going to be disappointed.

MORE: Stocks not in a bubble? Hold on there Yellen

In fact, try 4%.

On average, going back to the 1930s, the market has risen just 4% a year in the following decade when stocks have traded for a similar P/E as they are now, according to Birinyi Associates.

And among a growing group of forecasters, 4% is becoming something of a consensus.

Bridgewater’s Ray Dalio last week said that’s what he expects from the market for the next decade.

MORE: Why should JPMorgan and Twitter have to pay more taxes?

Robert Shiller says his calculations suggest stocks will rise about 2.5% a year for the next decade, plus inflation, which has recently been averaging 1.5%. Cliff Asness, who runs AQR, which manages one of the largest hedge funds in the world, says they will rise 4.5% annually on average.

4% is not horrible. After the two big stock drops of the 2000s, the market basically went nowhere for the following decade. In a 4% market, with compounded returns, the money you put in the market now will be worth nearly 50% more in a decade.

So while it might not be time to be scared of the market — the economy is improving — it’s certainly a time to be less excited about it.