After two commodity price spikes in four years, $100-a-barrel crude and $4-a-gallon gasoline are starting to seem like the new normal — the price we pay, unhappily enough, for an expanding economy. Every twitch in the Middle East sounds the alarm for a new oil shock.

But those cursing the wallet-thinning impact of high energy prices should be careful what they wish for. Sagging economic expectations, after all, are the likeliest route to lower energy costs, as we have seen with the recent decline in oil prices.

Retail gas prices haven’t come down much yet, averaging $3.91 a gallon at last check — a sign, says oil watcher Richard Soultanian, that “maybe even the refiners don’t trust the pullback in crude prices.” But if current trends continue, the pullback could go further than people expect.

There are plenty of signs economic growth is slowing. U.S. gasoline demand has fallen more than 2% over the past year, suggesting that high prices have already started to weigh on the economy. Uber-bear David Rosenberg cites recent declines in architectural billings, small business optimism, house price and manufacturing indexes. Goldman Sachs strategists note that while manufacturing employment has rebounded, recent gains are proceeding at just a quarter of the pace of the average monthly manufacturing job loss (a depressing 52,000) over the past decade.

Better late than never, economists everywhere are cutting economic growth forecasts that started 2011 in the 4% range – if probably not as much as they should. The wishful thinking impulse dies hard.

“Everyone’s demand and GDP forecasts for the next six to 12 months are just way overblown,” says Soultanian, whose NUS Consulting in Park Ridge, N.J., helps big companies plan for energy price shifts. “I think there is quite a good chance we will be revisiting $80-$85 oil by year-end.”

[cnnmoney-video vid=/video/smallbusiness/2011/05/04/smb_tips_gas_station_owner.cnnmoney/]

That forecast, mind you, presumes that the economy continues to limp along at a 2%-3% annual clip. But with the Federal Reserve scheduled to end its bond-buying program next month and federal support for overstretched state budgets ending as well this spring, a half-hearted U.S. expansion looks considerably more vulnerable than it did coming into 2011.

Fiscal tightening and monetary neutrality are hardly shocking developments, of course. But talk of the Fed heading for the exit ignores the impact of higher energy prices themselves. One rule of thumb has it that the economy goes into recession after the oil price triples — which it has done since its 2008 panic low.

And though almost no one talks about the dreaded double-dip recession any more, the soft data and the many shocks we have already taken this year, from the Mideast unrest to the Japanese tsunami, should remind us we are hardly out of those woods.

“I don’t see how we’re going to fill the vacuum” of all the federal money that has been pumped into the economy, says Soultanian. “That doesn’t mean you end up in a recession, but the chances are a lot higher than people appreciate. Maybe 25%?”

It is impossible to say how far oil prices might drop in a substantial U.S. slowdown, of course. A plunge to the 2008 lows seems implausible, but then until a few months ago so did a return to $4 gasoline.

In any case, a slowdown stands to undermine the fragile jobs recovery that Ben Bernanke has staked so much on. And the higher unemployment goes, the more people will be apt to remember that a little inflation pressure, as unpleasant as it is, beats the alternative.