In fact, Buffett seemed amazed that more people weren’t doing it.

“You would think that people would be lining up now to get mortgages to buy a home,” Buffett said at the conference. “It’s a good way to go short the dollar, short interest rates. It is a no-brainer. But so far, home construction pickup has been slower than I had anticipated.”

The advice seemed like classic homespun investment wisdom from Buffett, another example of how the average Joe could do better than Wall Street. “Worried that rising inflation or interest rates will crush your retirement account? You don’t need to buy a currency swap or a derivative. In fact, you can do better than the fancy hedge funders. Just buy a house, or refinance the one you have into a 30-year fixed mortgage, if you don’t have one already.”

But when it comes to buying real estate, the Oracle of Omaha’s crystal ball, as he seems to admit, is a little fuzzy.

Buffett does have some motive to push housing. He owns a home building company, a real estate agency, and a whole bunch of other companies that benefit when homes are built or remodeled, including paint company Benjamin Moore and building materials company USG. And the advice is not all that new for him. Last year, in his annual letter to Berkshire shareholders, Buffett wrote that some of his best investments over the years have been in real estate.

In theory, getting a fixed rate mortgage to hedge against inflation isn’t a bad idea. The idea is that as inflation rises, the value of your debt—relative to the price of your house or stock market portfolio or your paycheck—will fall, and it will become easier for you to pay off your mortgage. Of course, you have to hope that we have wage inflation as well as price inflation. If we just have the latter, paying off your debt will only get harder, and Buffett’s strategy will be a bust.

But here’s the real problem with Buffett’s advice on betting against the dollar: you have to buy a house, or own one. You can’t borrow against a house you don’t own. A bank won’t let you do that. And it’s not clear that a house would be the best investment if the dollar were to drop in value.

“Frankly speaking, I am not sure exactly what he meant,” says Jens Nordvig, a currency strategist at Nomura. “I guess he simply meant that 30-year mortgage rates at 4% are very cheap, and this is a way to borrow cheaply. But it is not really a short dollar position, unless you buy a foreign asset.”

So, while you will do better on the rent vs. mortgage payment part of the equation—rent will go up with inflation and your mortgage payment will remain fixed—you will do worse on the investment part. The extra growth you get on your stock market portfolio, compounded over 30 years, will more than make up for what you lose on rental inflation. And the higher inflation goes, the worse an owner will do.

Here’s the math: Inflation has averaged 2.8% a year since 1988, which is as far back as I had data on historical housing prices. Since then, housing prices have grown by 4% annually, or about 1.2 percentage points more than inflation every year. Stocks on the other hand have been up 10.3% a year, when you include dividends, or about 7.5 percentage points more per year than inflation.

Right now, inflation is at 1.7%. But if average inflation were to more than double to 4% over the next 30 years, a renter who put in the equivalent of a downpayment as well as annual principal payments into the stock market instead of toward a house would end up a little more than $415,000 richer 30 years later than someone who bought, even after factoring in the cost of renting. If inflation were to jump by 8%, the renter’s extra take would grow to $1.3 million.

My guess, and most people’s, is that stocks will not do as well over the next 26 years as they have done in the past, but the returns on all assets will likely be lower, housing included. And this is what you would get if you put your money in U.S. stocks. As Nordvig says, if inflation does take off in the U.S. and you have your money in foreign stocks, you will do even better.

What’s more, for this to work, the person who rents has to actually invest money they would have put into a downpayment into the stock market, as well as all the principal payments they would have made to pay down the debt. If you rent and don’t save, then buying a house will have turned out to be a better choice, inflation or no. Of course, if you buy a house and don’t make your payments and end up losing the house, borrowing money to buy a house won’t work out too well either.

Lastly, at least for now, high inflation doesn’t seem like a problem. In the week or so since Buffett made his housing comment, interest rates, which many have long predicted are headed up, have gone south. And there are new concerns about deflation, not inflation. Still, the inflation worry worts won’t go away. So if you want to worry about inflation, you’ll have company. But buying a house shouldn’t make you feel any more protected.

My advice to the legendary stock picker: Stick to your day job.