FORTUNE —  In another sign that the U.S. housing market is recovering, the closely watched S&P/Case-Shiller index report on Tuesday noted home prices over the summer posted its biggest percentage gain in more than two years. This is good news, but a crucial segment is missing out on better days: First-time homebuyers.

In October, the share of buyers purchasing their first home dropped to 34.7%, the lowest point in at least three years, according to a survey of real estate agents. This compares with 37.1% in June and the 40% range it has historically hovered around.

The decline comes as the sale of non-distressed homes has risen rapidly during the past year. It climbed to 64.7% in October, compared with 55.7% in February, according to the Campbell/Inside Mortgage Finance HousingPulse Tracking Survey. Current homeowners and even investors, which typically target heavily discounted foreclosed properties, have been riding the recovery wave. In October, current homeowners made up 54.2% of non-distressed sales, a jump from 50% in June. And investors made up 12.2% of sales, higher than 11.3% during the same period. 
By contrast, first-time buyers fell behind. Their share of purchases fell to 33.6% from 38.7%.

MORE: How to fix the charitable deduction

Since the start of the year — when the overall housing market was relatively weaker — the U.S. Federal Reserve has been drawing attention to the problem. Now as things start looking better with home prices and sales edging up, the momentum could be deflated. 
First-time buyers play an important role in the market. To be sure, these households are younger, with relatively new credit profiles and lower-than-average credit scores. Of any group, though, they are best positioned to absorb the excess inventory of distressed homes, says Tom Popkin, the survey’s research director. Investors could do the job, he says, but that may not help bump prices up as bulk and cash sales usually mean the properties are purchased at deep discounts.

A few reasons explain why first-timers have been left out. For one, the deteriorating finances of the Federal Housing Administration. About 50% of first-time buyers rely on the government agency for financing. The FHA doesn’t directly make loans to them, but it safeguards lenders if borrowers stop paying. In turn, this insurance makes banks more willing to offer mortgages to borrowers with lower credit scores or incomes, as well as with down payments as low as 3.5%.

All that is a big draw for first-time buyers, but FHA will have a much harder time supporting such households. The agency has struggled with huge losses from souring home loans — a problem that has existed since at least 2009, leading many to wonder if the agency will eventually need a government bailout.

Earlier this month, experts sounded another alarm. The FHA has used up its reserves — its capital cushion fell to -$16.3 billion at the end of fiscal 2012, according to a study by an independent actuary. The agency has tried to strengthen its finances. In recent years, it has established minimum credit scores for eligibility; required bigger down payments for riskier borrowers and raised premiums on insurance required for borrowers making smaller down payments. Starting next year, annual premiums will increase again, raising borrowers’ costs by an average of $13 a month.

MORE: Wells Fargo CEO: Why Americans are saving so much

This might help save FHA from financial abyss, but it also makes getting a mortgage more difficult and costly for first-time buyers. More broadly, the worry is that if the FHA goes down, so might the overall housing market.

Even if FHA weren’t in turmoil, first-time buyers may not be much better off. 

This of course depends where you live. It’s one of the cheapest times to buy, but prices across certain metropolitan areas — including New York, Los Angeles and Boston —  are still out of reach for households just starting out. As The New York Times  pointed recently, when prices peaked, the median cost of a single-family home nationwide was $226,000 — four times the median income. This figure has dropped significantly, to $166,000, or 2.7 times the median income of $61,592. But places where residents typically fill very specialized and higher-paying jobs, owning a home becomes a lot less affordable: In San Francisco, the median home costs about is 4.89 times the median income; in New York, it’s 4.62 times; in Boston, it runs 3.7 times.

But if buyers don’t live in a big and expensive city, there’s also the burden of student loans to consider. As total student debt surpass the $1 trillion mark, younger households scrambling to repay their loans may take longer than previous generations to buy a house, economists say. With banks more choosy over who it lends to, indebted graduates may take longer to save for a down payment. They might also take longer to qualify for mortgages.

So as the housing market turns a corner, it’s the young, the educated, and those living in America’s biggest cities that probably won’t be part of it. At least not for a while.