Why only half of Americans gain from the stock market party

FORTUNE – As U.S. stocks soar to new highs, economists argue the rally makes people feel richer. And when they think they’re wealthier, the theory is that they’ll spend more on everything from homes to cars and clothing.

The problem with that idea is fewer and fewer Americans actually have any stake in stocks. Only about half, 52%, say they actually have money invested in equities — the lowest level since 1998, according to a Gallup Poll released last week.

In recent years, overall stock ownership has declined. Between 1998 and 2008, Gallup found at least 60% of Americans reported they owned stock. That share fell in 2009 when the nation slipped into recession and saw the unemployment rate double. By 2012, stock ownership declined to 53%, as the unemployment rate remained elevated.

MORE: Enron: The real story behind Jeff Skilling’s big sentence reduction

Among the hardest hit were 30-to-49-year-olds, as well as among middle-income Americans. To be sure, a steady stream of individual investors have returned to the stock market. They pumped more than $60 billion into mutual funds and ETFs that hold U.S. stocks – more than any full calendar year since 2004, according to research firm TrimTabs.

This comes as the Federal Reserve continues buying up billions of dollars worth of bonds each month, a move to stimulate the economy and get investors to take on more risks. But while stock prices soar, it may not be a big enough incentive to lure many more investors back to the market. Lydia Saad of Gallup notes that Americans’ withdrawal from stocks may be correlated to high unemployment. So long as joblessness is a problem Americans may simply be unable to afford to invest in stocks. It could also be that investors still think the market is too risky after stocks nearly collapsed during the financial crisis.

In many ways, the same can be said for the nation’s housing market. What’s interesting is that stocks, and perhaps more so homes, are generally major sources of wealth for most Americans. Home prices have steadily risen since last year, but the share of Americans who own their homes is at its lowest level since 1995. This comes as large private investors buy up foreclosed homes at deep discounts, and as more people rent than buy.

MORE: The epic inside story of fraud at a generic Lipitor giant

All this says a lot about American wealth. True, higher home prices have returned many borrowers to positive equity; they now owe less on their mortgages than their homes are worth. This may give them the incentive to eventually sell and buy another home, but higher prices don’t appear to have drawn many more buyers back to the market. The home ownership rate was 65% during the first three months of the year, down from 65.4% a year earlier and the lowest level since the third quarter of 1995, according to the Census Bureau’s latest statistics released last month.

Like stock ownership, home ownership appears more a function of the health of the job market than any rise in prices. Which means so long as unemployment remains a problem, it puts the biggest sources of wealth farther from reach for most Americans.

Subscribe to Well Adjusted, our newsletter full of simple strategies to work smarter and live better, from the Fortune Well team. Sign up today.