A number of retailers have cut their spending forecasts in recent weeks, including Best Buy Co., Lowe’s, and more. Lowe’s saw “a greater-than-expected pullback in DIY discretionary spending, particularly in bigger ticket categories,” Marvin Ellison, the chairman, president, and CEO said Tuesday. Bloomberg reports that retailers that cater to the upper middle class, including Apple, Coach, and Nordstrom have seen significant drops in sales over the past three months.

In fact, those earning at least $100,000 in annual household income have been reigning in their spending since the summer, Kayla Bruun, senior economist at decision intelligence company Morning Consult, tells Fortune. Morning Consult research finds the group is pulling back the most on physical goods and housing (meanwhile, spending on experiences is holding strong), and doing so at a greater rate than lower-earning households.

That’s important, because wealthier Americans typically have more excess money to spend to keep the economy chugging along. When they pull back, as Bloomberg explains, that could be a bad sign. That’s especially true for our current economic environment. The wealthiest Americans’ “surge in consumption in the post-COVID recovery has been unprecedented,” according to a recent research note from Morgan Stanley. In fact, from 2020 to 2022, households in the top 20% of income have accounted for 45% of all consumer spending in the U.S. between 2020 and 2022. Typically, this group accounts for around 39% of all spending.

“Thrifty behavior has been climbing up the income ladder,” Morgan Stanley’s note reads. Though middle- and high-income households are still holding onto some excess savings from the pandemic, they “are less willing to spend it.”

Even the high-earning aren’t immune to inflation

Despite strong economic data, survey after survey has shown six-figure-earners down on the economy and struggling to keep up amidst years of high inflation and rising interest rates. As lower income workers benefit from larger income gains, wealthier Americans feel, comparably, that they are worse off.

“While this group remains in a relatively comfortable financial position compared to lower earning peers, they are not entirely immune to factors like prolonged elevated inflation, rising interest rates, and cooling wage growth that may be dampening spending this holiday season,” Bruun says. She notes that while inflation has cooled, the cost of living is still higher than it was, leading more consumers to walk away from a purchase when the price is high.

And the full effects of recent interest rate hikes by the U.S. Federal Reserve are yet to be felt in full, economists say. Housing, in particular, remains unaffordable for many—things haven’t been this bad since the 1980s. The typical family cannot afford to buy a home, and those who lucked out and bought when interest rates were historically low are now locked in to houses they may not like much. Wealthier households are more likely to be homeowners than lower-income groups, and they have benefitted disproportionately from the recent explosion in housing wealth, driving their consumption. But Morgan Stanley expects that to slow “as the boom years of the post-COVID services recovery moves further into the rearview mirror.”

“Our analysts who cover restaurants and luxury brands both point to an aspirational consumer that has begun pull-back spending on fine dining and luxury shopping,” the note reads. “As wealthy households approach satiety as well, aggregate consumer spending will shift into a lower gear.”

Bruun points to higher credit card interest rates as another pain point. “Most high earners have income leftover after paying for monthly expenses, but they appear to be more inclined to put this excess income toward paying off past debt rather than using it for new spending,” she says.