But now, high-end retailers are looking further afield within Manhattan, going to where the newly affluent are moving and the hot new areas tourists are exploring, as they seek to grab their share of New York’s still growing luxury retail market.

Hudson’s Bay Co said on Wednesday that its Saks Fifth Avenue chain is adding a second Manhattan store, an 85,000-square foot spot in Lower Manhattan near the World Trade Center set to open in early 2016. (Full disclosure: Time Inc., Fortune‘s parent company, will move to the same building, Brookfield Place, next year.) That announcement follows news from Neiman Marcus earlier this month that in 2018, it was opening its first ever New York City department store, a 250,000-square-foot location in the Hudson Yards commercial development near the popular tourist spot, the High Line park on the far west side of Manhattan.

Others are similarly setting up in untapped parts of Manhattan, at least from a luxury retail point of view: Barneys plans to open a new 57,000-square-foot store in the Chelsea district at its original 1923 location (it left that spot in the early 1990’s) in three years. And Nordstrom’s (JWN) first Manhattan department store on West 57th Street, a 10 minute-walk from Fifth Avenue, will be a seven-floor location that anchors what will be one of the tallest buildings in the world, scheduled to open in 2018. (The new Saks and Barneys stores will be much smaller than their uptown counterparts, while Neiman and Nordstrom are going in with full flagship stores.)

These retailers are looking to serve the more recently affluent who don’t necessarily flock to the Upper East Side or Central Park West anymore, as older money does, favoring trendier areas. Indeed, Neiman Marcus was drawn to the Hudson Yards location in part because of all the high-end residential developments popping up all over that area.

In the case of the Saks store, the opening shows how much faith it still has in New York luxury’s growth even as it has been closing stores in recent years in secondary markets such as Cincinnati and Portland, Ore., bringing its store count down to 39 stores from 55 only seven years ago.

“This is a perfect storm for luxury retailers: the reclamation of previously undesirable parts of Manhattan by the affluent, an upswing in the spending by millennials and the growth in tourism,” Greg Furman, founder and chairman of the Luxury Marketing Council, told Fortune.

Tourism to New York has indeed skyrocketed, bringing with it splurging by monied Brazilians, Russians and Chinese visitors, among many others: spending by tourists rose 30.9% to $36.9 billion between 2009 and 2012, according to New York City data. And wealthy locals are growing wealthier: the mean income of the top 5% of households in Manhattan rose 9% in 2013 compared to only a year earlier, making an average of $864,394, according to U.S. Census data crunched by the New York Times.

To be sure, these retailers all had to be nudged with some added incentives: Neiman and Nordstrom were given the opportunity to anchor developments by companies eager to have high-end, prestigious anchors. And HBC, which is also relocating its corporate offices to the building that will house the Saks Fifth Avenue and opening a Saks Off Fifth outlet store, acknowledged that incentives from the city and state weighed in the decision.

Still, the key motivator in each case was Manhattan’s seemingly unstoppable growth as a high-end spending destination.

“This store will cater to what we believe is an underserved and rapidly growing downtown luxury market, ” said Richard Baker, CEO of HBC, of the new Saks store.