Where have all the shoppers gone?

Damn slow, indeed. At a time of year when U.S. retailers should be riding the annual wave of back-to-school spending and gearing up for the bonanza of the ever-expanding holiday season, they are instead gloomily explaining continued lackluster results to Wall Street. Companies from Target to Kohl’s to Coach have reported disappointing sales for several quarters running. Wal-Mart, the world’s largest retailer, with $279 billion in sales in the U.S. alone, has seen a drop in store traffic in each of the past seven quarters. Once-shining brands such as Staples and Abercrombie & Fitch are closing stores—a reversal from the “build it and they will come” mentality that formerly served as a strategy. Even Apple, the king of sales per square foot, has seen a decline in same-store sales this year, according to Customer Growth Partners. Overall, according to eMarketer, retail spending in stores is expected to grow just 3.7% in 2014—a fairly anemic rate some five years past the depths of the Great Recession. No wonder Kip Tindell, CEO of the Container Store, while reporting yet another quarter of poor numbers in early July, complained that the retail business was in a “funk.”

What are the reasons for this malaise? Industry analysts and executives describe a perfect storm fueled by an unfettered addiction to promotions and discounts, continued economic instability, and the fact that there is simply too much store space for the current population. The result, says Mark Cohen, director of retail studies at Columbia University and former CEO of Sears Canada, will be a “Darwinian struggle for survival,” which may ultimately take down some of the best-known brands. Yes, e-commerce continues to grow rapidly, now accounting for an estimated 6.4% of sales, or $304 billion in 2014. But the bigger e-commerce gets, the more it is disrupting the industry as we know it.

There are, of course, winners as well as losers in this new world—not only the obvious online powers, such as Amazon, which continue to siphon off market share, but also more traditional brick-and-mortar plays like Dollar General and T.J. Maxx. And there are lessons in those success stories. U.S. shoppers haven’t gone away altogether. But there is a “sea change in how people are shopping,” says Daniel Busch, an analyst with real estate analysis firm Green Street Advisors. Retailers must change dramatically to reach future customers. To better understand how the business is evolving, we identified four major (and interrelated) trends that are roiling retail—as well as possible solutions to each.

PROBLEM NO. 1: Death by Discount
It’s the oldest strategy in merchandising: offering shoppers an artificial price lower than “retail” to spur our primal desire for landing bargains. But over the past decade, say analysts, things have gotten way, way out of control. Virtually every retailer—at both the high and the low end—has fallen so deeply into the trap that discounting has become an expectation of customers rather than a bonus. “There’s price deflation caused by ever-increasing discounts and price competition, which lowers the average transaction value,” says Columbia University’s Cohen. “That means you have to sell more things just to stay even.” This phenomenon has been spurred in recent years by the ability of shoppers to “showroom,” or compare prices on the web.

The most obvious beneficiaries of the discount trend are the off-price retailers, which sell designer fashion and home goods at lower prices than customers typically find at department or specialty stores. A star pupil in this class is TJX, which runs the T.J. Maxx and Marshalls chains and is likely to overtake Macy’s in revenue this year. There are now about 2,000 T.J. Maxx and Marshalls stores, compared with 1,680 just five years ago.

In an attempt to win back some share, the department stores that the off-price chains have squeezed are getting in on the action. Saks Fifth Avenue, a part of Canadian retail conglomerate Hudson’s Bay Co., now operates 39 department stores—compared with 53 five years ago—but has boosted the number of its OFF 5th outlet stores from 55 to 75 over the same period. And Nordstrom, long known for its high-end quality, expects that by 2016 it will have twice as many Nordstrom Rack outlets as full-service department stores.

While mall traffic overall continues to fall—some 15% of malls are expected to close or convert to other uses in the next decade, according to Green Street—outlet malls are booming. According to the International Council of Shopping Centers, there are now 340 outlet centers in the U.S., up from 311 in 2007. Developers like Tanger Factory Outlet Centers and Simon Property Group say they have plans to build many more in the coming years. That will only exacerbate the downward pressure.

SOLUTION: End the Race to the Bottom
Finding a way out of the discount trap has been elusive. The most daring attempt to do so—former J.C. Penney CEO Ron Johnson’s decision in 2012 to abruptly end most sale pricing in favor of “everyday low pricing”—was an unmitigated disaster. Johnson, who uncovered the shocking statistic that 72% of Penney’s products were sold at 50% off or more, sought to do away with what he called “the disease.” There was just one problem: If you’re the only one trying to change things, it doesn’t work. In the first year of Johnson’s experiment, Penney’s revenues dropped by a stunning $4.3 billion. Those sales were eagerly scooped up by the likes of Macy’s and Kohl’s, who responded to Johnson’s move with—you guessed it—even more discounting. Macy’s recently lowered its 2014 sales forecast, and Kohl’s is struggling to post consistent gains. Johnson, who was forced out in 2013, may have failed on execution. But the big retailers would be wise to study his analysis of the discount dilemma.

At least one expert sees a “Darwinian struggle for survival,” which may ultimately take down some of the best-known brands.

PROBLEM NO. 2: Too Much Space
Who would ever have imagined the decline of the supercenter? For much of the past three decades it was a no-fail concept. Pick a location with cheap real estate and a large nearby population, build a giant box with a vast supply of shelf space, and wait for the shoppers to show up in droves. Wal-Mart rode the supercenter strategy to No. 1 on the Fortune 500. Target and Best Buy followed suit and blossomed as well. But the era of megastore dominance may be coming to an end.

A big reason is that today the U.S. has far too much shopping square footage for its population—52.4 square feet per capita, compared with, for example, 16.4 square feet in Germany, according to data from research firm CoStar and the International Council of Shopping Centers. It’s not just supercenters either. The number of shopping centers with more than 50,000 square feet in the U.S. has doubled in the past 30 years, though growth has flattened recently. The first principle of U.S. retail has seemingly always been “more and bigger.” But given the ease of buying anything from a stapler to a toilet online, a continually growing footprint only adds up to empty parking lots.

SOLUTION: Shrink to Grow
Home Depot CEO Frank Blake recognized the square-footage dilemma earlier than most. When he decided in 2008 to stop opening new stores (with exceptions for certain demographically shifting areas, such as North Dakota), most analysts and employees alike thought he was nuts. Wasn’t the point of any public company to actively grow its business? “It was the right decision for us,” Blake says, “but it was painful. All the muscle memory in the organization was around building new stores.” Instead, Home Depot has invested that capital in both improving its existing stores and in building a better online presence. Under Blake, who is stepping down Nov. 1 after seven years in the job, Home Depot’s stock doubled, showing Wall Street that there may be another way to think about growth.

Other retailers are slowly getting the message. Two years ago Gap returned to same-store sales growth after closing dozens of U.S. stores. And growth in store counts at the 100 largest retailers by revenue has slowed to less than 3% from more than 12% three years ago, according to a recent Moody’s report.

PROBLEM NO. 3: The Nonrecovery Recovery
It should surprise no one that lower-income shoppers continue to be price-sensitive. Why? The economy, stupid. Yes, the U.S. is growing again after the trauma of the Great Recession. But not everyone is feeling it. The bottom 20% of Americans in terms of income saw their real income grow only 19.5% between 1967 and 2012, while the top 5% earned 88% more. What’s more, the way people use their disposable income has changed. For instance, think about the “smartphone effect.” With $100 or more per month going to service plans, apps, and music, consumers have less money for a new pair of shoes.

Those economic trends help explain the ongoing stagnation at the likes of Target, which posted negative comparable sales for 2013 and has followed that up with two worrisome quarters this year. Not only are there too many big-box stores (see problem No. 2), but they are also mostly located in places that require a long drive—an increasingly troublesome proposition in an era of high gas prices and food-stamp-benefit cuts.

SOLUTION: Assume This Is the New Normal
A morose economy has been a boon to Dollar General. Over the past five years the ultra-discount retailer has opened 2,500 new stores (for a total of 11,388), while its same-store sales have risen consistently. It succeeds by being hyper-local with fixed, low prices. Most of its stores are in municipalities with fewer than 20,000 people, and many are within walking distance of population centers.

Much of the rest of the retail world, meanwhile, spent precious time waiting for customers to come back to them. Few are counting on that anymore. Wal-Mart, for instance, is now trying to counteract the success of Dollar General and other dollar-store chains by speeding up the rollout of its own small-format stores in population-dense locations: Its goal is to have 700 by February, compared with around 400 at the beginning of 2014.

In an environment of perpetually yo-yoing consumer confidence, retailers must either win clearly on price and convenience, or wow shoppers with such an innovative product or unique buying experience that they can’t help opening their wallets. That leads us to…

PROBLEM NO. 4: Stunted Evolution
If any single trend in retail is crystal clear, it’s this: Online sales will continue to boom. E-commerce sales should leap 61% by 2018, according to eMarketer, reaching $491.5 billion. Brick-and-mortar sales, still a vastly larger number, are expected to grow a decidedly more modest 12.8% to about $5 trillion, from $4.43 trillion, over the same period.

Adapting to this brave new world has proved difficult for many in the sector. While virtually every retailer participates in online sales, not all do it well. And the long tail enabled by the web means that a growing portion of overall sales is going to retailers with no physical stores. The digital experience is changing shopping habits in other ways too. Even when customers do choose to purchase items in a store, they frequently do research online first and then go straight for the item without stopping to browse. That reduces spontaneous purchases and harms retailers without great inventory controls.

SOLUTION: Blend the Old With the New
Malls are particularly vulnerable to the ease of online shopping. Busch of Green Street Advisors estimates that about 15% of items typically stocked in malls are now regularly purchased online. Some mall operators have grasped that reality, investing in their “A” malls, which cater to the high end, and adding new features and services such as gyms and wine bars. “You can’t have a salad and a glass of wine online,” says Michael Glimcher, CEO of mall REIT Glimcher Realty Trust.

That’s the right idea. To thrive in an online world, retailers must enhance the excitement of the physical shopping experience while at the same time smothering customers with convenience. Happily, some major chains have lately been making better use of the very asset naysayers say condemns them to irrelevance: their stores. For instance, Wal-Mart’s online business is now growing faster than Amazon’s, thanks to big investments it has made to enable customers to place orders online and pick them up in many of the 3,300 or so Wal-Mart supercenters that dot the country. And Macy’s uses most of its 810 stores to help it fill e-commerce orders more quickly and not have to move as much unsold, extra merchandise on clearance. Getting better at this “omnichannel” approach is essential for any retailer planning to succeed with physical stores.

Is there hope yet for U.S. retail? Only if retailers themselves understand that the classic assumptions about the way people shop are no longer valid. “People haven’t stopped wanting things, and they won’t stop wanting things,” Cohen says. “The economy will eventually repair itself. But where are they going to transact? That is the question.”

This story is from the September 22, 2014 issue of Fortune.