How oil’s dramatic plunge has changed the energy equation

Goldberg had just launched his BBL Commodities Value Fund the previous September, but he was hardly a newcomer to oil markets. He had spent the previous decade as a trader, first for Goldman Sachs and then for Swiss commodities giant Glencore. His strategy for BBL was to trade oil “across the barrel”—refined products as well as crude itself—and to stay focused on short-term opportunities while keeping one eye on the horizon. “Our views are three months, six months, maybe a year,” says the 33-year-old Goldberg. “And we

are very sensitive to day-to-day, week-to-week fluctuations. I’ve never understood the point of putting a position on until I actually thought it was going to make money.” That January he rode bullish bets on natural gas and U.S. distillate prices to a 7% monthly gain. But he began to realize that something bigger might be brewing in crude.

Goldberg and his team watched closely as the stockpiling accelerated. By the end of the first quarter, the trend was clear. A gusher of new supply driven by the stunning growth in U.S. shale oil production—accounting for most of the 80% spike in overall U.S. production since the end of 2008—was beginning to overwhelm demand. And yet oil prices continued to defy gravity, propelled by fears of supply disruptions. Russia’s conflict with Ukraine and the emerging threat of a new Islamic state in northern Iraq caused crude to hit new highs in late June.

The trader stayed patient, waiting for a catalyst that would spur a drop. By the early fall Goldberg noticed that the cash prices being paid for physical oil were significantly lower than even the suddenly weakening futures prices. He became more confident that it was time to be short oil.

Then the bottom fell out. On Nov. 27, the Organization of the Petroleum Exporting Countries announced that, contrary to the assumptions of much of the oil industry, it would not cut its production to defend higher prices. Crude went into free fall. The price of a barrel of West Texas Intermediate (WTI), a benchmark for so-called light sweet crude oil, tumbled from its June high of $108 to a low in January of $44. Goldberg was well positioned for the crash. His fund made 26% in December alone and finished with a return after fees of 51.3% for the year, pushing assets up to $540 million.

Now the question for Goldberg and others is, what happens next? What are the implications of a 50% markdown of crude—in terms of cost structure, demand growth, and production? Will prices remain low for a sustained period or rebound quickly? To answer that, you have to study oil’s new calculus. For the near term anyway, expect volatility, not consensus. “The only thing that there’s agreement on is that it’s a lot more challenging than the old math,” says Dan Yergin, the vice chairman of information and analytics company IHS and the author of the essential oil-history tomes The Prize and The Quest.

For most of the players in the multi-trillion-dollar oil and gas industry, crude’s swift plunge has been almost as surprising as it has been costly. In November, S&P Capital IQ surveyed energy professionals and asked them what they thought the price of Brent crude would be at the beginning of January: $50, $80, or $100 per barrel? The most common answer (78%) was $80; just 2% picked $50. This conviction was based on a couple of assumptions: First, that the cost of adding new production, in U.S. shale fields and elsewhere, was somewhere around $75 to $80 per barrel. Second, that Saudi Arabia, the driving force within OPEC and the shock absorber in oil markets for the past 40 years, would cut production if necessary to keep prices from falling too low.

Simon Dawson—Getty Images

Blinded by these beliefs, the industry as a whole failed to recognize what, in hindsight, should have been obvious: that a developing glut made a price correction almost inevitable—especially with the Saudis signaling that they had no intention of cutting back production. “This was what they call in science an ‘available insight,’ ” says Jeremy Grantham, co-founder and chief investment strategist of asset manager GMO, which oversees $120 billion, and a widely respected identifier of market imbalances of all types. Grantham laments that he, too, failed to predict the price plunge.

The fallout from the drop has been swift and brutal. Publicly traded oil companies have lost billions in market value, and both public and private firms are moving aggressively to cut capital spending budgets for 2015—laying off thousands of workers and shutting down hundreds of rigs. This is especially true in the new boomtowns that have powered the shale oil revolution in the U.S. (For an on-the-ground report from North Dakota oil country, see “Waiting for the Reckoning.”) If the downturn drags on, there will almost certainly be bankruptcies and acquisitions.

“What we’re seeing is a textbook implosion with regard to exploration and production capital spending domestically because the industry was leveraged to very high oil prices,” says Bill Herbert, a senior researcher at Houston oil and gas investment bank Simmons & Co.  “And the E&P companies have been living outside of cash flow since the implosion of 2009, when the financial system essentially collapsed.”

Indeed, a recent paper by IHS concluded that spending on production growth in the U.S. from 2009 through 2013 had exceeded cash flow by an astounding $272 billion—and at least 40% of that was raised by taking on debt. Additional credit will presumably be hard to come by. But industry observers are anticipating a wave of opportunistic investments by private equity firms. Blackstone, for instance, reportedly just raised $4.5 billion for a new fund focused on energy investments.

Meanwhile, the industry is taking a hard look at costs. “Companies are reorienting themselves to a low-price environment,” says Yergin. “A decade ago they had to adjust to a higher price, and costs went way up.” But even before last year’s dive in prices, says Yergin, controlling costs had become the No. 1 issue for top execs at the major energy companies. “If that was a preoccupation before the price collapse, it’s now an obsession,” says Yergin.

Even as companies move quickly into survival mode on the ground, the industry is struggling to get a handle on future prices: Will the market’s corrective forces bring supply and demand back into alignment in a matter of months? Or will this be more like 1986—an eerily familiar scenario in which an OPEC decision to keep pumping oil after a flood of new supply ended up tanking prices for years? After bottoming—at least temporarily—in late January, benchmark prices rallied in early February, with WTI reaching $53 and Brent at $62 as of presstime.

But crude production continues to surge as a result of wells already drilled—and stockpiles of the excess continue to grow. In early February the amount of oil in storage in the U.S., excluding the strategic petroleum reserve, reached 417 million barrels, according to the U.S. Energy Information Administration, the highest level at that time of year in at least 80 years. Barring a major disruption of supply elsewhere in the world, it’s unlikely that prices will see a strong rally until those inventories begin to recede. But the slowdown in drilling should eventually put the brakes on production growth in shale. Unlike traditional onshore oilfields, which might have an annual production decline of 5% or less, shale oil wells often decline more than 50% in their first year.

The oil market remains in what’s known as contango—with the future price of crude trading at a higher level than today’s spot price. That suggests a belief that prices will rebound, at least somewhat. In mid-February, for instance, December 2017 Brent crude futures were trading above $73—or about 18% higher than the spot price. This has led traders to take oil off the market—in some cases by leasing supertankers for extra, floating storage capacity—in hopes that they will be able to sell it back later at a higher price. “It’s not a mystery,” says Chris Bake, a member of the executive committee of Vitol, the world’s largest independent oil trader. “We do it here because the market is asking us to do it. It’s saying, ‘Gentlemen, we do not have a home for this oil today. Can you please take it off my hands?’ ”

Daniel Shea

The outlook for consumption is murky. The International Energy Agency has lowered its demand forecast several times in the past couple of years, and in a recent report cautioned that both China, the biggest demand driver, and the global economy more generally were becoming less fuel-intensive. Still, the IEA projects demand to grow by more than 7 million barrels per day over the next five years. While lower crude prices mean cheaper gasoline in the U.S., which should spur demand, that’s not so much the case in other countries, where taxes represent a higher portion of the cost of gas (as in Europe).

A pair of recent reports suggest that the industry can stay in hunker-down mode for a while—as long as prices don’t crash much further. In January energy specialists Wood MacKenzie analyzed its database of 2,222 oil-producing fields around the world and found that a mere 0.2% of the world’s supply would be operating on a cash-negative basis at $50 per barrel for Brent. At $40, the number rose to just 1.6%. And a study by IHS concluded that wells accounting for 47% of new U.S. oil production in 2014 could break even with the price of WTI below $61.

Once supply and demand come back into balance, points out Fadel Gheit, Oppenheimer’s senior oil analyst, prices should gravitate toward the marginal cost of production of new barrels. But where that ends up is another subject of debate. Five years ago, says Gheit, the industry needed oil at $90 to justify the development of new production of U.S. oil shale. Now that figure is significantly less, he believes, and continued technology gains will drive it lower. “I’ve been following this industry for 30 years,” says Gheit, “and I can tell you that the shale revolution has changed the calculus. In my view shale will never die. It’s here to stay.” He forecasts higher prices but not a return to triple digits.

Grantham has a different view. After studying oil prices over long periods, the GMO chief strategist has come to believe that there have been two major paradigm shifts when oil reset at higher baseline levels. For decades, he says, the base price, calculated in today’s money, was $16. From there, oil occasionally doubled or dropped by half, but tended to return to $16. The rise of OPEC in the 1970s changed the game, says Grantham, and the baseline price jumped to $35. “Whenever you were down by half, of course, it was the end of the world, it was a New World Order,” says Grantham. “And whenever you went up, the same in reverse.”

Just after the year 2000, Grantham maintains, the cost of oil inflected as finding new supplies became more challenging and expensive. That drove the base price up to $75 or $80 per barrel. Since then we have experienced both the doubling (when oil surged to nearly $150 in July 2008) and the halving (when it crashed below $40 in December 2008). Despite the gusher of U.S. shale oil production, Grantham believes that prices are likely to reset higher again—to a baseline above $100—because, outside of shale, finding new oil is getting harder. And there won’t be enough shale production to cover the declines of existing supplies and meet future demand growth. “If you look at the size of the oilfields discovered, they get smaller and smaller at a heartbreaking rate,” says Grantham. Recent data support the observation. According to IHS, nonshale discoveries have fallen sharply since 2010, and 2014 is “likely to mark” the first year since the 1950s that no conventional giant oilfields (those with more than 500 million barrels in reserves) were found.

But while Grantham projects that oil will surge higher in the medium term, he ultimately—perhaps 20 or 30 years down the road—sees advances in technology making oil virtually obsolete in transportation. “It’s a wonderfully complicated world, isn’t it?” he says.

As for Goldberg, he plans to stick with his process and let the market guide him. The same indicators that made him bearish on prices, he says, could signal that prices are destined to rise again. “I can see a scenario where oil prices get super-bullish again if companies overcut,” he says on a recent morning, sitting in his conference room on the 38th floor of an office tower at the bottom tip of Manhattan. But he’s not positioned for that yet.

Meanwhile, he’s staying away from big theories or bold predictions. “There’s no key to the puzzle,” says Goldberg, who earned a degree in economics at Yale and wrote his thesis on behavioral finance under the guidance of David Swensen, the manager of Yale’s endowment. “It’s not like anyone can sit there and figure it out. Like, ‘Hey, oil’s going to be at $50 in three years.’ Well, maybe. I mean, if demand doesn’t grow, and industry cost cuts are 20%, and we don’t have a war with Ukraine, and Saudi Arabia doesn’t change its oil policy, and Libya maybe comes back online, and Nigeria doesn’t blow up—yeah, we’ll probably be at $50. Good luck getting any one of those variables right.” To solve oil’s new math, Goldberg knows, it’s better not to look too far ahead.

This story is from the March 1, 2015 issue of Fortune.