America’s painful doctor shortage is threatening health care reform

March 4, 2014, 8:28 PM UTC

FORTUNE — The new federal budget, unveiled on Tue., March 4, trumpets a major shift in America’s policy toward one of the most vexing problems in reforming our health care system: fixing the doctor shortage. Since 1997, the number of physicians entering the workforce each year has essentially been capped, while the demand for everything from hip replacements to treatments for diabetes to angioplasties has soared with our growing and aging population. Now, the Obama administration proposes spending an additional $5.23 billion over the next decade to mint new physicians. That sounds like a lot of money, and the proposal sounds like a big deal, because it officially breaks the old bottleneck that limits the supply of manpower we desperately need. But it doesn’t go nearly far enough.

The proposal is designed to swell the ranks of primary care doctors. Those family physicians, general internists, and pediatricians constitute the workforce that, far more than any other type of physician, is charged with making Obamacare — and the general shift from fee-for-service to managing giant populations of patients at reduced cost — the new paradigm for health care delivery.

Unfortunately, the new policy is a timid response to a looming crisis. Based on current demographic trends alone, today’s dearth of primary care practitioners is destined to get far worse. As a result, prices are rising, and competition is falling, as big hospital groups rush to hire all the primary care doctors they can find. Once one or two big chains control most of the family doctors and general internists in their markets, rivals can’t find the troops to challenge them. That’s creating oligopolies in markets across the nation. “The game today is, ‘He who gets the most primary care doctors wins,’” says Travis Singleton, executive vice president of Merritt Hawkins, the nation’s largest physician recruiting firm, and a division of AMN Healthcare .

MORE: How one hospital is thriving in the age of Obamacare

Because the shortage is so deep and the investment so modest — a little more than $500 million a year — the new measure fails to reverse the policy that caused the problem in the first place. The flow of doctors entering the market each year is determined by the number of U.S. residency positions, chiefly in teaching hospitals. Those positions are funded primarily by the program that oversees Medicare and Medicaid. In 1997, the federal government essentially froze spending on residency slots, limiting the number to around 100,000 over three-to-four years, and in turn freezing the number of newly licensed physicians available for hire each year to around 26,000. Over the past 17 years, a few hospitals have established new residency programs for primary care doctors, raising the number to around 27,000, or a less than 4% increase. Meanwhile, the U.S. population has risen by 50 million, or almost 20%.

Of those new doctors, only around one-quarter, or 7,000 a year, enter primary care. The reason is obvious: Family doctors earn salaries averaging $185,000 a year, vs. $336,000 for general surgeons and $266,000 for OB/GYNs, according to Merritt Hawkins. But those relatively few family doctors, general internists, and pediatricians are in incredibly high demand. At Merritt Hawkins, the searches for family doctors far exceeds those for any other category. The American Association of Medical Colleges estimates that the U.S. will face a shortage of 46,000 primary care doctors by 2020, equivalent to one-quarter of everyone practicing in that category today. And even though the total number of doctors has risen slightly over the past decade, the total hours they work each week has actually declined, and keeps dropping, as more and more physicians collect paychecks from hospitals and large medical groups.

And while it’s well known that rural communities are in desperate need for more primary care practitioners, the problem is universal: Around 50% of Merritt Hawkins’ searches for PCs are in towns and cities with populations exceeding 100,000.

MORE: The hidden number key to Obamacare’s success

To fulfill the needs of the marketplace, the American Association of Medical Colleges has long advocated creating 12 new medical schools and increasing enrollment by 30% — a shift that would eventually create an additional 8,000 doctors a year. Once again, the cap on residency slots renders that plan impossible. Indeed, the current proposal falls far short of the AAMC recommendation.

The new budget would create a total of 1,300 new residency slots, all for primary care. It would also allocate almost $4 billion over 10 years to grow the National Health Service Corps, an organization that provides loan forgiveness and scholarships to doctors who work in underserved areas for three to five years after graduation. “Keep in mind that this proposal does not create 1,300 new doctors a year,” says Dr Richard Cooper, an expert on physician supply issues. “Those slots are divided over more than three years, so that’s about 400 slots a year.” Hence, the total addition to the doctor workforce will amount to about 400 new doctors annually, or an increase of 1.5%.

That figure might not seem so insignificant if physician assistants and nurse practitioners could fill the gap. But that won’t happen. All told, those two categories of advanced practitioners encompass around 200,000. The total is growing, but the number practicing primary care is actually declining. Once again, both the NPs and PAs are in extremely short supply; Merritt Hawkins searches for the two professions jumped 164% from 2011 to 2013.

MORE: My afternoon with Imelda Marcos

Above all, health care reform needs basic, primary care to succeed as planned. Even before the passage of the Affordable Care Act in 2010, the market was moving decisively towards a model called Population Health Management. Instead of relying on fee-for-service, those plans provide coverage for many thousands of patients in a community or city, and require that doctors and hospitals work together in teams to manage the patients’ overall care, eliminating unnecessary procedures, and ordering tests and referrals to specialists only when needed.

Obamacare hastened the trend by advocating so-called accountable care organizations (ACOs) — giant, integrated groups of physician practices and hospitals that meet a long list of requirements established by Medicare, including expertise in compiling electronic medical records, following clinical protocols for diabetes or cardiac patients, and most of all, assigning each patient a family doctor or internist to coordinate all care and handle referrals to specialists. The primary care practitioner is hence the essential “raw material” for the new medical industry. If oil or grain is in short supply, the food or energy markets will fail to function. It’s the same story in health care.

The ACO regulations require that each provider cover a population of at least 5,000 patients. The need for broad size and scope has led to a flood of mergers among hospitals. Last year, Merritt Hawkins performed 64% of its searches for doctors who work in hospitals, on a salary, compared to 11% in 2004. It does another 16% of its recruiting for large medical groups, where physicians also get a paycheck. “Hospitals are trying to buy every physician group they can, they want to hire 30 or 40 primary care doctors at a time,” says Singleton.

MORE: Why Republicans are giving in to get ahead

The wave of mergers is creating high levels of concentration in many markets, for both hospital and physician services. “Hospitals are acquiring physician practices to enhance their pricing power,” says Robert Town, a professor at the Wharton School. “It’s clear that if you hire most of few primary care doctors available, it enhances pricing power even more.” Kaiser Permanente in California, and Intermountain Healthcare in Salt Lake City are just two examples of providers with powerful networks and huge market shares. The recent merger of Baylor Health Care and physician group Scott & White could reduce the bargaining clout of companies and insurers in the Texas market.

In a recent survey, Merritt Hawkins finds that waiting time to see most practitioners hasn’t changed much recently. But it’s still substantial — an average of 18.5 days for a new patient to visit a doctor in 15 of America’s biggest cities. As the doctor shortage gets worse, so will those waiting times. Most of all, the shortage of primary care doctors poses a fundamental economic problem. When supply is short, and demand strong — and no one denies that Obamacare’s generous subsidies will make it far stronger — market prices rise. Increasingly, Medicare and Medicaid set reimbursement rates at rates far below the “market clearing” levels that ensure patients get full access to care. As a result, 55% of all doctors refuse Medicaid beneficiaries, up from 45% in 2009, and a full one-quarter won’t see Medicare patients.

The solution is far more funding for residency positions, and licensing of highly qualified doctors who’ve completed programs in other nations and want to work in the U.S. Hospitals that want to train more primary care doctors at their own or the future doctor’s expense should be encouraged to do so. The reigning concept contradicts itself: providing universal health care while limiting the essential personnel needed to provide it. Without fixing the doctor shortage, reforming the health care system is a fantasy.