Earlier this week, on a conference call with analysts following its second quarter earnings, JPMorgan Chase CEO Jamie Dimon said his bank was pulling back, and may ditch completely, FHA lending, a key government program meant to promote homeownership. The reason: all those government fines that JPMorgan and other banks have had to fork over recently.

“We collected $600 million on [FHA] insurance. They disputed $200 million. The government called that fraud. We reimbursed $600 million to get out of the lawsuit,” said Dimon in response to a question from an analyst. “So the real question to me is, should we be in the FHA business at all?”

Bankers have been saying for a while that increased regulation will cause them to cut back on lending and other services. Banks have already started to step away from the business of offering cheap money transfers for immigrants and others. Regulators have forced banks to put in place more controls to make sure those services are not being used by terrorists or drug traffickers to launder money.

“Dimon was expressing a widely held frustration about the crackdown,” says Burt Ely, a bank consultant. “If the government is going to get tough on the banks, some are going to say forget it.”

The FHA program offers government insurance to lenders for mortgage loans with as little as a 3.5% down payment from borrowers, and it opens the mortgage market up to consumers with lower than average credit scores. Since the financial crisis, the program has been key in getting money to first-time homeowners. In recent years, about a third of all mortgages taken out for home purchases have been backed by the FHA program.

“I think [Dimon] is sick and tired of writing the government checks,” says Paul Miller, the analyst who asked the question that spurred the comment about the FHA.

Consumer advocates say if the nation’s largest banks were to pull out of the FHA program, many Americans would suffer. “What you would see is a severe reduction of blue collar people and minorities able to get loans,” says John Taylor, who is the head of the National Community Reinvestment Coalition.

On Wednesday, at a lunch in Washington of current and former bank regulators, Tony West, the U.S. Associate Attorney General, who has overseen the recent multi-billion settlements with banks, was asked whether he thinks the fines will limit the credit that banks offer consumers. He said the government was looking into the issue.

But here’s the thing: While FHA lending is down overall, JPMorgan loans in the first quarter accounted for a larger percentage of the direct loans made through the government program compared to the same time period last year. And the bank plays an even bigger role in the secondary market for FHA loans, buying up and funding mortgages that others make.

In the secondary market, JPMorgan’s activity is down 64% in the first half of this year. But so were all the big banks, according to Inside Mortgage Finance. For example, Wells Fargo funded 59% fewer FHA loans in the first half of the year. Overall, the FHA lending market is down. And the amount of FHA loans that JPMorgan is funding is down more than its rivals.

But JPMorgan’s home loan business in general was down to a larger degree compared to its rivals. So the drop in FHA lending could simply be a part of that trend. And Dimon could just be using FHA and his frustration with the government as an excuse.

JPMorgan loans that have some sort of government guarantee—including loans sold to Fannie Mae or Freddie Mac—were down 73% for the first half of this year compared to the same period last year. That’s a bigger drop than the shift in the bank’s FHA business by itself.

Of course, JPMorgan could be looking to stop doing mortgage business with the government completely. But that would be tough. Right now, the government—through FHA, Fannie, and Freddie—is involved in about 80% of all mortgage lending. “You can’t be a serious mortgage market player and not be involved in government programs,” says Guy Cecala, who runs Inside Mortgage Finance. “And it’s hard to be a large national bank without a larger mortgage business.”

What’s more, banks have to meet Fair Lending and Community Reinvestment Act requirements. If Dimon does chose to go it alone, he will still have to make many of the same loans, but hold them on the bank’s balance sheet without a government backing, at least until the mortgage market opens up again. That could end up increasing the bank’s risk, something Dimon is saying he is trying to avoid.

“I think Jamie just got a little hot under the collar,” says NCRC’s Taylor. “I don’t think he really thought through what his statement means.”