Peter Orzsag, who was President Obama’s first Director of the Office of Management and Budget, spoke at the Atlantic’s Next Four Years forum, where he criticized what he saw as a transition effort that got started too late and is having trouble staffing key positions. “A well functioning transition ramps up substantially ahead of the actual election,” Orszag said.

He argued that it was clear that this didn’t happen in Trump’s case, and now they are paying the price, as they are behind in identifying candidates for key positions and beginning the background-check process for thousands of other positions in the executive branch.

“A transition might seem like it’s a nice pre-governing period,” Orzsag said. “But it’s actually among the most stressful and hardest periods in an administration.” He gave the example that one of his tasks as budget chief was to write a complete and coherent budget to present to Congress roughly a month after inauguration. This is even more important for Trump, as he will have a razor-thin margin in Senate, making it necessary to pass major legislation through a complicated budget-reconciliation process.

For all these reasons, Orszag sees the balance of power in Washington shifting towards Congress. If Trump is unable to staff up and start crafting legislative proposals, he will have to rely on the work a Republican Congress has been doing for years now. “For example, [the House has] passed the [Obamacare] repeal—the legislative language is there,” he said. “That means when there’s a divergence [between Trump’s priorities and Paul Ryan’s] they’ll go with the congressional priorities.”

Though stock markets have been excited about the prospects of new infrastructure spending, Orszag is not so optimistic. He said that he talked to Speaker Ryan about a month ago, asking his opinion on new infrastructure spending. The Speaker’s response? Orszag said Ryan wasn’t all that keen. “[Ryan] said, ‘We just passed the highway bill. I don’t know why anyone is talking about infrastructure’,” Orszag said.

Fortune reached out to Ryan’s office for a comment, but has not heard back.

Finally, he argued that the infrastructure that Trump is talking about, in which he’ll rely on public-private partnerships, won’t be able to address the nation’s most pressing needs. That’s because private investors will only sign onto projects that create definite revenue streams. “That won’t work for the vast majority of U.S. infrastructure,” Orszag said.