Ryan isn’t the first speaker, or leader, to be in trouble. House Majority Leader Eric Cantor lost his seat in a surprise primary loss in Virginia in 2014. House Speaker Tom Foley, a Democrat, lost his reelection bid in 1994. But before that, you have to go back to 1862 for another example of an unseated Speaker.

“Our Wisconsin sources, both GOP and independent, have told us Ryan should be fine. I think that’s right, but I also recall being told that Eric Cantor would win in a walk in June 2014,” says Larry Sabato, head of the University of Virginia’s Center for Politics. “With low primary turnouts you just never know.”

What Donald Trump’s Mike Pence Pick Means for the Election

Ryan, like Cantor and former Speaker John Boehner—whose abrupt resignation last year made Ryan an unwitting Speaker—is being buffeted by the same forces that are rupturing the Republican Party; the same forces that led to this week’s nomination of Trump, an anti-establishment outsider. There are no heroes in Republican politics any more, and those that aspire to the title are chained to rocks to have their liver eaten daily by an eagle, a House GOP aide somewhat bitterly quipped citing the Greek God’s punishment of Prometheus for giving man fire.

Thus far, Ryan has managed to straddle the revolution and the establishment in a strikingly unique fashion. His 2010 fiscal conservative bible, The Path to Prosperity, gives him street cred with Tea Partiers and his turn as Mitt Romney’s 2012 running mate and 2013 budget deal author with Democratic Sen. Patty Murray have burnished his establishment bonafides.

But the longer he’s a leader, the more prone he becomes to attacks from the anti-establishment crowd. Take his primary challenger: Paul Nehlen, a businessman and entrepreneur. Nehlen says Ryan’s a faker. “Paul Ryan says he’s for smaller government and he’s funded every big government idea that there is,” Nehlen says in an interview with TIME. “Not only has Paul Ryan not died on any hill for a position he believes in to make government smaller, he hasn’t even skinned his knee.”

House Republicans Demand Another Hillary Clinton Email

Nehlen brags that he will this week be rolling out the endorsement of two to 10 sitting members of Congress. If true, that would mark a small but significant rebellion against the Speaker from within his own ranks.

[fortune-gallery id=”1737535″]

Though he isn’t yet facing the outright rebellion Boehner once did, Ryan’s record as Speaker has been spotty. He has been working with committees to craft a conservative agenda for America, a substantive policy platform in response to what he sees as Trump’s relatively policy-free campaign. He views this as the foundation for the future of a party struggling to look beyond this week, let alone past November.

But he’s also failed, as a former Budget committee chair, to get consensus on a budget—the first time a GOP-controlled House has not passed a budget in recent history—and has let slip through some must-pass items, like funding to fight the Zika virus. (Ryan rejected a bipartisan Senate bill and Senate Democrats rejected his bill as full of “poison pills,” as Democratic Sen. Chuck Schumer called them, such as defunding Obamare and Planned Parenthood, leading to an unresolved stalemate.)

What Donald Trump’s Personality Says About a Potential Election

Is Ryan really in trouble? He’s in trouble enough to be running television ads. A recent poll had Ryan ahead 43% to Nehlen’s 32%, but that shows a race tightening—Ryan was ahead by 40 points or more in two previous polls. Wisconsin is one of the few states that bucked the Trump train, its powerful GOP establishment depriving Trump of a primary win. Ryan also spends an enormous amount of time in the district, returning home every weekend to spend time with his wife and three young children.

At the same time, Ryan’s initial refusal to support Trump and his not-so-thinly veiled criticism of his party’s likely nominee hasn’t helped him with GOP primary voters in Wisconsin’s 1st District, according to David Wasserman, who follows House races for the nonpartisan Cook Political Report. “My impression is that [Ryan] only would’ve been in danger had he continued to hold out on Trump,” Wasserman says. “He starts off with a much deeper base in the district that Cantor ever did.”

They say in politics that where you sit is more important than where you stand. And the more Ryan sits atop anything—the House, the convention—the more of a target he becomes. He’ll likely beat Nehlen, but as the Wall Street Journal warns, Nehlen is only the first of many who would see Ryan ‚and what he stands for in the GOP—fall.