About three out of five managers believe that remote work is on the wane and workers will be back in the office full-time by the end of this year. 

And they’re apparently prepared to use some tough tactics to get workers back at their desks. 

About 77% of managers said they’d be willing to implement “severe consequences”—including firing workers or cutting pay and benefits—on those who refuse to return to the office, according to a recent survey by employment background check company GoodHire of 3,500 American managers.

Although many surveys have shown that the majority of workers prefer remote and hybrid work structures, most managers still believe in-person work is best. Former Google CEO and chairman Eric Schmidt even recently weighed in about the return-to-work debate, saying that it’s important people be at the office and he’s happy the remote era seems to be ending. “I don’t know how you build great management [with remote work]. I honestly don’t,” he said. And about half of managers, 51%, genuinely believe that their workers want to return to the office. 

“Clearly, managers are struggling,” said Max Wesman, GoodHire’s chief operating officer. “Organizations that find a work arrangement that satisfies the majority of their workforce will benefit in the areas of recruitment, productivity, employee satisfaction, and retention.”

Workers may have more say in their job structure amid labor shortage

When companies announce return-to-office plans, workers don’t have a lot of choice but to comply. Unless a worker was specifically hired for a remote position, they are probably required to work out of the office at least part of the time. 

Some workers (particularly those with health issues including an immunocompromised system, chronic kidney disease, serious heart conditions, diabetes, and obesity who may be more vulnerable to COVID-19 infections) may be able to get remote work accommodations under the Americans with Disabilities Act (ADA) and state regulations, but that typically requires a formal review process.

It’s worth noting that only about 10% of employed Americans worked remotely in March because of COVID, according to data from the Bureau of Labor Statistics. Over half of workers, 50.9%, reported being already required to return in-person full-time, according to Workhuman’s April Human Workplace Index, a monthly survey of 1,000 full-time U.S. workers.

But just because a company wants to head back to the office doesn’t mean it always goes smoothly. Financial giant Goldman Sachs, for example, reopened its New York headquarters in February and mandated its 10,000 employees return. Only about half showed up on the first day. 

That’s because despite the threats of severe consequences, workers do have some leverage right now thanks to the ongoing worker shortage. The U.S. had 11.3 million open jobs available in February, and that number hasn’t really wavered in recent months. So rather than spend a lot of time, effort, and money replacing noncompliant employees right now, some employers are letting it ride. 

But with warnings of a recession ahead, those workers who take their chances now by ignoring return-to-the-office mandates may find themselves to be at the top of the layoff list down the line—if not sooner.