For years, many tech workers in Silicon Valley have enjoyed free meals — one of several cushy perks offered the likes of Google, Facebook, and countless startups. But complimentary grub could become a thing of the past if the Internal Revenue Service has its way.

A report on Tuesday by the Wall Street Journal revealed the IRS is pushing to tax employees for their free meals. Companies would have to add in the value of free food when calculating employee tax withholding.

News of a potential tax on free meals has many worried in Silicon Valley, where all-you-can eat buffets are a basic recruiting tool. They’re also a subtle way to get employees to work longer hours by giving them no reason for them to leave the office except to sleep.

“Having food available or catered in is kind of expected of most tech firms, so this is a bit of a concern,” admits Steve Sarner, VP of Marketing, at the social networking site Tagged, where employees get at least one free meal a week cooked up by nearby restaurants in San Francisco’s Financial District.

Nathan Spady, a front-end engineer at Weebly, a service that lets users build web sites with custom software tools, called the idea of taxing free food awkward. The practice is a social catalyst that makes it easy for a company’s staff to talk to one another, he said. Weebly makes that easy enough by serving free catered lunch daily.

“You start doing that, well then, do you start taxing free coffee as well?” Spady said. “It just doesn’t make sense.”

Google, which popularized the free Silicon Valley buffet, did not immediately respond to a request for comment. Facebook, which followed in Google’s footsteps by offering its employees free food, declined to comment.

The idea of a food tax rankles Matt MacInnis, CEO of the digital publishing startup Inkling. His company has served its employees free lunches for nearly four years via the catering business Ryan Scott 2 Go. ZeroCater, another caterer, was hired last year to serve dinner. MacInnis argued that free meals in the tech industry are a lot like standard benefits such as employers matching 401k contributions by their workers.

Companies like Google GOOG , Facebook FB , and Apple AAPL can easily afford to pay employees more to cover any additional taxes for free meals, he said. Therefore, their recruiting wouldn’t suffer. But start-ups, which feel compelled to offer free laundry, yoga classes, and free food to compete for top talent, would be put at a big disadvantage, he said. In Inkling’s case, MacInnis says he would need to increase employee’s annual salaries by more than $5,000 to cover any extra food taxes.

“It seems there would be pretty terrible, unintended consequences,” MacInnis said.