On Saturday, Buffett fielded dozens of questions from the crowd of shareholders and journalists who packed CenturyLink Center for Berkshire Hathaway’s (BRKA) annual meeting. A number of the questions were on his proposal that the U.S. should institute a rule that would guarantee the richest Americans pay higher rates than average tax payers. At one point in the meeting, a shareholder was greeted with applause when he said that Buffett’s outspoken position on taxes was hurting the company’s stock. “My 84-year-old father won’t invest in Berkshire because of your stance on taxes,” said the shareholder.

MORE: Millionaires effective tax rates top most

Even Charlie Munger, Buffett’s long-time partner in running Berkshire Hathaway, seemed to somewhat disagree with Buffett on taxes, even though Munger says he is “100%” behind the Buffett Rule. Still, Munger says the focus should not just be on taxes, but how government spends its money as well. “We need more sacrifice and we need more sensible ways of spending money,” says Munger.

At a dinner on Friday night, money manager Mario Gabelli, who has attended Berkshire’s annual meetings for the past decade, told a crowd of attendees that he thought Buffett made a mistake by getting too political, though he seemed to be in the minority. “It’s BS,” said Gabelli. He asked the room to raise their hands if they had wished Buffett hadn’t proposed the tax rule. Only a few hands went up. “It’s the elephant in the room and we should talk about it,” said Gabelli. “It’s a negative on the stock.”

A feisty Buffett on Saturday responded that he has a right, like every citizen, to free speech and that he didn’t give that up when he became the CEO of a public company. “Sounds like your father should buy stock in Fox,” says Buffett. “When I and Charlie took this job we weren’t required to put our citizenship in a blind trust.”

MORE: What exactly is the Buffett Rule?

Buffett says it’s fair to disagree with him, but that he doesn’t think that should change anyone’s choice to buy or not buy Berkshire’s stock. He says he didn’t know the political stances of many of the CEOs of the companies that he and Berkshire invests in and didn’t care. Though presumably the reason Buffett doesn’t know their political stances is because those CEOs have chosen not to be as vocal about them as Buffett has recently.

Buffett said his proposal has been butchered by others, but he didn’t really say how. “It would affect very few people, and raise a significant amount of cash,” says Buffett.

Buffett said about a third of the 400 richest people in America pay a 15% tax rate or less. He said rates used to be much higher for the top earning Americans and that we should restore that. What’s more, Buffett said despite recently getting voted down in the Senate, he was optimistic that some form of the rule he has proposed would be adopted. “Not dead,” says Buffett, “People are bothered by inequality. Something will happen.”

Buffett said he doesn’t use offshore accounts or other tax strategies. Yet, every year, Buffett says, he compares his tax rate to others in his office, and he routinely pays the lowest rate. “They all are in the 30s, and I am in the range of 17%,” says Buffett. “It’s time we look at that.”

That comment got applause from the Buffett faithful as well.