Buffett, the billionaire investor who runs Berkshire Hathaway , said Tuesday at Fortune’s Most Powerful Women Summit in Washington that the nation’s tax code “has gotten distorted to a huge extent,” by levying higher taxes on secretaries and janitors than on CEOs and private equity whiners.

He called, as he has in the past, for policymakers to redress that iniquity by raising taxes on the rich. Buffett said taxes will have to rise in general in coming years if we want to dig our way out of a giant budget deficit.

“We are not taking in enough money at the federal government level,” he said. He said tax collections (see chart, right) will have to rise back into the 18-20% range from below 15% lately.

But Buffett also added a new twist in an interview after his appearance with CNNMoney.com’s Poppy Harlow: He said it’s time to cut taxes on those outside the top tax brackets.

[cnnmoney-video vid=/video/news/2010/10/05/f_mpw_buffett_taxes.fortune/]

Buffett has been railing for years about the absurdity of the current tax structure – he remarked Tuesday with some disbelief that he pays the lowest rate in his Omaha office — and has accordingly called for higher taxes on those earning large wages, including himself.

But he hasn’t previously pushed for lower taxes on those making less. Asked why not, he said that no one ever asked.

A tax cut for low-income people could boost consumer spending at a time of high unemployment and fading government stimulus funds. But such a shift might not play well with the Tea Party types who have made an issue of the fact that 47% of U.S. households paid no federal income tax last year.

Regardless, Buffett says the tax numbers don’t add up and will have to rise. So why not raise taxes on those who can afford to pay?

“We’re going to need to get more money,” said Buffett. “Why not get it from me instead of the guys who will serve us lunch?”