FORTUNE — The scuttlebutt among some investors at the annual Morningstar investment conference this week in Chicago is that Bruce Berkowitz is toast. All well-known managers go through rough times, especially value investors who buy stocks others hate. But when you’re named fund manager of the decade by research giant Morningstar, as Berkowitz was last year, scrutiny can quickly reach fever pitch.
Barry Ritholtz sent the first shot in April when he “fired” Berkowitz from his recommended funds list. Indeed, Fairholme looks all wrong: its top five stocks include AIG (AIG), Bank of America (BAC), Goldman Sachs (GS), and Sears Holdings (SHLD). (See also Bruce Berkowitz, BofA’s flag-waving fan) Of 1,062 U.S. large-cap value mutual funds tracked by Morningstar, Fairholme ranked in the 99 percentile in year-to-date returns as No. 1,061. The fund is down 13% in 2011, compared to a 2.5% rise in the S&P 500. Investors’ withdrawals from Fairholme might be upwards of $200 million this year.
“St. Joe is a concern,” said one of the many Fairholme shareholders at the conference, of the Florida land developer that Berkowitz tried to take control over in a very public fight earlier this year. “I think a good investor needs to know the business,” this shareholder added, “but to be active and operating in the business seems to me is a separate thing from investing.”
Judging by the gossip around the conference’s cavernous 124,000-square-feet space, the drumbeat might get louder if Fairholme’s returns don’t turnaround.
“It’s too big, there aren’t enough analysts,” said a financial advisor who sold his Fairholme stake this spring. “I just think the money went to his head. He should have closed the fund.” Fairholme’s assets rose to more than $17 billion at the end of 2010 from around $13 billion a year prior.
Those investors not scared by a couple quarters of underperformance might be reassured that Berkowitz says he isn’t deterred by redemptions. It hasn’t forced him to sell entire positions in companies. In fact, it’s a scenario he envisioned when building a $4 billion-plus cash hoard late last year. “This has happened to me before,” Berkowitz told Fortune before he took Morningstar’s stage in front of 1,078 financial advisors Wednesday. “Remember, that’s why we had that much cash in the first place.”
Indeed, when you look at Fairholme’s returns in 2000 you see that the fund lost money the beginning of the year when tech stocks peaked. Fairholme trailed the S&P 500 by a wide-margin for almost four months. Eventually its insurers and other companies gained favor. Fairholme returned 46.5% in the year to beat the S&P 500 by 56 percentage points.
The difference this time? “It’s a whole lot more money,” admits Berkowitz.
Agree with him or not, Berkowitz is still loading up on financials. He told Fortune that Fairholme added more AIG shares when the government sold some of its stake for $29 a share. (For more back on his AIG stake, see How Bruce Berkowitz stumbled with AIG.)
Berkowitz’s sterling long-term — Fairholme’s returned nearly 10% a year for the past decade — may explain why some shareholders offered a compassionate view.
“We’re not selling but we’re not buying more either,” said one, who went on to express the sentiment of many of the people who spoke with Fortune: “It’s not doing what we want it to do.”
