Mutual fund managers could be tomorrow’s buyout kings

February 20, 2013, 11:55 PM UTC

FORTUNE — U.S. mutual fund managers have recently been on a tear as the stock market has climbed to 5-year highs. Higher equity prices mean fatter portfolios and more fees for the asset managers. But dig down deeper, and there are some major cracks in the U.S. mutual fund business. T. Rowe Price’s protest against Dell’s buyout is an illustrative example of how they are fighting to stay relevant.

Revenues for the mutual fund industry overall are roughly what they were in 2007, according to Goldman Sachs. Between the mid-1990s and 2007 they had tripled. In 2000, equity mutual funds accounted for 80% of the fund business. Today it is 58%, as investors have moved money into instruments like exchange-traded funds (ETFs).

What’s most worrisome is that asset values (and therefore assets under management and fees) are rising at companies like T. Rowe, but that doesn’t mean investors are sticking around. T. Rowe had $4.2 billion in net outflows during the fourth quarter last year, despite the fact that assets under management rose 18%.

“There are big secular changes going on,” says Marc Irizarry, research analyst at Goldman Sachs. “Some of it is structurally related to demographics, and risk tolerance is changing as well. Then there are permanent changes in the product landscape.”

MORE: Hey Dell, what the hell?

ETFs are a big part of the problem. They aren’t actively managed, so they cost less and yet they still produce decent returns. Individual investors have caught on — over the last five years, more than $144 billion has flowed into passive funds like ETFs, while $460 billion has been moved out of actively managed mutual funds, according to research from Marquette Associates.

But alternative asset managers, like private equity firms, are also putting mutual fund managers in a tough position. Consider Dell (DELL). T. Rowe (TROW) owns 4.7% of the shares outstanding, according to FactSet Research. The company won’t say exactly when it bought its shares. But a public investor would have had to take a stake before February 1998 to break even on the stock’s current price.

Private equity investors are swooping in after 15 years of a slog, and with the backing of Dell’s founder. If anyone is going to make money on the company, the buyout group will. T. Rowe has joined a group of institutional investors opposing the buyout, claiming the $24.4 billion offer is too low.

Being squeezed from all sides, what is a mutual fund manager to do? It is bound to pick some losers sometimes. But with competition so stiff, it can’t afford to. Mutual fund managers need to offer investors something more than just decent returns on stocks. Perhaps it should give the average retiree a shot at the buyout business too.

MORE: Private equity for the average Joe? It’s coming

“Traditional asset management either has to pivot their business model to a new reality of the structure of portfolios, or they can sit around and hope that things go back to how they were,” Irizarry says.

There are several ways T. Rowe could break into buyouts. It could try to nudge its way into existing deals, like Dell, by putting up a fight when a deal is announced. There is some precedent for that. Buyout groups opened up their consortiums to shareholders in buyouts of Clear Channel Communications, Michaels Stores, and Harman International, among others.

In these cases, some shareholders protested a buyout offer because they thought the company was selling on the cheap. In order to get them to agree to the deal, the private equity team offered these public shareholders something called “stub equity.” (So the buyout group carved out a spot for public shareholders to come on in the buyout alongside them.)

T. Rowe may not be protesting the Dell deal with intentions of joining the buyout. It declined to comment. But generally, that would only be a temporary way into private equity anyway, not to mention that these one-off deals have challenges. When KKR (KKR) and Goldman Sachs (GS) walked away from their $120 a share offer for Harman, public shareholders were left holding the bag. The stock is now just above $44 a share.

MORE: Why a record month for fund inflows makes me cautious

T. Rowe — or any other mutual fund manager — might be better off considering a merger with a private equity firm, like Blackstone (BX) or KKR. They have something to offer buyout barons. Private equity firms are struggling to meet their historically large returns. Many would be eager to access new pools of cash, like the large groups of retail investors that mutual fund managers already attract.

Meanwhile, it could also give that retiree something he can’t get anywhere else: an exclusive chance to partner with the founder of a company in a buyout. As Dell may show, that would be something worth sticking around for.