Viacom’s plummet illustrates what can go wrong at a “controlled” company—a publicly traded firm where voting power is concentrated in the hands of founders, the founding family, or a handful of executives. Even more than other public companies, these businesses rise or fall on the whims of just a handful of people. And it sometimes seems that for every huge success story at a controlled-company—think Berkshire Hathaway (BRK-B), where Warren Buffett holds a decisive number of voting shares, or Facebook, under founder Mark Zuckerberg’s grip—there’s a Viacom-like cautionary tale.

Just over 100 of the companies in the S&P 1500, or around 7%, qualify as controlled, according to the nonprofit Investor Responsibility Research Center Institute (IRRCI). They fall into two categories: those where founders control 30% or more of the total voting shares, and those that use dual-class or multi-class share structures, in which only a few people own “super-voting” shares that far outweigh the common shares they offer to outsiders. Google (now Alphabet) famously went public with the latter structure, and later strengthened the voting rights of founders and chairman Eric Schmidt by issuing a nonvoting class C share. The three men now hold nearly 60% of the voting power.

For owners, running a controlled company means being insulated from pressure to satisfy other shareholders and from the threat of being ousted by activists seeking better returns. It also means being able to stock their boards with like-minded folks. (Perhaps not coincidentally, multi-share-class controlled companies pay their CEOs more than average.)

So how do their non-empowered shareholders fare? It’s a mixed bag. UBS says family-run midcap stocks worldwide returned 113% since 2005, compared with 64% for all midcaps. Belen Villalonga of New York University has found that controlled companies have higher-than-average profitability. Performance seems to weaken as companies get bigger, however. Over the past 10 years, controlled U.S. large-cap companies underperformed noncontrolled ones in shareholder returns and revenue growth, according to the IRRCI (see graphic).

In any case, a controlled structure puts more onus than usual on shareholders to figure out whether they can trust a company’s management and owners. A tale of two brewers shows the difference an owner can make. At Boston Beer Co., creator of Sam Adams, founder Jim Koch holds the vast majority of voting rights. The company rode high on the rise of craft brews and ciders. But the stock has fallen 44% in the past two years, and revenue has shrunk as Sam Adams came to be seen as corporate “big beer.” Koch, 67, makes some investors uneasy because he still holds most decision-making power and hasn’t announced a succession plan. “He’s one of those ‘not going to die’ type of guys,” says Cowen analyst Vivien Azer.

For a contrast, look at the multinational Molson Coors. The Molson and Coors families control 80% of the board. But the firm has let nonfamily CEOs manage day-to-day dealings, and the families “benefit as the business prospers,” says Azer. Since its 2016 buyout of SABMiller’s stake in U.S. brewer Miller-Coors, investors have been bullish: The stock is up 72% over the past three years, compared with 11% for the S&P Food and Beverage Select index.

When family members are poised to take over a controlled company, experience matters. Media giant 21st Century Fox has been gradually transferring power from Rupert Murdoch to his children. His sons have taken on significant roles at the company, gaining experience and testing their readiness for greater responsibility. Rupert remains co-executive chairman, but his son James, who became CEO in July 2015, understands the business well enough that there’s “potential strong leadership” in place, says Michael Morris, an analyst at Guggenheim Securities.

Sometimes longevity is the best sign that a controlling family lets good leaders lead. A.O. Smith, a maker of water heaters and treatment systems with about $2.6 billion in revenue, has been around since 1904. The Smith family retains 96% of the shares that select the majority of the board, but it has given management the flexibility it needs. A.O. Smith recognized China’s importance early: The country now accounts for 31% of sales and helped the firm ride out the U.S. housing crash. KeyBanc analyst Jeff Hammond expects 8% growth in 2017, which would outshine competitors. And its stock has returned almost 700% over the past 10 years—about 12 times as much as the S&P 500.

Of course, the risk remains that another Smith could decide the family should take over. Your shares wouldn’t stop that—but you could always vote with your feet.

A version of this article appears in the February 1, 2017 issue of Fortune with the headline “When Shareholders Are Spectators.”