Ascend to the pinnacle of Wall Street – the chairmanship of Goldman Sachs (GS) – and you’re pretty much destined to become a household name. Bob Rubin and Hank Paulson went on to be Treasury Secretaries. Jon Corzine became Governor of New Jersey. Lloyd Blankfein is the public’s current personification of all that is evil about Wall Street.

And then there’s Stephen Friedman, who was co-chair of the investment bank with Bob Rubin from 1990 to 1992 and sole chairman from 1992 to 1994. Sixteen years later, you could be forgiven for thinking that Friedman had settled into a comfortable retirement, his days of influence long past. But you would be wrong.

The surprise announcement this week that little-known hedge fund manager Todd Combs had been tapped to head Warren Buffett’s nearly $100 billion investment portfolio at Berkshire Hathaway (BRKA) was notable for a number of reasons, not the least of which was the fact that virtually no one (in the business press, at least) had ever even heard of Combs. But the man was not unknown to his peers: private equity outfit Stone Point Capital, of which Friedman is chairman, seeded Combs’ Castle Point Capital in 2005. (Both invest primarily in financial services companies. Stone Point via the private markets, Castle Point via the public.)

Forget the fact that the Buffett-Goldman connections now have another notable strand. Friedman has a lot more going for him than that. While he didn’t land the plum job of Treasury Secretary like Rubin and Paulson, he’s no stranger to Washington. He served as chairman of the President’s Intelligence Advisory Board and director of the National Economic Council under George W. Bush. (Rubin and Friedman ran Goldman together for two years, and couldn’t be farther apart politically. Wonder how often those two meet at The Four Seasons for cocktails?)

Friedman is also a trustee of The Council on Foreign Relations—the rich man’s foreign policy club—and still serves on the board of Goldman. He also served as chairman of the board of directors of the Federal Reserve Bank of New York.

All in all, Friedman has kept a low profile while still remaining front-and-center in the realms of power and influence on Wall Street and beyond. And it seems to have done his reputation well, relatively speaking. Bob Rubin took the big paycheck at Citigroup (C) for what he later argued was a job with no responsibilities, but his image took a blow when Citi imploded. Hank Paulson thought he was going to Washington to serve a nice sinecure post-Goldman, but the financial crisis nearly broke the man. And Jon Corzine, where to start?

Friedman’s only tarnish came during the financial crisis, when news broke that while serving simultaneously as chair of the New York Fed and a board member of Goldman Sachs, he was buying shares of Goldman in explicit violation of stated Fed policy. (For what it’s worth, he’d been granted a “waiver” from the Federal Reserve itself, as profound a  piece of evidence of the clubbiness of Wall Street and Washington as you could ask for.) He stepped down from the chairmanship as a result of the controversy.

Remarkably, Friedman was a Class “C” director—supposedly appointed by the Board of Governors to represent the public. It’s impossible to contemplate in just how many ways that is offensive to the spirit of the founding of the Fed itself. But that’s neither here nor there, really, as now he’s off the board.

But still behind the scenes, exercising influence.