“It was like revenge of the nerds,” Carter, 43, told his wife, Rebecca, after leaving a 2010 barbecue at the Silicon Valley home of investor Joe Lonsdale. “The world was going into a very different place,” he recalls thinking. “And the power was shifting.”

At Lonsdale’s home, he met eight or nine founders, and was dumbstruck by how different their view of the world was. Carter left San Francisco with an unsettling feeling—he needed somehow to be part of this changing world of mobile phones apps, ride sharing, and more.

Carter’s path to tech moguldom has been a long and unlikely one. At the time of that fateful Lonsdale barbecue, Carter was the manager for Lady Gaga—the pop phenom filling stadiums worldwide. And before that, in the 1990s, he was an aspiring rapper. Living in Philadelphia, a 22-year-old Carter would wait outside actor and musician Will Smith’s recording studio for months, even in a feet of snow, to pitch Smith on his rap record.

Carter says that eventually, Smith took pity on him and made him an assistant for him and his business partner, James Lassiter. He then met Sean Combs and started working with Bad Boy Records as an assistant, where Carter says he lived through what he calls the “beginning of the dark ages for hip hop,” as artists like Tupac and Notorious BIG were both killed within months of each other. He was on the brink of bankruptcy when he came across the little known musician named Lady Gaga, who had just been dropped by her record label Def Jam in 2006. “She knew who she was as an artist,” Carter remembered of his first time meeting her. “My job was to help protect that vision.”

By the time Gaga fired Carter in 2013 she was already one of the most famous musicians of all time—and Carter was already enchanted with the tech world. The entire music industry had been transformed, and the key players were playing catch-up. “The music industry did a terrible job of building a relationship with consumers,” he says. “Spotify was one of the first services that actually focuses on the consumer because they don’t have to spend hundreds of dollars a year on music.”

So Carter joined forces with not just Spotify (where he’s currently head of creator services), but he invested in Uber, ride-sharing rival Lyft, eyeglasses upstart Warby Parker, and cloud file sharing giant Dropbox. Over several years, through a mix of “dumb luck and desperation” he managed to amass early stakes in several of the valley’s most successful startups. Fun fact: he is the only investor who has backed both Uber and Lyft.

Over the past five years Carter has backed more than 80 tech startups, and now works as general partner of his VC firm, Cross Culture Ventures. These days, when Carter isn’t at Spotify, he’s nurturing Cross Culture and Atom Factory, his talent management firm, which has become a family affair. His wife is the CFO of the agency, and one of his five kids has a hands-on role at Cross Culture. Mixing work and life is ok with Carter—the other kids are always in and out of the office.

After all, he says, it’s as much play as it is work. Says Carter: “The only two things that ever came naturally to me are music and investing.”

A version of this article appears in the October 1, 2016 issue of Fortune with the headline “From Musician to Mogul.”