While viral videos of robots performing parkour and backflips dominate social media feeds, industry insiders suggest these acrobatic feats are misleading indicators of progress. Industry executives at the Fortune Brainstorm AI conference, held in early December in San Francisco, argued that the true revolution in robotics is not physical agility, but the ability for robots to “think” for themselves—a capability that is finally bringing them closer to conquering the mundane, yet deceptively difficult, task of, say, opening a door or climbing a set of stairs.
For the past 70 years, robotics relied on a specific paradigm: intelligent humans pre-programming machines with complex mathematics to execute specific tasks. This approach is now obsolete, argued Sequoia Capital partner Stephanie Zhan and Skild AI CEO Deepak Pathak, in conversation with Fortune‘s Allie Garfinkle. The industry is undergoing a massive shift where robots, much like the Large Language Models (LLMs) behind tools like ChatGPT, are learning directly from data and experience rather than following rigid code.
“The change is things in robotics used to be driven more by human intelligence,” said Pathak, noting that the new wave is defined by models that can generalize and learn. “What has now changed is that these models or these robots can now can learn from data.”
In July 2024, Zhan wrote for Sequoia’s blog about Pathak’s deep credentials in the space and what distinguishes him as a robotics CEO: his computer vision and deep learning chops. In comparison, traditional robotics focused on collecting specific data to train robots for particular tasks. Pathak and his partner, Abhinav Gupta, leveraged large-scale data to build a foundation model. Hailing from a small town in India, Pathak made national headlines by gaining acceptance into the Indian Institute of Technology Kanpur without leaving his rural hometown, Zhan wrote. He learned to program by writing code by hand at home and used limited minutes at the local cafe to run his programs. He later pursued a Ph.D. in AI at Berkeley while joining Facebook AI Research, on the way to co-founding Skild.
Zhan and Pathak’s conversation with Garfinkle touched on a paradox in artificial intelligence known as Moravec’s paradox: what looks hard is often easy, and what looks easy is incredibly hard.
Why backflips are easier than doors
A robot doing a backflip essentially requires controlling its own body in free space, a physics problem that computers have been good at solving for decades. “It’s actually a lot easier to program a robot to do a backflip than it is to get them to climb stairs,” Garfinkle pointed out, to agreement from both of her panelists.
The real challenge—and the holy grail of “physical intelligence”—lies in interaction with the chaotic real world. Climbing stairs or picking up a glass requires a robot to continuously use vision to correct its movements in response to a changing environment. This “sensory motor common sense” is the root of human general intelligence, and it is the barrier that new “brain” software is attempting to break.
Investors and executives see this as a market opportunity comparable to the recent explosion in generative AI. Zhan noted that just as OpenAI unlocked the market for digital knowledge work, companies like Pathak’s Skild are aiming to unlock the market for all physical labor. The goal is to create “generally intelligent software” that can act as a brain for any robot hardware, reducing costs by an order of magnitude.
Unlike the software world, however, robotics faces a unique hurdle: a lack of data. While LLMs were trained on the entire internet, there is no equivalent database for robot physical interactions. Pathak argued that the company that deploys first will win by creating a “data flywheel,” in which field robots generate the data needed to make the system smarter.
For consumers wondering when a robot will be doing their laundry, the timeline remains staged. Pathak and Zhan predicted that robots will first proliferate in industrial settings and “semi-structured” environments like hotels and hospitals before entering the more chaotic environment of a private home.
Despite fears of job displacement, they argued that the technology is necessary to address the “Three S’s” of the future: Safety, Shortages, and Social evolution. Robots are poised to take over jobs that currently force humans to risk their lives or health. Furthermore, with millions of job openings currently unfilled due to labor shortages, robots could fill the gap in essential blue-collar work. Ultimately, the hope is for a social shift where dangerous or drudge work becomes optional, allowing humans to focus on tasks they enjoy.
