The fall and rise of VR: The struggle to make virtual reality get real
Paul McCartney made his modest contribution to the future of virtual reality with a little help from a bike mechanic.
The unlikely union of the Beatles great, a bike-shop employee in Palo Alto, and a promising if underachieving technology is the accomplishment of Scott Broock, once an enterprising executive with a fledgling camera company called Jaunt VR. In 2014, Broock offered to pay the mechanic $50 to ride around a skate park on a BMX bike while being filmed with a specialized camera rig that could shoot video and record sound in 360 degrees—all around and up and down. Broock hoped the bike’s chain clanging around the fishbowl would be ideal for something called ambisonic audio, surround sound hearable above, below, and around the listener.
A few months later, Broock managed to show a clip of the video to McCartney, who was so impressed that he invited Jaunt to film his concert the very next night at San Francisco’s historic Candlestick Park, the same venue where the Fab Four had performed their final show 48 years earlier. The startup company quickly mobilized and recorded one of the first videos of its kind, an immersive stadium concert film that would give a viewer the sensation of being among the pulsating crowd. Broock left Jaunt in 2016 and subsequently served a yearlong stint as a “global VR evangelist” for YouTube. But he still looks back at the concert video as a breakthrough achievement. “There’s a moment recorded in time of Paul McCartney playing in front of people captured in a way that, maybe 100 years from now, seems like black-and-white films”—primitive but pioneering. “That’s a powerful thing.”
The vintage film comparison—think: grainy footage of silent passersby shuffling around in top hats among horse-drawn carriages and Model T–esque cars—is standard fare for virtual reality’s boosters. Just as movies showed viewers places they’d never go, VR would transport them directly into those same filmed environments. That was the promise that led Facebook to pay $3 billion for headset maker Oculus VR in 2014, and every year since, evangelists have proclaimed virtual reality the next new thing. Consumer tech players including Google, HTC, Samsung, and Sony joined Facebook in a race to bring consumer-ready headsets to market. Venture capitalists poured billions into content development and hardware applications. Time magazine put the then-22-year-old founder of Oculus, Palmer Luckey, on its cover and announced the technology was “about to change the world.” Mark Zuckerberg in 2017 famously said he wanted a billion people to be using Oculus headsets—though he conspicuously didn’t say by when.
That omission is understandable. Because for all the hype-filled promises, virtual reality remains, well, virtually absent from everyday American life. Oculus in 2018, for example, shipped just 354,000 units of its flagship VR headset, the Oculus Rift, according to estimates from SuperData, a gaming-focused research unit of Nielsen. Contrast that with the more than 17 million PlayStation 4 game consoles Sony moved in the same period or global smartphone sales that year of 1.4 billion, according to IDC. Consumers are finding that VR is typically too expensive, too clunky, or too uncomfortable, and lacking in content that is worth trying more than once or twice. Skeptics compare the experience to the short-lived 3D-TV fad of the early 2010s.
The sluggish adoption has claimed multiple victims. Cinema operator IMAX, which used $50 million in venture capital funding to open virtual reality arcades in cities from New York to Bangkok, shuttered all the locations after just two years. Google’s in-house VR film studio, Spotlight Stories, folded earlier this year. And CCP Games, a popular Icelandic video game developer, laid off 100 people and ceased its development of new VR projects in 2017. “We saw in our own data that this is gonna take a while to get to the place it needs to be,” says CEO Hilmar Veigar Pétursson, adding that the wait will be “years, not months.” Even Jaunt, despite the boost from McCartney and more than $100 million of funding, including from Disney, couldn’t make a go of VR. Last year it shifted its attention to a related technology, augmented reality, which adds visual cues to real-life settings rather than trying to immerse users in distinct worlds. “We were focused on driving consumer adoption and understanding what consumers want to watch in VR,” says CEO Mitzi Reaugh, who oversaw a mass layoff at the Silicon Valley company. “It just wasn’t moving on the timeline that made sense for our company.”
1987: the year VR pioneer Jaron Lanier is said to have coined the term “virtual reality”
It is tempting to write off virtual reality as yet another overhyped fad. Yet that would ignore the technology industry’s long history of fallen pioneers paving the way for someone else’s breakthroughs. The Apple Newton and the Polaroid Polavision died, after all, so that the iPad and camcorder might live. It took a decade for smartphones to become ubiquitous. Early VR headsets themselves date back to the 1960s, while Nintendo and Sega in the 1990s forayed into the consumer market with the ill-fated Virtual Boy and Sega VR systems, respectively. And even if VR has been a disappointment for the entertainment industry—the McCartney VR concert video will never go platinum—the technology is proving useful in sensible business applications, like workforce training, and yet new entertainment concepts. After all, when a technology is so exceedingly cool that it attracts a legion of true believers, it is extremely difficult to kill.
Inside a building on Facebook’s sprawling Menlo Park, Calif., campus, past a literal Facebook wall scribbled with employees’ handwriting and motivational quotes like “If you never try, you’ll never know,” a spacious gray room is set up to demonstrate the highly anticipated Oculus Quest. This is the device VR enthusiasts believe can change everything. Released in May, the Quest is Oculus’s first all-in-one headset built for high-powered gaming. It requires no wires or connection to a PC and can operate with a full six degrees of freedom that allows users to look around and walk in all directions, unlike last year’s similarly wireless but less immersive Oculus Go. At a starting price of $399, it’s on par with mainstream consoles like Sony’s PS4 and Microsoft’s Xbox One.
Being placed into a VR device by another person is an awkward experience. Once the headset snugly fits over your face, the person who was just assisting you could be giving you the middle finger for all you know because you are now staring at, yes, another reality. In my case, it’s a very satisfying one, in which my Oculus Home, or the home screen, looks as if it were designed by Frank Lloyd Wright, complete with a maple wood interior and a domed glass roof peering up at the Northern Lights.
But the Quest is not about architecture; it’s about games. More than 50 of them launched with the device, none more popular than the colorful rhythmic sensation Beat Saber, developed and published by indie Czech studio Beat Games. Best described as Dance Dance Revolution meets Star Wars, Beat Saber in March became the first VR game to claim to surpass 1 million copies sold, and it shows no signs of slowing down. That’s thanks to an active fan community on YouTube, generating millions of hits from videos showcasing standout players. In April, it was featured on a Tonight Show segment with the host Jimmy Fallon and actress Brie Larson each playing the game on national television. VR enthusiasts nearly hyperventilated in their praise. “This is huge!” tweeted popular VR YouTuber Nathaniël “Nathie” de Jong. “True killer marketing for the entire VR industry.”
After 15 minutes of playing the game, I am sweating. You’re “exercising without knowing you are,” says Beat Games CEO Jaroslav Beck. “You are feeling the music in the most powerful way because you are physically experiencing it.” People across the industry, from developers to investors to company executives, say that this is, right now, the closest thing VR has to a “killer app”—a piece of content so good that it’s possible consumers will buy VR headsets just to play the game. It’s exactly the kind of outcome Facebook hoped for when it started Oculus Studios, a division that gives funding and technical advice to third-party game developers like Beat Games.
$34.5 billion: estimated worldwide VR total market forecast by 2023 (Source: Greenlight Insights)
Facebook’s initial vision for VR was far grander than games. It thought cinematic virtual reality would be a breakthrough application and that Facebook itself, rather than third-party developers, would create the masterpieces. Facebook established the Oculus Story Studio in 2015 as an in-house film department dedicated to making movies for virtual reality. Yet despite winning an Emmy for its animated short “Henry,” Facebook shuttered the studio in 2017. Yelena Rachitsky, a Facebook executive producer who’d been with the defunct studio, says Facebook realized its clout was better deployed encouraging an ecosystem approach. “I think there is just a reality that a lot of the creativity doesn’t necessarily happen within a big corporation,” she explains. “It’s the creators out there who aren’t limited or confined by specific corporate structures [who] I think have the innovative and creative thoughts that are going to continue to push the boundaries in VR.”
Hollywood also figured prominently in Facebook’s VR dreams. Edward Saatchi, whose father, Maurice, cofounded the ad agency Saatchi & Saatchi, was a founding member of the Oculus Story Studio. He says the goal was to create VR content that could “inspire an industry.” Five or so years ago, Hollywood directors approached then Oculus CEO Brendan Iribe, intrigued by the technology’s prospects, says Saatchi, who now heads a “virtual beings” company called Fable. “They were super excited and said, ‘Let’s make a VR movie.’ But he was like, ‘I have no idea how to do that.’ ” The Story Studio was Oculus’s attempt to find out how. “Our goal was to get film schools teaching VR movies, to have film festivals accepting VR movies, to have famous directors do VR movies,” Saatchi explains, noting that director Alejandro González Iñárritu, whose Birdman won an Academy Award for Best Picture in 2014, took home another Oscar for his 2017 VR short, Carne y Arena. “So, in that sense, we succeeded. Except it didn’t become a mainstream thing. There just isn’t any evidence that anyone is willing to pay for narrative VR content outside of a theme park.”
In retrospect, Mark Zuckerberg was so enamored with the theoretical potential of VR that it appears he spent billions without having thought through how to make a business of it. “It was a platform play,” says Blake Harris, author of the optimistically titled The History of the Future: Oculus, Facebook, and the Revolution That Swept Virtual Reality. “He had a popular app. But in his mind there’s always going to be this problem of living on other people’s platforms. You’re beholden to Microsoft, Google, and Apple.”
Indeed, Zuckerberg and his minions have described VR as the logical next step in the social experience Facebook itself created for billions of people. Just as it digitized the analog behavior of keeping up with one’s friends, now Facebook wants people inside a virtual reality to “span geographical boundaries,” as Facebook director of VR product management Sean Liu says. “We’re really thinking and pushing the notion of how we bring you and your avatar into VR. How do we allow you to emote and have social expression to really connect together and do different activities?”
In the reality we live in today, VR isn’t a prevalent tool of communication. But that hasn’t dampened Facebook’s enthusiasm for it. “I don’t know exactly when it’s going to be a big deal,” Zuckerberg said in a call with investors last year. “When we started talking about this, I said that I thought that this is going to be a 10-year journey before this was really a very mainstream and major platform.” Just about halfway down the road, the futuristic technology is nowhere near realizing Zuckerberg’s vision.
The masked robber points a gun in my face and shuffles me and a sobbing woman into a back room. “Take this fucking bag, pick it up, and fill it up!” he screams. “Everything!” Now he’s motioning toward a white wall lined with packaged phones and accessories. Before we can react, a flash, a whirring noise, and then time cuts forward. The woman is now stuffing electronics into the bag, panicked. “Hurry the fuck up!” the robber’s accomplice shouts. “Let’s go!” And then black.
As I remove my Oculus Go headset, all is bright and peaceful in an empty classroom inside an unassuming office building in Manhattan’s Flatiron District. Jeremy Bailenson, a Stanford professor and founding director of the university’s Virtual Human Interaction Lab, stands beside me. He begins explaining what I have just witnessed: a VR training module for Verizon store employees to learn how to deal with armed robberies. “If you work at a Verizon store, there’s so much expensive material that’s right near the door,” he says. “They have dozens of robberies at gunpoint each year. They want to train their employees to be safe.” Verizon had been offering traditional training procedures for years, utilizing classroom instruction and hiring actors to simulate robberies. But the company found it minimally effective. “Despite having been trained, [our employees] weren’t necessarily equipped to manage through the robbery,” says Lou Tedrick, Verizon’s vice president of global learning. “We thought VR would be a good use case because it would help the muscle memory of what it had felt like to be robbed. You want to be able to feel it in a safe environment and be able to talk about it.”
To improve its safety training, Verizon approached Strivr, a VR software training company Bailenson cofounded in 2015. Impressed by the startup’s work with other large corporate partners like Walmart, Verizon tasked Strivr with developing modules to train store managers in high-fidelity heist scenarios. Since late 2018, roughly 1,500 of these managers have undergone Strivr’s training experiences. When surveyed, 95% said they better understood the factors they would need to consider during an actual burglary attempt. Asked about the ethical concerns of purposefully traumatizing employees, Tedrick says that professional trainers walk employees through every step of the way. “In fact, we had many people thank us for creating an incredibly realistic experience versus trying to sanitize the experience,” she adds. Verizon now plans to have store managers at all its retail locations trained in these VR simulations.
It turns out that while VR movies or virtual hangouts may not be ready for prime time, the technology is ideal for certain practical applications. VR is gaining traction in fields like surgical training, STEM education, industrial design, architecture, real estate, and more. At Facebook’s F8 developer conference in April, Oculus announced an expanded Oculus for Business program slated to begin in the fall. It includes access to enterprise-grade headsets, such as the new Oculus Quest, and “a dedicated software suite offering device setup and management tools, enterprise-grade service and support, and a new user experience customized for business use cases.” Microsoft and HTC, meanwhile, have pushed heavily into industrial enterprise with the mixed-reality HoloLens headset and the HTC Vive, respectively. “Our bigger market is on the consumer side,” says HTC’s Dan O’Brien, general manager of the Americas for the Vive product line. “But our more aggressive growth area is enterprise.”
“I think there is just a reality that a lot of the creativity doesn’t happen in a big corporation.” – Yelena Rachitsky, Oculus executive producer
Strivr, the Verizon vendor, is solely focused on business-to-business VR applications. In addition to the giant phone company, it counts Chipotle, Jet Blue, Fidelity Investments, and Tyson Foods as clients. It has distributed 17,000 Oculus Go headsets embedded with Strivr’s software in Walmart superstores and smaller stores across the country, all for internal use. From there, the startup says it can provide analytics that track performance and eye movement. “When a company tells us, ‘I need to know that the trainee looked at that bucket on the floor,’ we can tell you that they did not look at it,” says Strivr CEO and cofounder Derek Belch, a former graduate student of Bailenson’s. “That means they’re not going to look at it in the real world. Like, unequivocally.”
It isn’t unusual for business technology applications to find commercial success before their consumer versions do. Belch of Strivr says he doesn’t own a headset at home. One of Strivr’s backers, Zaw Thet of Signia Venture Partners in Menlo Park, Calif., is clearly pleased with his firm’s bet on an enterprise application. “There isn’t a killer app here on the consumer side,” he says. “Yeah, in 10 minutes you can get scared in a zombie house, and my 4-year-old likes to go look at the solar system for five minutes. But it’s not something he’s in every day.”
Why don’t more people use virtual reality—besides the issues of price, discomfort, and lack of good content? Because VR requires you to completely abandon reality. And, honestly, who has time for that? “You’re inside of a walled garden, you’re inside of a headset where you don’t have access to the real world,” says Jacob Mullins, a partner at Shasta Ventures, an early backer of the technology.
In fact, where VR has found limited success is by tweaking its approach, especially with augmented reality. AR shares similar properties with VR, but rather than completely immersing a viewer in another reality, it adds digital elements to the real world, typically through a smartphone. Think of Pokémon Go or the Ikea app that enables users to place and visualize new furniture within their homes. As confidence in virtual reality falters, AR is now experiencing levels of hype similar to the VR wave of five years ago, with startups like Magic Leap raising close to $2.5 billion to develop AR glasses and related content. Even Facebook is hedging its bets. Earlier this year, it moved hundreds of employees from its Facebook Reality Labs research division to a team dedicated to AR hardware projects. “In the future, our AR glasses will merge the physical and digital worlds, blending what’s real with what’s possible, resulting in the next mainstream, must-have, wearable consumer technology,” promises a Facebook Research web page.
The thought for some is that perhaps it’s more compelling to enhance our world than to replace it or create a new one. While stand-alone consumer AR glasses are still a ways away because the technology is less developed than VR, AR is already widely available on smartphones, thanks to Apple’s release of a set of software development tools enabling easy-to-use applications. The tech has proved popular with retailers, for example, including Target, Walmart, and Bed Bath & Beyond, each of which has incorporated AR features into its iPhone app to help shoppers visualize purchases.
The pivot from VR to AR is particularly noticeable in venture capital trends. “I’m equally interested in both, but as an investor, I’m forced to have to pay attention to where the customer and market opportunity and demand is,” Mullins says. “Two years ago, VR appeared to have more excitement and scale behind it. But then Apple essentially enabled 300 million–plus devices and growing.” Indeed, in 2016, VR’s peak venture year, VCs pumped $857 million into VR startups, according to SuperData; AR and MR, or mixed reality—which allows virtual imagery to actually interact with the real world—received just $455 million combined. But in 2018, the equation had flipped: VR funding was down to $280 million, while AR/MR jumped to $859 million.
354,000: number of Oculus Rift headsets shipped in 2018 (Source: SuperData, a Nielsen Company)
Another burgeoning approach has found its way back to the original promise of VR: entertainment. This compelling commercial application is called “location-based entertainment,” or LBE. A crop of companies are operating what is essentially a cross between an arcade and a movie theater, with a dash of theme park. These are brick-and-mortar venues where participants use virtual reality in custom-designed spaces, freely moving alongside a small group of fellow participants who appear to each other as avatars when wearing VR headsets manufactured by Oculus, HTC, and others. Some of these experiences play out more like games, with participants wielding plastic-model guns. Others are more like a narrative film that viewers interact with. LBE experiences offer another advantage over headset-bound, individual VR uses. They further immerse users by having them strap on haptic equipment that vibrates. Some venues even feature fans, sprinklers, and heaters to simulate conditions such as wind, water, or heat.
Dreamscape Immersive is a Los Angeles–based LBE “exhibitor” that’s raised $36 million from the likes of 21st Century Fox, Warner Bros., and AMC. It hopes to entice customers with immersive narratives, a kind of interactive moviegoing experience, says Hollywood veteran Walter Parkes, a Dreamscape cochairman. Parkes says he finds LBE more compelling than typical in-home VR—in other words, a single user wearing a headset—because users are an “actual character in a real, rendered world with other people able to be in touch with all of [their] senses.”
“Location-based VR is compelling: You’re a character in a rendered world, in touch with all of your senses.” – Walter Parkes, Dreamscape cochairman
The hope among VR adherents is that concepts like LBE will act as a gateway to overall VR (and AR and MR) adoption, in the same way cinemas begot additional ways to watch movies. Dreamscape charges $20 for its experiences, not too far off the average price of a movie ticket, though its run times are much shorter, at around 20 minutes. The Void, the most expansive of a burgeoning collection of LBE companies, with 11 locations in four countries including the U.S. and Canada, charges about $35 for its 30-minute Secrets of the Empire experience, in which you get to infiltrate an Imperial base and shoot Stormtroopers on a molten-lava planet. (The firm has rights to Star Wars and other blockbuster Disney intellectual property.) Businesses like The Void also move VR forward because they make it possible for consumers to experience the technology without spending serious money. “It takes down that investment barrier to entry,” says Tuong Nguyen, an analyst at Gartner research.
I have tried many virtual reality products by now. Oculus’s and HTC’s and Google’s and films and video games and job-training simulations. Someone was always there to strap me into the headsets, to prepare me for the experience. And then they were always gone when it started. And I was always alone, even if I saw other people, or things, inside the new reality, even if I could still hear people outside in the old reality.
When I enter the Alien Zoo at Dreamscape, inside a Westfield mall in Los Angeles, I’m thinking about how people typically describe their VR experiences. They fly over the Manhattan skyline or dive into the Pacific or head for outer space. And they always use the word “I.” I, too, am now in space. But there’s a significant difference: It’s not “I”, it’s “we.” Moments ago, my partners and I strapped blue-lit haptic sensors around our hands and feet and slung computer-stuffed backpacks around our shoulders. We stepped into a dark, bare room; slid the headsets over our faces; and watched one another’s bodies transform into human avatars. Our Dreamscape minder instructed us to shake hands to confirm this astonishing mix of the physical and fake, and then we set off for a safari on a vibrant planet occupied by brontosaurus-giraffes and gigantic praying mantises that make Jurassic Park seem positively Neanderthal. It’s a mind-blowing experience—and absolutely worth paying for. Now all virtual reality needs to do is to persuade hundreds of millions of people to arrive at the same conclusion.
A version of this article appears in the July 2019 issue of Fortune with the headline “The Fall and Rise of VR.”
Editor’s note: This story has been revised to more clearly reflect the status of CCP Games’ VR operations.
More must-read stories from Fortune:
—How cord cutting is driving big changes across the media landscape
—Hands on with Google’s Stadia video game-streaming service
—Nintendo’s president on streaming, 5G wireless, and e-sports
—Netflix takes baby steps into video gaming
—Game Stop wants to be the “local church” of gaming
Catch up with Data Sheet, Fortune‘s daily digest on the business of tech.