The Race to Make Virtual Reality an Actual (Business) Reality

Big corporations are investing billions—but the technology may not play out the way they expect.
April 27, 2016, 10:30 AM UTC
Photographs by Bob Croslin for Fortune Magazine

The Race to Make Virtual Reality an Actual (Business) Reality

Big corporations are investing billions—but the technology may not play out the way they expect.

In mid-April, virtual reality passed a milestone. Using an app created by a company called Medical Realities, viewers from around the globe, each presumably with an iron constitution, witnessed a surgeon at the Royal London NHS Hospital delve into the bowels of a 70-year-old cancer patient. If that sounds like a Nova special, it was so much more. Doctors, medical students, and the merely curious experienced the operation in real time, with 360-degree control over their vantage points, as though they were actually in the operating room.

By now you’ve probably heard the hype about the coming wave of virtual reality (VR) and the miracles that it can render in videogames, entertainment, and exploration. Interested in flying a Black Hawk helicopter over the Dubai skyline just after dawn? It can put you there. How about walking in Neil Armstrong’s footsteps on the moon, or on the summit of Mount Everest? Done and done. But the live stream was a showcase of the wondrous potential of VR to educate, to train, and, more generally, to transport the masses into situations that would otherwise be impossible to experience.

It was a mere glimpse of what Medical Realities has in mind. Imagine, for example, enabling students to actually feel the resistance when a scalpel cuts muscle. For now, workplace applications for VR tend to be rather prosaic—think virtual conferencing—and much of the effort to create compelling content is focused on games and entertainment. But it provides a striking vision of the potential reach and practicality of VR in the future.

Tech powerhouses—Google (GOOGL), Samsung, Microsoft (MSFT), Sony (SNE), Apple (AAPL), Intel (INTC), and others—are creating VR hardware and software to establish a platform that they all say will rival the appeal and utility of the mobile phone or even the Internet itself. And a second, potentially much broader echelon of companies is trying to get out in front of the trend, dreaming up VR applications to change how we shop, advertise, learn, and communicate.

By some estimates, the market for VR will be worth $150 billion within four years. A decade from now, who knows? It’s impossible to say because no one’s clear on what the platform is best suited to do. But the race is on to figure it out. Never mind that some of the largest players have tried to put things on your face before: 3-D spectacles, Google Glass, and previous incarnations of VR headsets. Those were inferior technologies, they say. This time is different. Try it for yourself and you’ll understand that those technologies died so this one could live.

Programmer Tina Roh works in the Stanford Virtual Human Interaction Lab.Photo: Linda A. Cicero—Stanford News

Despite feverish press coverage of VR, two-thirds of the U.S. population remains unaware of the technology, according to a survey by research firm Horizon Media. That suggests a definition is in order. There are a few flavors of the technology: virtual reality, augmented reality, and a combination known as mixed reality. They all use a special headset to alter human perception and, in effect, take you somewhere or enable you to see something that would otherwise be invisible. We’re lumping all the technologies under the term VR in this article, both for simplicity’s sake and because all the permutations share the same roots.

Remember View-Master, the toy that debuted 77 years ago and delighted children by transporting them from their bedrooms to the Taj Mahal or the Great Smoky Mountains? Consider it an ancestor. The flight simulators of the 1980s were early incarnations. Then there were the headsets developed and abandoned by Sega and Nintendo in the ’90s. At that point the Internet captured everybody’s attention, and VR was largely forgotten.

That all changed when Facebook (FB) paid $2 billion to buy the current VR poster child, Oculus, in 2014, from a 21-year-old entrepreneur named Palmer Luckey. Oculus didn’t have a commercially available product at the time. But its demo was powerful enough to convince Mark Zuckerberg of VR’s far-reaching utility.

The wisdom of his bet will be tested in the coming months. The company’s first headset, Oculus Rift, priced at $599, or roughly $1,500 with a PC powerful enough to run it, began shipping in March. The Rift provides an experience unlike anything that has ever been available to consumers, thanks to remarkable advancements in screen resolution, positional tracking and movement sensors, haptic technologies (the processes that make phones and videogame controllers vibrate), computational horsepower, 3-D graphics, and audio.

Oculus executives are careful not to promise immediate mass-market success. For starters, the very mention of VR may cause ennui in some quarters and trigger memories of past promises unfulfilled. And companies will need to overcome people’s aversion to wearing a snorkel-style mask. So executives have talking points to address those issues.

“VR has come and gone many times,” says Jason Rubin, the head of content at Oculus and a legend in the gaming community. His videogames, including the Crash Bandicoot series, have sold more than 40 million copies. He’s been into VR for as long as he can recall, but lately he’s become fanatical. “If you go back and look at the ’90s VR, people were basically wearing [cathode ray] TVs on their faces,” he says. “They were -blurry and loud. Now we have high-resolution, low-cost screens with high refresh rates and low latency.”

The constant improvement of technology has enabled the Oculus Rift to cross an indefinable but crucial threshold: It can now create situations that simply feel real. Rubin thinks gaming will be the first market, then compelling content will bring wider use. “The ramp up from launch to widespread adoption might not be as fast as it was with the smartphone,” he says. “But over the long run it will be the same scale.”

The tech giants are staking their claims. Google moved for the low end with a cardboard-and-plastic device that turns a smartphone into a VR experience. It also recently launched a VR division with far greater aspirations and invested in the startup Magic Leap, which has raised more than $1.3 billion in funding.

Meanwhile, Microsoft has the HoloLens. It “augments” reality by layering holographic images on top of what you see of the actual world, with a headset that looks like a combination of sunglasses and a typical VR mask. The company, which is working with Autodesk (ADSK), Japan Airlines, NASA, and Volvo, among others, intends it to enable all sorts of training and exploration scenarios. Microsoft began shipping developer kits in March but hasn’t said when a consumer version will appear.

Intel is putting its weight behind VR as a way to help boost PC sales. Netflix (NFLX) has a VR app. Amazon (AMZN) is reportedly working on one as well. And Apple is definitely up to something. The company is mum about its plans (though CEO Tim Cook has declared interest), but officials are happy to forward leaked reports about how they’re hiring hundreds of people.

At Nvidia (NVDA), general manager Jason Paul tells me the graphics-chip company has created a new line of chips for VR and even recently formed a new division devoted to the cause. He identifies VR as one of its three strategic growth areas, and with good reason: VR rendering requires seven times as much graphics horsepower as 1080p HD rendering. “Over the last three years the technology and performance has come down to the point where it’s possible to think about a mass-market audience,” Paul says. “VR is going to touch every industry from gaming to automotive to entertainment, and we’re focused on helping build the ecosystem and deliver performance.”

VIR.05.01.16.sidebar VR getting real_v2 Illustrations by Nico 189 for Fortune Magazine

On my recent visit, he and his team show several demos. One put me in a $180,000 Audi R8. I was able to sit in the sports car’s front seat and even stick my head inside the engine. From this and the work Volvo is doing with Microsoft, it seems clear that the traditional test-drive will soon be augmented by the living room experience. You’ll hear the roar of the engine and feel the vibrations of the steering wheel as you hit autobahn speeds on a twisting alpine road—all from the safety of your sofa.

The arms race has already begun for selling headsets. Samsung has its Gear VR, produced with Oculus, which retails for $99 and uses a Samsung phone as the display. The company won’t disclose the number of units sold since the Gear launched in November, but it’s using TV ads and promotions to seed the marketplace. One recent offer: Anyone who buys a Galaxy S7 gets a free headset and six VR games. “There’s a lot of friction to adopting VR. If we can lower the barriers, that helps us explain why VR is so amazing, innovative, and powerful,” says Nick DiCarlo, Samsung’s GM of immersive products and VR. “You’re getting a level of simulation quality that might’ve been possible only in a $1 million system just 20 or even 10 years ago.”

At the tech trade show CES early this year, the real excitement surrounded VR gaming. Thousands lined up outside the Oculus, HTC—which launched its $799 Vive system in April—and Sony PlayStation booths for the chance to spend five minutes playing VR games. PlayStation executives hope that excitement reflects how 36 million PS4 owners will react when the $399 PlayStation VR is released in October. Sony will be doing everything it can to turn gamers into proselytizers. Because as executives readily admit, the power of VR can’t be conveyed through TV commercials. In the words of VP of marketing John Koller, “You just have to try it.”

VIR.05.01.16.pullquote Nick DiCarlo_v3 Photo: eVRydayVR

The emergence of any new platform inevitably brings questions about its utility—and even the businesses that first roll it out often fail to predict how it will ultimately be used. AT&T (T) introduced the telephone as an aural replacement for the telegraph, according to America Calling: A Social History of the Telephone to 1940. Ma Bell was so wrapped up in the paradigm of the telegraph—an expensive technology used mostly for business and life-and-death announcements—that for decades it discouraged customers from using the phone for socializing.

In 1943, IBM’s (IBM) Thomas Watson predicted “a world market for maybe five computers”—most likely because he was thinking about computers in terms of what they were (the size of a room) rather than what they would become (the size of a fingernail). Same with the Internet and the mobile phone. Visionaries predicted companies would one day send geotagged coupons to people’s cellphones. No one saw Uber coming.

Plenty of people are trying to envision compelling applications of VR. Among the more astute and sober is Jeremy Bailenson, head of Stanford University’s VR lab and co-author of Infinite Reality: The Hidden Blueprint of Our Virtual Lives. He’s responsible for exposing VR to captains of industry (including Zuckerberg) and heads of state (President Obama opted not to wear the headset).

Bailenson is delighted by the attention VR has been getting. It has meant large increases in funding and staff. “For the first time in history,” he says, “there’s a real energy in the space, and we can actually produce things.”

But he’s quick to manage expectations. “Most things don’t work in VR. If you show me 20 ideas, I’ll say 19 of them would be better in another medium,” Bailenson says. “I think VR is best for special, intense experiences … things that are expensive, dangerous, counterproductive, or impossible.”

VIR.05.01.16.pullquote Jeremy Bailenson_v3Photo: Brad Wenner

At his lab, I experience a series of demos that turn me—a forty-something, able-bodied white guy—into various form factors, including a black woman confronting an epithet-spewing racist on a city street and a disabled veteran without full use of his limbs. Clearly, Bailenson feels VR is great at fostering empathy.

He also has a capitalist streak. In January, Bailenson co-founded the VR software firm Striver, which is working with a dozen professional and college athletic teams. “With football,” he says, “the teams see a huge win just getting extra repetition. You learn to recognize defenses and how to change the play at the line of scrimmage.” Without, I hasten to add, fear of concussion. “In baseball, it’s batters realizing what pitch is coming. It’s about making decisions, experiencing something over and over, and driving that experience into your recognition.”

The value of such education is hardly limited to sports. NASA uses VR to teach technicians how to repair satellites. Lowe’s recently rolled out its Holoroom, which uses Oculus Rift to help customers make remodeling decisions, in seven test markets. “Our first name for the Holoroom was the Marriage Savior,” says Kyle Nel, executive director of Lowe’s Innovation Labs. “The intent was to learn what we could do to be successful in a world where virtual and augmented reality aren’t just cool but part of everyday life.”

Lockheed Martin often combines virtual reality with real-world equipment when training pilots to fly fighter jets. The company’s Shawn Muir, using an Oculus Rift headset with Lockheed Martin training equipment.Photograph by Bob Croslin for Fortune

A suspicious white van is parked 200 meters up the road, not far from the entrance to the U.S. embassy. “It wasn’t there when we walked into the room,” says Shawn Muir, a senior systems engineer at Lockheed Martin (LMT). Virtually speaking, I’m not sure where we are. Maybe Central America. Spanish-speaking pedestrians are milling about, surrounded by ramshackle buildings.

In reality, we’re in Lockheed Martin’s Innovation Demonstration Center in Orlando, peering around inside a 360-degree octagon. Muir is showing off the convoy training experience. He’s perched atop an actual Humvee, ducking behind a shield while gripping an actual modified .50-caliber machine gun and an M4 automatic rifle. The rifle is equipped with infrared lasers and a wireless BlueFire magazine, and it uses 3,000 PSI of compressed nitrogen to simulate the sounds of gunfire.

“Hold tight here,” he says. The Humvee stops, and he fires a few rounds into the van. It explodes. “Go ahead and move forward. We’re going to take a right up—”

Rattatat! Boom! Boom! Boom! Insurgents begin attacking from all directions. “Pull back! Pull back! Pull back!” Muir yells.

Demonstrating a F-35 fuselage at Lockheed Martin’s Innovation Demonstration Center in Orlando.Photograph by Bob Croslin for Fortune

The room’s shaking. My heart’s pounding, and I’m just a spectator. Eventually the insurgents retreat, and the simulation stops.

It’s not the most realistic demo I’ve seen. We’re not wearing headsets. The characters projected on the screen are cartoony. But people don’t require photo-realism to feel what VR experts call “presence.” Many factors signal reality to our brains, including motion, peripheral action, and sound. And the sound here is absolutely unnerving. It’s crucial to get used to it, says Lockheed Martin’s director of advanced technology, Atul Patel.“The worst thing that could happen on the battlefield is that you end up reacting negatively because you’ve never been in that scenario.”

Lockheed Martin has been using VR since the late ’90s. It has trained more than 400,000 soldiers in the convoy training experience alone. On my visit, I “fly” an F-16 and take a tour to see how the company is using VR to train pilots of the world’s most advanced fighter jet, the F-35 Lightning II. Sending a pilot into an actual F-35 for a warlike experience with adversaries, air threats, and partners might cost $1 million per hour or more. Do it virtually and it’s a small fraction of that.

It’s not just war games. Halfway across the country, Darin Bolthouse manages one of the world’s largest VR facilities for space systems, Lockheed Martin’s collaborative human immersive lab in Denver. Bolthouse and his team have incorporated VR into some 90 projects over the past year, including the next generation of GPS satellites. It helps engineers discover new approaches and answer vexing questions. “I can set up a VR scenario in a matter of minutes,” says Bolthouse, adding, “We’re saving in the range of $10 to $15 million a year by using it. That’s a conservative estimate.”

VIR.05.01.16.pullquote Kyle Nel_v3 Photograph Courtesy of Lowe’s

What would you do if you had 300 million fans in China? For the NBA, the question isn’t hypothetical. The basketball league enjoys an enthusiastic Chinese following, not to mention millions of fans in other countries, the vast majority of whom will never attend an NBA game. In the U.S. it has inventory constraints. With some teams, at least, demand for seats far outstrips supply. Which leaves a lot of potential revenue on the table.

Jeff Marsilio thinks VR can help solve these problems. He’s the NBA’s vice president of global media distribution. Marsilio first experienced VR as a teenager in a Connecticut arcade and now declares it the future of live sports. The NBA broadcast its first live VR game last October, and Marsilio invited me to Los Angeles to witness one of its production vendors, NextVR, film a game between the L.A. Clippers and the Denver Nuggets. We join the NextVR crew at the Staples Center. It’s a fairly bare-bones operation: just a few people peering into monitors and a feed from two stationary, unmanned cameras, one behind each basket.

It’s clear the experience needs refining. The Gear VR uses a mobile phone as its display, the electronic equivalent of watching a game through a screen door. That said, I’m transported to somewhere it would otherwise be impossible to be. It’d be an exaggeration to say it feels like I’m on the court, but it’s far more immersive than regular TV. The camera angles switch with the game flow, so the action is always coming at me, and the NBA’s grueling pace of play becomes especially pronounced.

Testing a virtual reality telecast for the NBA.Photo: Andrew D. Bernstein/NBAE Via Getty Images

As I watch, the NextVR team brainstorms how best to use audio, where to put the score, how to incorporate social sharing, and how to establish the new rules of engagement. “The technologies are all progressing so fast,” says Brad Allen, NextVR’s executive chairman, holding up a headset. “Think of these as the equivalent of the brick cellphone.”

When I ask Marsilio what kind of revenue opportunity VR represents, he demurs. “We’re putting monetization on the back burner. Our fundamental strategy is to bring fans closer to the game, and VR does that in a new way. This has the potential to be the most fundamental shift in how we experience content, ever. The money will follow.”

The next day, at NextVR’s offices in Laguna Beach, Calif., executives show me a VR demo of sports, from mixed martial arts to baseball, and discuss a deal they signed with FoxSports. They explain that NextVR is using the same video-compression technologies they originally developed for 3-D TV in 2009.

What makes them so sure VR won’t follow the same trajectory as 3-D TV? “There’s no comparison. 3-D was a TV that had depth added to it. But VR is a whole new medium,” says DJ Roller, a NextVR co-founder. “With VR, you’re no longer experiencing something from the outside. You’re in the event. It allows fans to not just be in a front-row seat but to be in a place that you can’t even buy a ticket.”

Elizabeth Baron, a Ford technical specialist uses VR to evaluate a vehicle before it is built.Photo: Courtesy of Ford Motors

Ted Leonsis knows a thing or two about the emergence of new media. He spent 14 years at AOL (VZ), including a stint as president. These days he’s the CEO of Monumental Sports & Entertainment, which owns the Washington Capitals and Washington Wizards, among other things. While reporting this story, I hear he’s become a VR fanboy and that the Wizards staff is using it to train players. So I arrange a call. Leonsis gushes over VR’s utility.

I ask whether the goggles will impede acceptance. After all, we humans have a thing about our eyes. Some cultures abhor eye contact. In others, looking away is rude. But covering one’s eyes with anything other than sunglasses just seems antisocial. I ask whether he’s ever tried Google Glass.

Leonsis laughs. “I have them right here next to me,” he says, launching into a soliloquy. “I haven’t worn them in two years. But that’s first generation. We know optical tools and the Internet of things will become more and more mainstream. They’ll get faster, cheaper, lighter, and smaller, and better apps will be developed. What will drive the social adoption? It’ll be videogames … two-screen environments where you’re playing a game, having an experience. Say we want to go see Niagara Falls together. We’re both wearing the headsets, standing out on that ledge, looking down together. It’ll be personal and emotional.”

I’m not sure I want to go to Niagara Falls with Leonsis. But I get his point. Given a powerful enough experience, we can get over the barriers to entry (which are lowering every day). Once, you needed a desktop PC and a modem to experience the Internet. Then you got a laptop and Wi-Fi. Now you have a supercomputer in your pocket and can’t imagine life without it. “It always takes longer for new technology to be socially accepted than what evangelists believe,” Leonsis says. “But once it takes off, it becomes bigger than anyone imagined. VR will change every industry.”

So think about a special experience you’d love to have—perhaps something expensive, dangerous, or otherwise impossible. Because there’s more than $2 trillion in market capitalization betting that you’ll say yes to wearing the scuba mask. Which means resistance is virtually futile. 

Jeffrey O’Brien is a former Fortune senior editor and a co-founder of StoryTK, a Bay Area studio that crafts stories for companies.

A version of this article appears in the May 1, 2016 issue of Fortune.