Regina Dugan loves to tell the story of how she got her current job. It was a little over two years ago, and Dugan, a mechanical engineer by training and an expert in counterterrorism, was finishing a three-year stint as director of DARPA, the Defense Department’s prodigious technology research organization that gave birth to things like the global positioning system, the stealth fighter, and the Internet. During her tenure, she sharpened its focus in areas like cybersecurity and new forms of manufacturing and on delivering tangible results. “DARPA is a place of doing,” she told Congress in 2011. It’s an attitude that earned her praise among the tech elite—including veteran venture capitalist John Doerr, who sums her up in four words: “She’s an impressive leader.”

Among Dugan’s many fans was Eric Schmidt, Google’s chairman, who suggested she go on a two-day visit of the Googleplex in Mountain View, Calif. The idea was to see if there might be a fit between Dugan and some project or other at the sprawling search and advertising giant. After making the rounds of various groups, Dugan sat down with Dennis Woodside, then the CEO of Google’s Motorola unit, who was charged with turning around a brand that was once synonymous with cellphone innovation but that had lost its way in the smartphone era. Woodside said that with a renewed focus on innovation, Motorola could leapfrog rivals like Apple and Samsung. His plan was to hire a mobile-industry veteran to lead an advanced-technology group that could deliver the inventions that would restore Motorola’s status as a pioneer.

What, he wondered, did Dugan—whose job had been to nurture DARPA’s decades-long streak of breakthroughs—think? “It’s a great strategy for not losing and a lousy strategy for winning,” she answered. A week later the Motorola innovation gig was hers.

Google ATAP's Regina Dugan Wall of Secrets: Regina Dugan keeps a close watch on the progress of ATAP’s various projects. (Her team insisted that all of their scribbled sticky notes be covered by blank ones.)Photo by Cody Pickens for Fortune

Today Dugan is leading a cadre of big-idea special forces—called Advanced Technology and Projects group, or ATAP for short—on an even grander mission for parent Google, where the unit is slated to remain after the sale of Motorola to Lenovo is completed later this year. (Woodside left the company to become chief operating officer of Dropbox.) The challenge? Bringing to fruition a slew of next-generation mobile wonders—not in a generation’s time, mind you, but in a handful of years instead. And though the group is still in its early days, it appears on track to deliver some jaw-dropping technological leaps that would help cement the 51-year-old Dugan’s cachet in Silicon Valley and beyond.

The first woman to head DARPA, she is comfortably at home in the top echelons of tech research, a field overwhelmingly dominated by men. The New York City–born engineer, who received both her undergrad and master’s degree at Virginia Tech before earning a Ph.D. at Caltech, is swaggering and stylish, with a helmet of thick, dark hair, piercing eyes, and a penchant for jeans, leather jackets, and scarves.

Bolstering that cool-geek cred is her mastery of the tech talk. In June at Google I/O, the company’s developer conference in San Francisco, for example, she filled her presentation on her Motorola work with catchy—and sometimes trite—one-liners that mixed showmanship and bravado. “Here we don’t tinker,” she said. “We build new things, sometimes seemingly impossible things.” Moments later she mused that her group innovates “without compromising the tech, or the beauty or a sense of soul.” Her talk had the desired effect: The hundreds of engineers in the audience hooted and hollered throughout as if it had been a Steve Jobs keynote.

But by all evidence, it seems, Dugan’s style is backed by substance. Since she joined Motorola, she has brought a radically different kind of disruptive innovation to Google. ATAP has emerged as something of a counterpart to Google X, the better-known moon-shot factory that is working on self-driving cars, glucose-detecting contact lenses, and other wonders. ATAP’s projects are more narrowly focused on mobile, but they, too, could easily land a leading role in a sci-fi movie script. There’s a digital tattoo that you can paste onto your forearm and use to unlock your smartphone. There’s Ara, a project to reinvent the smartphone so that it can be assembled on the fly to a customer’s specs. It would give users the ability to choose hardware components to fit their needs—a camera, sensor, battery, or, say, an oximeter that measures pulse rates and blood oxygen levels—just as they choose apps. There’s Tango, a prototype tablet that can see the world around itself in 3-D to map the inside of your home, for instance, or help a blind person navigate. The most whimsical—or perhaps Googley—ATAP project is Spotlight Stories, an exploration of how smartphones can help rewrite the rules of animation and storytelling. Staffed by a group of movie industry veterans that includes the animator behind Disney’s Aladdin and The Little Mermaid and an Oscar-winning director of Pixar’s Ratatouille, Spotlight Stories is creating a new form of immersive video content that is controlled by the user and that is also pushing the limits of a smartphone’s capabilities.

Google ATAP project Tango Renzo De Nardi works on the Project Tango prototype at Google ATAP’s officesPhoto by Cody Pickens for Fortune

Dugan won’t discuss ATAP’s budget or, for now, its other projects. There are 11 in all, and they were listed on a recent presentation slide under names like “auth,” “imaging,” and “MB,” which did little to shed light on their objectives. (Before we could photograph one of ATAP’s brainstorming rooms, the company insisted on covering the Post-its on the walls with blank ones.) But Dugan presides over this collection of public and secret endeavors with obvious pride. ATAP, she said at Google I/O, is “a small band of pirates trying to do epic shit.” After a theatrical pause, she added, “A small band of pirates in a very fast boat.”

Hyperbole aside, Google’s mobile-innovation lab is fascinating not only for the ambition of its disparate projects but also because of how it aims to innovate. Dugan brought from DARPA, based in Arlington, Va., an unorthodox approach that’s never been tried at scale in the private sector. It seeks to get around problems that have plagued other advanced-research labs in the tech industry. Some are so focused on pure science that their work, while academically interesting, only rarely yields practical applications. Others who have tried to meld breakthrough research with product innovations have had mixed results. More often than not, product teams shy away from science that is too risky, and scientists squirm at the idea of being hemmed in by product specs and deadlines. The diverging priorities, Dugan says, lead to compromises, and soon projects devolve from “epic shit” to “shitty in an epic way.” (The sprinkled profanity seems to be part of Dugan’s pirate shtick.) A rare exception to this skunkworks syndrome is DARPA, which has consistently opened new scientific doors as it delivered useful products—and that’s exactly what ATAP is trying to do at Google. “The question is how you have an enclave that produces a string of breakthrough advances time after time,” Dugan says.

Google ATAP project Tango Project Tango: Technical lead Johnny Lee holds a modified Android tablet equipped with extra sensors and cameras that allow it to see and map its surroundings; inset: A park bench is “mapped” in real time; Right: False color rendering of an interior staircase as mapped by a Project Tango tabletLeft: Photo by Cody Pickens for Fortune

So what’s the ATAP playbook? It starts with identifying a project that demands a quantum leap in both scientific understanding and engineering capability to pull off (more on that soon). Once that is done, Dugan works to assemble a core team of experts at Google. But that team quickly casts a much wider net, tapping what are often a huge number of outside collaborators from across a mix of disciplines in industry and academia. That allows ATAP, with a staff of just 75 full-time members, to be far smaller and scrappier than traditional research labs.

Together with its partners, ATAP forms what Dugan calls special-forces research teams. For project Tango, for example, the lab had to rally experts in computer vision and robotics, as well as experts in lenses, cameras, manufacturing, and more. The core ATAP group involved just a dozen scientists and engineers, who joined with some 40 outside partners, including universities in the U.S. and Europe and NASA’s Jet Propulsion Laboratory.

In all, ATAP has worked with 326 partners—universities, startups, large system integrators, government and nonprofit organizations—in 22 countries. (Dugan crafted a rare multi-university research agreement with more than two dozen leading universities that allows ATAP to quickly contract with researchers at places like Stanford, MIT, Caltech, and the University of Michigan.) “Today things are moving so fast that a diversity of skills and of points of view matters,” says John Sealy Brown, who once headed the Xerox Palo Alto Research Center, one of the most prestigious and innovative industry research organizations. “Often you need to have a multitude of disciplines brought together quickly.”

Then there’s the requirement that ATAP projects deliver a finished product. That goal works to inspire ingenuity in researchers and intensify their focus. Neil Gershenfeld, an MIT professor who directs the university’s Center for Bits and Atoms and who is collaborating with ATAP on its modular phone, says the product focus helps ground his research. Dugan points out that many technical challenges can only truly be solved when a researcher builds not just a prototype but also a product that can be made at scale. (Tango expects to ship devices to developers soon. And Spotlight Stories has already created two animated shorts, and a third one is on the way.)

The final, but critical, piece of ATAP’s formula is impatience. Projects are on a two-year deadline so that they don’t become open-ended research initiatives. If tangible results aren’t achieved quickly, they’re shelved to make room for new ones. DARPA emphasized this by printing employees’ last day on their badges. At ATAP, techies are reminded periodically that, with every passing week, they are 1% closer to their last day. “There is a sense of urgency,” Dugan says. “You don’t come to build a career. You come to do a project, to do something epic, and then you go.”

Google ATAP's Regina Dugan Regina Dugan holds a glass optical fiber a technolgoy used for aesthetic lighting in tight spaces at Googles ATAP officesPhoto by Cody Pickens for Fortune

Research groups have a storied place in America’s technology industry. Over the years corporate giants have hired armies of scientists to build labs whose research rivaled that of prestigious academic institutions. AT&T’s Bell Labs, for instance, is credited with developing the transistor and the laser, and work performed within its walls was recognized with seven Nobel Prizes. IBM gave birth to Watson, the artificial-intelligence system. And not every engineering whiz thinks the classic innovation model is broken. “IBM is not an ivory-tower research enterprise,” says Zachary Lemnios, a DARPA veteran who oversaw the agency at the Defense Department during Dugan’s tenure and who is now a top executive at IBM Research, a massive organization of 3,000 researchers that dwarfs ATAP. “We have a balanced portfolio with near- and long-term projects.” Yet many tech giants have struggled to capitalize on the breakthroughs of their own labs, either because the research was too far removed from practical applications or because corporate parents didn’t grasp the value of their labs’ inventions. (Xerox PARC famously developed the computer mouse and the graphical user interface, only to see them commercialized by Apple and Microsoft.)

Google stands out for having largely shunned the approach. The company has long embedded its advanced researchers within product groups, leading to breakthroughs in areas like search (Google Now), speech recognition (dictation), and machine translation (Google Translate). Google X was one of the company’s first formal efforts to cordon off a research group, but it, too, is focused—not on basic scientific research, but on specific projects (like autonomous vehicles) that require technological leaps to achieve. Google X is going after the kind of big, audacious ideas that have been the hallmark of CEO Larry Page and that may take years to bear fruit. For breakthroughs in mobile computing, an industry that was born in earnest only in 2007 when Apple introduced the iPhone, ATAP’s shorter time frame makes more sense. “I don’t even know what it means to imagine a 10-year time horizon in mobile right now,” says Dugan.

As it is with most research groups, the business case for ATAP projects—or, for that matter, for Google X’s—is not always apparent. Google’s philosophy is that huge conceptual advances will pay off in some way, even if it’s not clear how at first. And in the process the research and development involved helps attract and retain some of the world’s top talent. The buzz linking Google with the notion of bold innovation doesn’t hurt either.

Dugan aspires for her projects to be in what the political scientist Donald Stokes dubbed “Pasteur’s quadrant.” Stokes categorized scientific research by whether it sought a fundamental understanding of nature (“Bohr’s quadrant,” after the physicist Niels Bohr) or whether it tried to solve immediate problems (“Edison’s quadrant,” after Thomas Edison). He defined the Pasteur quadrant as something of an intersection of the two other quadrants: research that expands basic science but that is driven by an immediate application. (Stokes left the fourth quadrant empty, as it would refer to work that pursues neither new science nor practical uses.) To find projects that fit the bill, Dugan’s teams follow one of two approaches: They make an observation as to where technology is heading and find an application to push it there faster, or they find an application that needs a new type of technological solution.

Project Ara fits that “Pasteur” rubric well, it would seem. The idea for it came at a time when Motorola was releasing the Moto X, which was the first customizable smartphone on the market, allowing users to choose the color for its back, buttons, and bezel. Through sales data and its own research, Motorola discovered that consumers put a premium on such customization—but not just for the look of the device; they valued it for function as well. Moreover, they felt a special affinity for a product that they had a hand in creating. The obvious next question for ATAP was this: How far should it take customization in the mobile space? To answer it, Dugan tapped Paul Eremenko, a former DARPA colleague. Eremenko knew nothing about mobile phones, but he had worked on modular satellites.

Google ATAP project Ara Left: The components for Google’s fully customizable phone are slick, contactless plastic tiles that snap into place onto a 3-D-printed frame. Allowing a multitude of customization; Center: Paul Eremenko the Tech Lead for Project Ara worked on modular;satellites at DARPA. Right: Mock-up of an Ara phone with customized modules. *Psst! Artist’s rendering only. Not an actual phone.left and center: Photo by Cody Pickins for Fortune; Right: Courtesy of Google-ATAP

When he proposed that ATAP pursue a fully modular phone, whose components could be swapped in and out at a user’s fancy, the overwhelming reaction in the tech press was negative. The main criticism was that a modular phone would, by necessity, be inferior—bulkier, slower, more power-hungry, uglier—to one built entirely by Apple or Samsung. Eremenko, for his part, doesn’t dispute the claim at all. But he considers the challenges of modularity to be merely “overhead,” and if that overhead can be brought down sharply, as he thinks it can, the modular phone would be viable. Indeed, it’s in such challenges that the ATAP model shines. The Ara team is pushing engineering science forward in a variety of fields, from building miniature electro-permanent magnets that can latch components together to designing antennas that can be moved and reconfigured.

With just five staffers at ATAP, Ara has relied on its partners to make big strides toward its goal: a modular device that can boot Android, Google’s mobile operating system. While the prototype looks like something out of a hobbyist’s workshop, with wires sticking out of it, the finished phone that Ara envisions is attractive and slick—built around a 3-D-printed core, where the modules are colorful square and rectangular tiles with rounded corners that snap snugly into place. If it is successful, the implications for the mobile industry could be huge. With just a few modules, for example, a phonemaker could build an entry-level smartphone for the developing world, where hundreds of millions of people still use feature phones. Users could then upgrade gradually. A modular phone could also lower the barriers to participating in an industry that’s controlled by a few giant manufacturers. High-end audio companies like Bose and camera makers like Leica could make premium sound or photography modules that they could sell directly to consumers; Fitbit and other makers of activity trackers could build modules for phones rather than standalone wearable devices. And over time, customization could be pushed further inside the modules themselves, tackling new technical challenges along the way. “Ara is a first step toward making the hardware as malleable as the software,” says Gershenfeld, the MIT scientist.

Glen Keane may be the last person you’d expect to find toiling away in a cubicle at Google. Keane began drawing as a child, starting with dinosaurs, so that he could imagine himself living the worlds and characters he was creating. Later, over four decades at Disney, Keane shared with millions of moviegoers the experience of living inside characters he animated—Aladdin, The Little Mermaid’s Ariel, the Beauty and the Beast. Now Keane sits in front of an easel, a box of pencils at the ready, and draws frame after frame on white sheets of paper inside ATAP’s otherwise nondescript temporary offices at the Googleplex. (The group just moved from Motorola, in Sunnyvale, as it waits for its more permanent office.) Keane says he’s never before felt so palpably “inside” his illustrations as he does now. “As an artist, I got to live in the skin of those characters,” Keane says. “It was very true to the experience I have in my imagination.”

Google ATAP Spotlight series Left: Hand-drawn animation meets the future of digital storytelling for the mobile era: above, a character in Duet, a new short animation that unfolds in a 3-D world; Center: Rachid El Guerrab, technical lead for Spotlight Stories, works with one of Glen Keane’s illustrations; Right: Animator Glen Keane created over 10,000 pencil drawings for the film “Duet”.Left and Center: Photo by Cody Pickens for Fortune: Right: Courtesy of Google ATAP

Keane’s Google project is Duet, a whimsical and visually stunning animated short that tells the story of a girl, Mia, and a boy, Tosh, following them from birth to their falling in love as young adults. Like the two other Spotlight Stories produced so far, Duet, which has not yet been released, is meant to be watched on a mobile device. The screen serves as a movable window into a story that’s unfolding in a three-dimensional world that surrounds the viewer. Point the screen in one direction and you’ll see one part of the story; to see another part, point up or down or around. In Duet, a viewer might choose to follow Mia for a while, then Tosh, and then Mia again as the two characters cross paths repeatedly. “At every meeting point, you can choose to follow one or the other,” says Rachid El Guerrab, the technical lead for Spotlight Stories. And as you watch Duet again and again, you could choose to see different things each time. If a traditional video is akin to looking at a painting—a fixed frame in which the story unfolds—Duet and its ilk are more like exploring the inside of a cathedral.

Spotlight Stories grew out of Dugan’s observation that while just about everyone uses smartphones and tablets for entertainment, the typical experiences (traditional videos, simple games) rarely take advantage of the devices’ power and capabilities. El Guerrab, a veteran of the gaming world, and a handful of others began exploring how to turn the Moto X’s graphics capabilities into a 3-D canvas. Soon after, they decided to bring in a team of Pixar veterans, led by Jan Pinkava, who won Oscars for co-directing Ratatouille in 2007 and directing an animated short a decade before. At ATAP, the group produced two computer-generated shorts, Windy Day and Buggy Night, which are available on Moto X phones. Duet is the group’s first hand-drawn animated short.

It’s something of a paradox that Spotlight Stories appears to be the least techy of ATAP’s projects and yet is also the one that most resembles the kind of blue-sky exploration done at research labs of yore. With its three-dimensional canvas, the Spotlight Stories team creates what is arguably an entirely new art form, one that future artists may explore in ways currently unknown. For Keane, someone who is used to controlling a story’s viewpoint with cuts, close-ups, wide shots, and cutaways, it has meant learning a whole other language. “You think more as a magician in some ways,” he says. “Instead of forcing a viewpoint, you coax, you seduce somebody over.”

Because of technical requirements having to do with video compression and how video is rendered on mobile screens, he had to learn to draw in 60 frames per second rather than the typical 24. In the end, the result, while meant for a small mobile device rather than a giant movie screen, was more satisfying; it established a new, closer relationship between artist and audience. “Sitting in a big movie theater—there’s something cold about that,” Keane says. “When someone is holding your art in their hand, it’s very personal.”

This new art form may spread to Hollywood and beyond, as Keane believes. Or it may not. And that’s just fine for Dugan. “It’s a gift,” she says of Duet and the other shorts. A gift that combines beauty with a sense of soul.

This story is from the September 1, 2014 issue of Fortune.