You would think Google’s executive triumvirate — CEO Eric Schmidt and co-founders Sergey Brin and Larry Page — would be worried about YouTube. Almost three years after they forked over $1.65 billion in stock to acquire the video-sharing site, YouTube last year delivered only an estimated $240 million in revenue and is deeply in the red.

YouTube is the largest video platform in the world. Users upload 20 hours of video to it each minute, at tremendous cost to Google (GOOG). The company doesn’t break out YouTube’s expenses, but analysts believe it spends tens of millions of dollars each month just on network capacity to host all those videos.

And, oh, what videos! Four years after its inception YouTube remains a repository for “long tail” content that appeals to niche audiences: clips of cats on skateboards, babies laughing, and kids lip-synching. (There are occasional mass-audience moments, like the clip of Susan Boyle on Britain’s Got Talent, viewed 71 million times.)

Gallery: See YouTube’s greatest hits

But despite Google’s repeated efforts, YouTube has failed to create an environment for professional video content, where many advertisers are clamoring to put their money right now.

Instead, Google’s executives have watched rival Hulu, a joint venture of GE’s NBC, News Corp.’s Fox, and now Disney’s ABC, catapult to success among consumers and advertisers alike. With its sharp media player and an interface that feels comfortingly like traditional television, Hulu has become the second-most-watched video site (after YouTube) since it launched in March 2008 and began hosting episodes of Saturday Night Live and The Office. YouTube’s attempt to capitalize on professional content? A tab on the site marked “Shows” that is home to programs like Bewitched. Really.

Yet Google isn’t backing away from its cash-guzzling purchase. In fact, the parent company appears to be doubling down on YouTube. In the past year Schmidt has quietly moved some of Google’s brightest execs to YouTube’s offices. Engineers have been tweaking the technology that sorts and spits out YouTube content to make it easier for marketers to buy sponsored search links and simultaneously run advertising in relevant YouTube videos.

The site now is able to make money from an estimated 13% or more of its videos, up from roughly 3% a year ago. Patrick Pichette, Google’s cost-conscious finance chief, was surprisingly upbeat on a recent earnings call, saying soon “we actually see a very profitable and good business.”

Ultimately, though, Google’s ambitions for YouTube have nothing to do with cute cat clips or viral web video, and everything to do with data. Video is still in its infancy on the web, but it is growing up quickly. Laptops and iPhones now have quality videocameras. Web surfers are as likely to search YouTube for videos on how to cook a turkey as for the latest Miley Cyrus single. And teenagers increasingly use video instead of e-mail to send each other messages, posting the clips on their Facebook walls.

So while Google aims to profit from the videos on the site today, it ultimately is more interested in making sure that the company becomes the primary platform consumers use to generate, store, sort, and view all their video content and communications. (This, by the way, is exactly the kind of wide-reaching power that concerns privacy and antitrust advocates.)

“If you look at it from a Google perspective, video is another data type,” says David Eun, a vice president of strategic partnerships at Google, who spends most of his time on YouTube these days. “Why would I want to be an entertainment box when we’re talking about all video on the web?”

Getting a Google makeover

Much has changed at YouTube since Chad Hurley, 32, founded the site along with fellow PayPal alumni Steve Chen and Jawed Karim in 2005. At the San Bruno, Calif., office, just south of San Francisco, a Google chef provides fresh-baked pastries to cap off daily gourmet lunches. Chen left YouTube to work on a special project at Google last fall. Around the same time, Salar Kamangar joined Hurley to help build out the site. Hurley “was bugging me about spending more time over here, and I thought it made sense,” Kamangar recalls.

Kamangar, 32, is part of a core group of engineers that Google has installed at YouTube in the past few years, and he’s in charge of the quest to make money from the site. Though he has always kept a low profile, he’s well regarded by such Silicon Valley luminaries as investors Ron Conway and Jim Breyer, who calls him a “young superstar.” Kamangar was the early Google employee who came up with the concept for AdWords, Google’s biggest revenue source, which lets advertisers pay a fee each time users click on their links. Google hopes he can work that same magic at YouTube.

He’s making progress. In the first half of 2009, Kamangar and Hurley launched more new products than the unit had in the previous year. Although most of the ad sales have been tied to the larger banner ad on YouTube’s front page, it has introduced different types of ads, including small ads that pop up inside videos. It has also added click-to-buy options to some videos. (Watch Michael Jackson’s “Billie Jean” on the site, and you have the option to buy the song online from Amazon (AMZN).)

Kamangar also continues to improve YouTube Insight, the analytics tool that now includes a heat map that shows video producers which parts of their video get the most attention. And it’s experimenting with combining YouTube buys with media buys on other Google networks. If an advertiser wants to reach, say, an online audience of 5 million people of a certain demographic, Google, mining its incredible cache of user data, can produce a plan that shows the marketer the combination of YouTube ads and other online and cable TV inventory that would be most effective. It’s a nascent service but one that no other video-sharing site can offer today.

Still, it is worth putting YouTube in context: Cowen & Co. analyst Jim Friedland calls YouTube “a smart strategic deal that created incremental value,” but he nonetheless sees the video site in its current incarnation producing at best $1 billion a year in revenue in, say, five years. By contrast Google’s search will be a $30 billion business, he estimates.

“Not the Red Cross”

YouTube has also invested substantial resources into copyright protections for professional content that used to find its way onto the site with alarming regularity. (After Google acquired YouTube, media conglomerate Viacom (VIA.B) sued the video-sharing site for illegally showing its protected programs.)

Today YouTube is using digital-rights management software to identify professional content and in turn help media companies make money from their YouTube videos. Media companies are grudgingly pleased with the results: NBC’s general counsel, for example, was forthright about how effective the technology was in keeping Olympics coverage off YouTube during the summer airing.

It is backhanded praise: NBC’s comment is a reminder of how little of YouTube’s content is produced by network studios — the stuff that’s most easily monetized today. And if Google merely wanted to sell ads against its inventory of online videos, that might be a problem since it costs the company way more to host and deliver videos than it currently makes on video ads. (A Credit Suisse report suggests that the site spends $375 million a year on network infrastructure alone; a recent study by strategic outsourcing adviser RampRate puts the annual costs at closer to $83 million.)

But Google is thinking well beyond video ads. The goal is to make YouTube the top online destination for all kinds of videos — not just the high-quality network television shows or sports highlights found on competitors’ sites. Establishing that loyalty with users, Google’s Eun suggests, will pay off in the future when consumers are looking for a tool to organize the proliferating volume of online videos they require for information, entertainment, and online conversations. “It’s not the Red Cross,” he says. “We are building a business here.”

One particularly promising area is instructional videos (how to put on eyeliner, say, or how to ride a horse). Today if you’re going to make risotto you might simply go to Google and enter a bunch of keywords, click on a recipe, and print it out. But soon you may instead go to YouTube to find a short clip showing a local chef making the dish. An advertiser such as Williams-Sonoma (WSM) might run a short video ad during the segment — complete with the opportunity to purchase a new stockpot — while foodie-friendly advertisers sponsor the search results with video advertising. You could even record yourself cooking the dish, post the clip on YouTube, or forward it (via Gmail) to your best friend.

Earlier this year YouTube quietly became the second biggest search engine on the web; only Google.com is larger. If video online becomes pervasive and consumers first gravitate to video searches rather than look for articles or websites, it may be only a matter of time before the long tail wags the dog.

Related: Online video sites fizzle

More from Fortune:

Bartz and Ballmer on the Yahoo Microsoft search pact

Why the market’s mad at Yahoo

Microsoft’s $2 billion online problem

Reporter associate: Kim Thai