(Fortune) -- Sean Knapp had it made. As a young computer scientist, he couldn't have had a better gig: working at Google, the engineer's paradise. He had all the usual perks - a massage every other week, onsite laundry, free all-you-can-eat haute cuisine. Even better, he got to work on some of Google's highest-profile products, including the search technology that is the heart and soul of the company. And he made full use of his "20% time," that famous one day a week that Google gives its engineers to work on whatever project they want. A little over a year ago he and a couple of colleagues, brothers Bismarck and Belsasar Lepe, ages 28 and 21, respectively, did what many of the young geniuses do at Google: They came up with a cool idea, in this case a new way to handle Web video.

Then Knapp, who's 27, and the Lepes did something truly remarkable. They expelled themselves from paradise to start their own company.

That sort of thing just doesn't happen at Google - or it didn't used to anyway. In April 2007, when the trio informed Google they were leaving, they didn't give any specifics, just that they were going to do their own thing. Without even knowing what their idea was, Google wanted them - and their project - to stay. "They told us, 'Here's a blank check,'" recalls Knapp, a baby-faced former track star at Stanford. "I said, 'You're asking me to be a surrogate parent.'" Knapp and his pals would do the hard work, in other words, but Google would own the product. Instead, off they went, leaving behind the perks, the 20% time, and a combined seven-figure pile of unvested options. (Google declined to comment on Knapp's account.)

A year later the company the three founded, Ooyala, is precisely the kind of budding success Google wants to be creating inside its walls. Ooyala's 28 employees are building a system that runs videos for independent Web sites, and eventually they plan to sell video ads in the same way Google (GOOG, Fortune 500) hawks text ads for other web publishers. The startup has raised $10 million in venture capital, which doesn't even come close to matching Google's resources. But the Ooyala founders say what they lack in institutional backing they make up for in speed and the ability to communicate with one another by turning around in their chairs and talking. Google was like that too, about eight years and 18,000 employees ago.

It would be easy to dismiss the exodus of some of Google's best people if it were an isolated occurrence. It isn't. Paul Buchheit, an early Google engineer who coined the "Don't be evil!" battle cry, is a founder, with three ex-Google colleagues, of a social-networking company called FriendFeed. Yanda Erlich, once a popular Google product manager, started an instant-messaging company called Mogad. Nathan Stoll, who managed Google News, is hard at work on his new company, Mechanical Zoo. (It's in "stealth mode"- no details.) Former business-development guys Salman Ullah and Sean Dempsey have a new venture capital firm, Merus Capital, that aims in part to fund startups founded by ex-Googlers. (That's employees, in Googlespeak.) The departures have grown so numerous that the exiles have formed an informal alumni club of ex-Googlers turned entrepreneurs. David Friedberg, another former biz-dev executive, who started a company called WeatherBill, which sells insurance pegged to climate risks, recently attended the club's first meeting at a conference center in Palo Alto. "I was surprised by the number of things that were being done that could have been done at Google," he says.

There's been an exodus of executive talent too: Its chief information officer, Douglas Merrill, just left. Several top people have gone to Facebook, most notably Sheryl Sandberg, who ran Google's automated ad sales, and Elliot Schrage, who ran PR. George Reyes, Google's CFO, announced his retirement last summer and has yet to be replaced.

Employee turnover is the norm in Silicon Valley, especially at companies where early hires get rich enough to do whatever they want (and post-jackpot hires don't). For his part, Google CEO Eric Schmidt - who left Sun Microsystems for Novell and then Novell for Google - brushes off the effects of all those departures. "We've been hiring on the order of 100 people a week," he says. "So in one week we hire more people than the people you just named."

But fleeing executives and star engineers aren't the only challenges Google faces these days. You may be thinking: Challenges? Google? The company that nets $1 billion - plus a quarter and is so powerful it even scares Microsoft (MSFT, Fortune 500)? Yes, the very same. Fact is, Google's torrid growth is finally slowing, as the company's sheer size dictates it must. And size necessitates changes. Gone are the days when Google could take full advantage of its quirkiness. It's the market leader now, which presents a classic conundrum: Which is more important, process or innovation? For all Google's success, it still has just one meaningful way of making money: its powerful search-advertising system. It's a gusher, but it's the only one. All the other projects it has in the works are just that, projects.

The mere hint that Google may have issues to deal with clearly hits a nerve. As I was researching this article, Marc Benioff, the CEO of Salesforce.com (CRM) and a Google business partner, phoned to discuss an alliance his company had inked with Google. I'm always happy to hear from the voluble Benioff - except I hadn't called him in the first place. Someone "at the top" at Google had asked him to reach out to me. "What they need to do is build a full portfolio of revenue, as Microsoft has," says Benioff. "They have a fantastic cash cow. They need a goat and a chicken."

More than diversification, though, Google has to prove that its quirks - its odd hiring practices (e.g., asking 45-year-olds their GPAs), refusal to play the guidance game with Wall Street, the free food, etc. - will stand the test of time. It has to show that its success is because of its Googleyness (more Googlespeak), not despite it. Even its friends harbor doubts. "I'm not convinced they're in the ranks of GE or P&G or even Microsoft, for that matter," says Peter Chernin, president of News Corp., whose MySpace unit is a key Google partner. "Not yet."

A basic tenet of Google's way of doing business is that it is not like other businesses. Founders Larry Page and Sergey Brin celebrated this quality in their famous letter to prospective shareholders before the company's 2004 IPO, and they promised to keep things that way. For example, Google's operations themselves are unconventional. One of the most fundamental precepts of modern management has to do with how to allocate resources: deciding which projects to pursue, where to spend money, when to take a pass. In fact, MBAs learn in their first classes at business school that resource allocation is a manager's most important task. Yet it's a concept that, while not exactly alien to Google's top dogs, isn't their highest priority. After all, why focus on allocating scarce resources when the resources aren't all that scarce? At the end of the first quarter Google had cash and other liquid assets of $12 billion; it generates almost $2 billion of cash per quarter.