"These books try to impress you with the massive amounts of data that they gather, but much of the data are not valid," says Phil Rosenzweig, a professor at Switzerland's International Institute for Management Development and author of "The Halo Effect," a 2007 book that set out to debunk much of the business-success literature. "These sorts of data are seen through the lens of the company's success. They don't explain the company's success, they are explained by it." Along with Rosenzweig's, the past few years have seen books by Robert Sutton and Jeffrey Pfeffer of Stanford arguing for a more truly evidence-based business-success literature.

But a few consultants and business school professors have begun to argue that much of this literature is, in fact, useless. Far from a science, they argue, the success literature is made up of little more than just-so stories in which authors use dramatic anecdotes - often drawn from previously published magazine profiles or interviews with the very executives whose performance is being examined - as evidence for "secrets" that amount to little more than warmed-over homilies. The critics accuse the success gurus of cherry-picking their evidence, of doing little to double-check their results, of circular reasoning, and of making elementary statistical errors.

At their most ambitious, these books purport to elevate the study of excellence to a science, its nuggets culled from exhaustive research and refined by painstaking analysis. Jim Collins, coauthor of "Built to Last" and author of "Good to Great," likens what he does to physics. Readers of his books, he writes, have their eyes opened to the "immutable laws of organized human performance."

While the particulars vary, the basic idea underlying the literature is the same: that the secrets of success can be divined by careful study of the institutional habits of the world's business all-stars - companies that set the standard for their industries, that thrive in tough times, companies that win the war for talent, companies that are built to last. In the imperturbable focus on core values of Hewlett-Packard or the restless innovativeness of Google or the ruthless accountability of GE, there are lessons for us all.

Even those people who haven't read the books (or listened to the CDs or watched the DVDs) live in their shadow. Working at companies whose executives have embraced the wisdom of one or another success bestseller, many people have their daily lives shaped by its prescriptions, and grow dimly familiar with the catch phrases - like "stick to the knitting" and "level 5 leadership" - that the books have spawned.

NOTHING SUCCEEDS LIKE success - and few books succeed like books about success. Since the 1982 publication of "In Search of Excellence," by the McKinsey consultants Tom Peters and Robert Waterman, the business-success book has grown from a genre into a juggernaut. The most influential among them, like "The Leadership Engine" and "Reengineering the Corporation" and "Built to Last" and "Good to Great," sell millions of copies, are translated into dozens of languages, and shape the habits of managers from Detroit to Dubai. Their authors launch research firms and leadership education institutes, consult for Fortune 500 companies, and command $50,000 fees to speak at conferences and corporate retreats.

Running through "In Search of Excellence," though, is also a note of reassurance for a powerful country that feared it had lost its edge. "There is good news from America. Good management practice today is not resident only in Japan," Peters and Waterman write early in the book. To back this up, the two authors picked out 43 American companies with especially good reputations in the business world (they determined this, Peters later recounted, simply by asking around the offices of McKinsey) that had also delivered strong long-term financial growth. Among the book's paragons are Hewlett-Packard, Boeing , Johnson & Johnson , IBM , and Caterpillar. Then Peters and Waterman, sifting newspaper and magazine articles about the companies and talking to the firms' executives, set out to divine what set those companies apart from the mediocre and the merely adequate.

"It's around that time that business pages became more popular," recalls Jacqueline Murphy, editorial director of Harvard Business School Press, which has published several business-success bestsellers. "People became more interested in investing, in finance, and in management as a topic."

t was perhaps inevitable that something like the contemporary business-success book would arise out of the early 1980s, an era defined both by a growing popular interest in the worlds of business and finance and, like today, a deep insecurity in American boardrooms.

Still, critics like Raynor believe we can do better, and they believe that their work will help pave the way to a more truly scientific study of success. But, at the same time, their critical insights also raise fundamental questions about how possible such a science is, and remind us just how much success - whether in business or in life - can be out of our hands.

"It's not the same kind of data that you would use to refute or confirm Einstein's theory of relativity," says Peters. "It's exploratory social science research."

For their part, authors like Collins and Peters see such critiques as caricatures of their work, and defend their methodologies as the best one could hope for in distilling the endlessly complex contingencies of business into digestible lessons for working managers.

"When we look at the samples of great companies in most studies, by our measure, the companies that they call great by and large aren't," says Raynor. "The conclusions they come to are more a function of the researcher than the company."

And most recently, a team led by Michael Raynor, a researcher and consultant at Deloitte Consulting, has begun to argue that business-success books aren't even particularly good at spotting success in the first place: When you take a closer look at the companies they study, the accomplishments of the vast majority are just as likely to be due to simple luck. It's the equivalent of finding someone who flipped a coin seven times and happened to end up with seven heads and asking for her secret.

Recent years, however, have seen a proliferation of critiques of books like "Good to Great," fueled in part by the same quantitative urge behind the push for evidence-based medical care. In last fall's issue of The Academy of Management Perspectives, two separate articles took issue with "Good to Great." One of them, by Bruce Resnick and Timothy Smunt, professors at Wake Forest University's Babcock School of Management, found that if the 15-year window during which Collins looked at each of his "great" companies was moved by as little as a few months, the exceptional stock market performance that distinguished them all but disappeared.

But he also makes bolder claims for the principles he unearths. In "Good to Great," he lays out the road map for merely good companies to transform themselves into great ones, using as his case studies companies like Wells Fargo, Philip Morris , and Gillette. The process starts with having leaders with "a paradoxical blend of personal humility and professional will" who get "the right people on the bus," instill a "culture of discipline," and apply "technology accelerators." Such elements, he writes, gain their power from "the enduring physics of great organizations."

Criticisms like these did little to dent the authors' influence, especially that of Peters, who became the first of the high-profile management gurus and inspired a raft of imitators. The most famous of his heirs has been Collins, also a former McKinsey consultant. Collins's own books - "Built to Last" was published in 1994, "Good to Great" in 2001 - address some of the criticisms leveled at "In Search of Excellence"; in particular he makes sure to include comparison companies to contrast to the exemplars he lauds in his books.

And the fact that Peters and Waterman had looked only at companies that they deemed successful - without comparing them with less excellent competitors - meant the authors were in little position to identify what factors mattered and which were irrelevant. They had no way of knowing, in other words, whether 43 utterly dysfunctional companies might be just as likely to be characterized by a "bias for action" and "loose-tight properties."

The book was an immediate sensation and went on to sell more than 6 million copies. But even at the time there were questions raised about its reliability: few of the companies Peters and Waterman showcased kept up their market-beating stock performance in the years after the publication of the book (though a good portion did later return to posting impressive gains). Many of them struggled, and a few, like Atari, Wang, and Data General, collapsed.

Their answer was a set of eight "attributes of excellence." Number one was "a bias for action," number two was "staying close to the customer," number eight was "simultaneous loose-tight properties," which is defined as "fostering a climate where there is dedication to the central values of the company combined with tolerance for all employees who accept those values."

Another line of criticism aims not at the math but at the sort of evidence that success books rely on. Much of that information, argues Rosenzweig, is biased. In many cases researchers rely on newspaper and business magazine articles and business school case studies, as well as internally produced documents and interviews with managers at the companies being evaluated, and the amount of data gathered can be impressive. These sources give the books their breezy, intimate, anecdotal feel.

But Rosenzweig argues that all of these sources can be easily tainted by whether or not a company is understood to be successful or not. The business press, for example, can hail a CEO as a genius when his stock price is up then turn around and assail him as a cretin when it drops. And even the perceptions of company insiders can be sensitive to how someone else tells them the company is doing. A host of psychological studies, for example, have shown the extent to which arbitrarily telling a person they have either excelled or failed at a task shapes their memory of the task: people told they succeeded are far more likely to remember their team as tightknit and their team leader as competent and inspiring. Those told they failed remember team squabbling and ineffectual leadership.

These tendencies, Rosenzweig argues, are only likely to be exacerbated by questions that ask them to explain the success of their company - one of Collins's interview questions, for example, asks managers, "Can you think of one particularly powerful example or vignette from your experience or observation that, to you, exemplifies the essence of the shift from good to great at [good-to-great company]?"

The newest and perhaps most radical critique, though, comes from Raynor, of Deloitte, along with the Deloitte consultant Mumtaz Ahmed and the University of Texas business school professor Andrew Henderson. In a paper currently under review, the three argue that not only are business gurus bad at identifying the causes of success, they have no way of telling true greatness from mere luck - if enough people are flipping coins, someone is likely to string together an impressive run of heads. According to their analysis of 13 of the most influential business success books, three quarters of the purportedly great companies had track records that could just as easily have been explained by the vicissitudes of random chance - performances that looked impressive on first glance were simply akin to being the lucky person in a stadium full of coin-flippers.

And if that's your data, says Raynor, "you're not inferring the underlying causes of great performance, you're basically just imposing patterns on randomness."

In response, Collins points to unpublished work he is doing that shows that luck does not, in fact, explain the difference between the winners and losers in his model. "There is, however, a significant difference in how the winners and losers view the role of luck - and therein will lie an absolutely fascinating chapter!" he writes in an e-mail. On a broader level, he objects to the characterization of his work as "success literature." It is more concerned, he argues, with discovering why companies do not become great than with why they do.

Raynor and his coauthors, though, are not arguing that luck explains success, but that it masks it. And they've set themselves the tricky - and perhaps impossible - task of coming up with a set of metrics that avoid the pitfalls of subjectivity and self-fulfilling prophecy. Some of the factors they're looking at are the timing of product launches and mergers and acquisitions and the degree of geographical diversity. These are, Raynor admits, measures that don't yet add up to a rich portrait of an organization, and may not do much to anatomize its excellence or lack thereof.

Ultimately, argues Robert Sutton of Stanford, a lot of what people look for in advice books, whether in business or any other realm, isn't so much advice as encouragement. And that can have value.

"There's value in mastering the obvious," he says. "If Jim Collins's impact is to get people to do stuff that they know they should do already - facing the hard truths or being selfless or whatever - I certainly don't think that's a bad thing."

Drake Bennett is the staff writer for Ideas. E-mail drbennett@globe.com.

© Copyright 2009 Globe Newspaper Company.