Let's start with the invention of air conditioning. This is only one of approximately a zillion topics addressed by the science writer Steven Johnson during the course of lunch at an Italian restaurant in downtown Manhattan; some of the others include Darwinian evolution, the creation of YouTube, the curiously perfect population density of the Park Slope neighbourhood of Brooklyn, the French Revolution, the London cholera epidemic of 1854, the first computer, The Wire, and why 9/11 wasn't prevented. But air conditioning provides a useful way to introduce Johnson's current overarching obsession – the mysterious question of where good ideas come from – because it encapsulates how we generally like to think about inventors and inventions. One night in 1902, an ambitious young American engineer named Willis Carrier was waiting for a train, watching fog roll in across the platform, when he had a sudden flash of insight: he could exploit the principle of fog to cool buildings. He patented the idea, protected it fiercely, put his new invention into production, and made a fortune. In 2007, the still-surviving Carrier Corporation generated sales worth $15bn. As eureka moments go, even Archimedes might have had to concede that Carrier's was impressive.

For Johnson, though, what's really interesting about that story is how unusual it is: although the eureka moment is such a cliche, big new ideas almost never get born like that. "It's weird," says Johnson, ignoring his carpaccio of beef, and launching into an engaging quasi-lecture that effortlessly expands to dominate most of the rest of the meal, "but innovation is one of those cases where the defining image, all the rhetoric and all the assumptions about how it happens, turn out to be completely backward. It's very, very rare to find cases where somebody on their own, working alone, in a moment of sudden clarity has a great breakthrough that changes the world. And yet there seems to be this bizarre desire to tell the story that way."

Johnson's new book, Where Good Ideas Come From, is an attempt to sketch out a radically different theory, which is in keeping with his personal policy of upending received wisdom in as all-encompassing a fashion as possible: one of his most popular works to date was entitled Everything Bad is Good for You: How Today's Popular Culture is Actually Making Us Smarter. His latest book should further entrench his position in today's army of commercially successful, male, fortysomething writers of sweeping, occasionally grandiose ideas books (commander-in-chief, Malcolm Gladwell; high-ranking generals, Clay Shirky and the authors of Freakonomics). Except that, if anything, it's even more audacious, since in principle he seeks to explain the origin of all their ideas, too. Oh, and the origin of life. And some other things.

At the core of his alternative history is the notion of the "adjacent possible", one of those ideas that seems, at first, like common sense, then gradually reveals itself as an entirely new way of looking at almost everything. Coined by the biologist Stuart Kauffman, it refers to the fact that at any given time – in science and technology, but perhaps also in culture and politics – only certain kinds of next steps are feasible. "The history of cultural progress," Johnson writes, "is, almost without exception, a story of one door leading to another door, exploring the palace one room at a time."

Think of playing chess: at any point in the game, several ingenious moves may be possible, but countless others won't be. Likewise with inventions: the printing press was only possible – and perhaps only thinkable – once moveable type, paper and ink all existed. YouTube, when it was launched in 2005, was a brilliant idea; had it been launched in 1995, before broadband and cheap video cameras were widespread, it would have been a terrible one. Or take culture: to 1950s viewers, Johnson argues, complex TV shows such as Lost or The Wire would have been borderline incomprehensible, like some kind of avant-garde art, because certain ways of engaging with the medium hadn't yet been learned. And all this applies, too, to the most basic innovation: life itself. At some point, back in the primordial soup, a bunch of fatty acids gave rise to a cell membrane, which made possible the simplest organisms, and so on. What those acids couldn't do was spontaneously form into a fish, or a mouse: it wasn't part of their adjacent possible.

If this seems completely obvious, consider, Johnson says, how it explains the otherwise spooky phenomenon of the "multiple" – the way certain inventions or discoveries occur in several places simultaneously, apparently by chance. Sun-spots were discovered in 1611 by four different scientists in four different countries; electrical batteries were invented twice, separately, one year apart. (Similar things happened in the earliest days of the steam engine and telephone.) People have tried to explain this using vague terms such as the "zeitgeist", or of certain ideas just being "in the air". But there's a simpler possibility, which is that the innovation in question had simply become part of the adjacent possible. Good ideas, as Johnson puts it, "are built out of a collection of existing parts", both literally and metaphorically speaking. Take the isolation of oxygen as a component of air, which was another multiple. It couldn't have happened before the invention of ultra-sensitive weighing scales. But it also couldn't have happened before the birth of the idea that air is something, rather than nothing, and that it might be made up of gases.

What all this means, in practical terms, is that the best way to encourage (or to have) new ideas isn't to fetishise the "spark of genius", to retreat to a mountain cabin in order to "be creative", or to blabber interminably about "blue-sky", "out-of-the-box" thinking. Rather, it's to expand the range of your possible next moves – the perimeter of your potential – by exposing yourself to as much serendipity, as much argument and conversation, as many rival and related ideas as possible; to borrow, to repurpose, to recombine. This is one way of explaining the creativity generated by cities, by Europe's 17th-century coffee-houses, and by the internet. Good ideas happen in networks; in one rather brain-bending sense, you could even say that "good ideas are networks". Or as Johnson also puts it: "Chance favours the connected mind."

Another surprising truth about big ideas: even when they seem to be individual flashes of genius, they don't happen in a flash – though the people who have them often subsequently claim that they did. Charles Darwin always said that the theory of natural selection occurred to him on 28 September 1838 while he was reading Thomas Malthus's essay on population; suddenly, the mechanism of evolution seemed blindingly straightforward. ("How incredibly stupid not to think of that," Darwin's great supporter Thomas Huxley was supposed to have said on first hearing the news.) Yet Darwin's own notebooks reveal that the theory was forming clearly in his mind more than a year beforehand: it wasn't a flash of insight, but what Johnson calls a "slow hunch". And on the morning after his alleged eureka moment, was Darwin feverishly contemplating the implications of his breakthrough? Nope: he busied himself with some largely unconnected ruminations on the sexual curiosity of primates.

Johnson, who lives with his wife Alexa Robinson and their three sons in Brooklyn, has written seven well-received books and launched three notable web startups, including the pioneering web magazine Feed, gives around 50 lectures a year, and writes plenty of high-profile opinion columns, all of which he has accomplished by the not-exactly-ancient age of 42. (While we're on the topic, he also has an enormous 1.4 million followers on Twitter, far in excess of superstars from Kanye West to Sarah Palin to Hugo Chávez to the Dalai Lama.)

I'm not sure whether it heightens or quells that feeling to learn that part of his secret isn't soul-destroying workaholism, or amphetamines, but rather working methods that directly echo his arguments about creativity. For many years he has collected thousands of short quotes, along with his own writing, in a computer program called DevonThink, which uses artificial intelligence to identify connections between them. "I can put a quote in and ask it to show me things that are related to this – which is literally a way of exploring the adjacent possible," he says. "Half the time it'll suggest something completely irrelevant, but amid that noise there's always some crazy little new connection that I hadn't thought of."

Unlikely connections are a characteristic of Johnson's business ventures, too: the hyperlocal news site Outside.in grew from thinking about neighbourhoods as networks because of their role in The Ghost Map, his book about the spread and eventual defeat of London's 19th-century cholera epidemic. This was eventually traced to a baby's nappy that contaminated a well on Broad Street – not the most auspicious beginning, you might think, for a site full of restaurant recommendations and reviews of yoga classes. But such is the power of serendipity.

A certain kind of businessperson, I suspect, will buy Where Good Ideas Come From in order to learn to how to come up with a killer business idea, bring it to market, and clean up financially. They may find themselves slightly alarmed, therefore, by a sequence of striking graphics in which Johnson demonstrates that the vast majority of major innovations since 1800 have come from outside the free market – from universities and other environments where profit wasn't the overwhelming motivation. The urge to hoard, protect and directly profit from good ideas can work against the sharing-and-recombining ethos that the adjacent possible demands. And it's often the case that those who do attain vast wealth have done so by finding ways to exploit the creativity of the non-market world. Facebook's Mark Zuckerberg is so rich today only because Tim Berners-Lee developed the web as a non-profit venture. (And a non-profit venture, incidentally, that had no eureka moment either. Johnson quotes Berners-Lee as saying that interviewers are always frustrated when he admits he never experienced one.)

There's a sense, of course, in which this way of looking at the world is itself a "multiple": Johnson's argument has plenty in common with Gladwell's Outliers, and with books by Shirky and others on the power of amateur creativity, harnessed by the web. But Johnson distances himself from the most extreme version of that view, which is that individual gifts count for nothing – that every blog-commenter has as much of a contribution to make as every Booker winner, or that the "hive mind" is always in the right. It's not that creative individuals don't matter; it's that connectedness makes us more creative.

Politically speaking, none of this is rightwing in any traditional sense: it's a rejection of America's cherished belief in the primacy of individualism and free markets. But the focus on grassroots connectedness isn't really left-wing, either, "if leftwing means you're enamoured of large state interventions in society". A philosophy of innovation that rejected both of those "might be called anarchism", Johnson says, then looks slightly surprised at himself. "Huh. I'm not sure I'd want to be associated with that word, though."