In January 2016, the rapper BoB took to Twitter to tell his fans that the Earth is really flat. “A lot of people are turned off by the phrase ‘flat earth’,” he acknowledged, “but there’s no way u can see all the evidence and not know … grow up.” At length the astrophysicist Neil deGrasse Tyson joined in the conversation, offering friendly corrections to BoB’s zany proofs of non-globism, and finishing with a sarcastic compliment: “Being five centuries regressed in your reasoning doesn’t mean we all can’t still like your music.”

Actually, it’s a lot more than five centuries regressed. Contrary to what we often hear, people didn’t think the Earth was flat right up until Columbus sailed to the Americas. In ancient Greece, the philosophers Pythagoras and Parmenides had already recognised that the Earth was spherical. Aristotle pointed out that you could see some stars in Egypt and Cyprus that were not visible at more northerly latitudes, and also that the Earth casts a curved shadow on the moon during a lunar eclipse. The Earth, he concluded with impeccable logic, must be round.

The flat-Earth view was dismissed as simply ridiculous – until very recently, with the resurgence of apparently serious flat-Earthism on the internet. An American named Mark Sargent, formerly a professional videogamer and software consultant, has had millions of views on YouTube for his Flat Earth Clues video series. (“You are living inside a giant enclosed system,” his website warns.) The Flat Earth Society is alive and well, with a thriving website. What is going on?

Many ideas have been brilliantly upgraded or repurposed for the modern age, and their revival seems newly compelling. Some ideas from the past, on the other hand, are just dead wrong and really should have been left to rot. When they reappear, what is rediscovered is a shambling corpse. These are zombie ideas. You can try to kill them, but they just won’t die. And their existence is a big problem for our normal assumptions about how the marketplace of ideas operates.

The phrase “marketplace of ideas” was originally used as a way of defending free speech. Just as traders and customers are free to buy and sell wares in the market, so freedom of speech ensures that people are free to exchange ideas, test them out, and see which ones rise to the top. Just as good consumer products succeed and bad ones fail, so in the marketplace of ideas the truth will win out, and error and dishonesty will disappear.

There is certainly some truth in the thought that competition between ideas is necessary for the advancement of our understanding. But the belief that the best ideas will always succeed is rather like the faith that unregulated financial markets will always produce the best economic outcomes. As the IMF chief Christine Lagarde put this standard wisdom laconically in Davos: “The market sorts things out, eventually.” Maybe so. But while we wait, very bad things might happen.

Zombies don’t occur in physical marketplaces – take technology, for example. No one now buys Betamax video recorders, because that technology has been superseded and has no chance of coming back. (The reason that other old technologies, such as the manual typewriter or the acoustic piano, are still in use is that, according to the preferences of their users, they have not been superseded.) So zombies such as flat-Earthism simply shouldn’t be possible in a well‑functioning marketplace of ideas. And yet – they live. How come?

One clue is provided by economics. It turns out that the marketplace of economic ideas itself is infested with zombies. After the 2008 financial crisis had struck, the Australian economist John Quiggin published an illuminating work called Zombie Economics, describing theories that still somehow shambled around even though they were clearly dead, having been refuted by actual events in the world. An example is the notorious efficient markets hypothesis, which holds, in its strongest form, that “financial markets are the best possible guide to the value of economic assets and therefore to decisions about investment and production”. That, Quiggin argues, simply can’t be right. Not only was the efficient markets hypothesis refuted by the global meltdown of 2007–8, in Quiggin’s view it actually caused it in the first place: the idea “justified, and indeed demanded, financial deregulation, the removal of controls on international capital flows, and a massive expansion of the financial sector. These developments ultimately produced the global financial crisis.”

Even so, an idea will have a good chance of hanging around as a zombie if it benefits some influential group of people. The efficient markets hypothesis is financially beneficial for bankers who want to make deals unencumbered by regulation. A similar point can be made about the privatisation of state-owned industry: it is seldom good for citizens, but is always a cash bonanza for those directly involved.

The marketplace of ideas, indeed, often confers authority through mere repetition – in science as well as in political campaigning. You probably know, for example, that the human tongue has regional sensitivities: sweetness is sensed on the tip, saltiness and sourness on the sides, and bitter at the back. At some point you’ve seen a scientific tongue map showing this – they appear in cookery books as well as medical textbooks. It’s one of those nice, slightly surprising findings of science that no one questions. And it’s rubbish.

A fantasy map of a flat earth. Photograph: Antar Dayal/Getty Images/Illustration Works

As the eminent professor of biology, Stuart Firestein, explained in his 2012 book Ignorance: How it Drives Science, the tongue-map myth arose because of a mistranslation of a 1901 German physiology textbook. Regions of the tongue are just “very slightly” more or less sensitive to each of the four basic tastes, but they each can sense all of them. The translation “considerably overstated” the original author’s claims. And yet the mythical tongue map has endured for more than a century.

One of the paradoxes of zombie ideas, though, is that they can have positive social effects. The answer is not necessarily to suppress them, since even apparently vicious and disingenuous ideas can lead to illuminating rebuttal and productive research. Few would argue that a commercial marketplace needs fraud and faulty products. But in the marketplace of ideas, zombies can actually be useful. Or if not, they can at least make us feel better. That, paradoxically, is what I think the flat-Earthers of today are really offering – comfort.

Today’s rejuvenated flat-Earth philosophy, as promoted by rappers and YouTube videos, is not simply a recrudescence of pre-scientific ignorance. It is, rather, the mother of all conspiracy theories. The point is that everyone who claims the Earth is round is trying to fool you, and keep you in the dark. In that sense, it is a very modern version of an old idea.

As with any conspiracy theory, the flat-Earth idea is introduced by way of a handful of seeming anomalies, things that don’t seem to fit the “official” story. Have you ever wondered, the flat-Earther will ask, why commercial aeroplanes don’t fly over Antarctica? It would, after all, be the most direct route from South Africa to New Zealand, or from Sydney to Buenos Aires – if the Earth were round. But it isn’t. There is no such thing as the South Pole, so flying over Antarctica wouldn’t make any sense. Plus, the Antarctic treaty, signed by the world’s most powerful countries, bans any flights over it, because something very weird is going on there. So begins the conspiracy sell. Well, in fact, some commercial routes do fly over part of the continent of Antarctica. The reason none fly over the South Pole itself is because of aviation rules that require any aircraft taking such a route to have expensive survival equipment for all passengers on board – which would obviously be prohibitive for a passenger jet.

OK, the flat-Earther will say, then what about the fact that photographs taken from mountains or hot-air balloons don’t show any curvature of the horizon? It is perfectly flat – therefore the Earth must be flat. Well, a reasonable person will respond, it looks flat because the Earth, though round, is really very big. But photographs taken from the International Space Station in orbit show a very obviously curved Earth.

And here is where the conspiracy really gets going. To a flat-Earther, any photograph from the International Space Station is just a fake. So too are the famous photographs of the whole round Earth hanging in space that were taken on the Apollo missions. Of course, the Moon landings were faked too. This is a conspiracy theory that swallows other conspiracy theories whole. According to Mark Sargent’s “enclosed world” version of the flat-Earth theory, indeed, space travel had to be faked because there is actually an impermeable solid dome enclosing our flat planet. The US and USSR tried to break through this dome by nuking it in the 1950s: that’s what all those nuclear tests were really about.

Flat-Earthers regard as fake any photographs of the Earth that were taken on the Apollo missions Photograph: Alamy

The intellectual dynamic here, is one of rejection and obfuscation. A lot of ingenuity evidently goes into the elaboration of modern flat-Earth theories to keep them consistent. It is tempting to suppose that some of the leading writers (or, as fans call them, “researchers”) on the topic are cynically having some intellectual fun, but there are also a lot of true believers on the messageboards who find the notion of the “globist” conspiracy somehow comforting and consonant with their idea of how the world works. You might think that the really obvious question here, though, is: what purpose would such an incredibly elaborate and expensive conspiracy serve? What exactly is the point?

It seems to me that the desire to believe such stuff stems from a deranged kind of optimism about the capabilities of human beings. It is a dark view of human nature, to be sure, but it is also rather awe-inspiring to think of secret agencies so single-minded and powerful that they really can fool the world’s population over something so enormous. Even the pro-Brexit activists who warned one another on polling day to mark their crosses with a pen so that MI5 would not be able to erase their votes, were in a way expressing a perverse pride in the domination of Britain’s spookocracy. “I literally ran out of new tin hat topics to research and I STILL wouldn’t look at this one without embarrassment,” confesses Sargent on his website, “but every time I glanced at it there was something unresolved, and once I saw the near perfection of the whole plan, I was hooked.” It is rather beautiful. Bonkers, but beautiful. As the much more noxious example of Scientology also demonstrates, it is all too tempting to take science fiction for truth – because narratives always make more sense than reality.

We know that it’s a good habit to question received wisdom. Sometimes, though, healthy scepticism can run over into paranoid cynicism, and giant conspiracies seem oddly consoling. One reason why myths and urban legends hang around so long seems to be that we like simple explanations – such as that immigrants are to blame for crumbling public services – and are inclined to believe them. The “MMR causes autism” scare perpetrated by Andrew Wakefield, for example, had the apparent virtue of naming a concrete cause (vaccination) for a deeply worrying and little-understood syndrome (autism). Years after it was shown that there was nothing to Wakefield’s claims, there is still a strong and growing “anti-vaxxer” movement, particularly in the US, which poses a serious danger to public health. The benefits of immunisation, it seems, have been forgotten.

The yearning for simple explanations also helps to account for the popularity of outlandish conspiracy theories that paint a reassuring picture of all the world’s evils as being attributable to a cabal of supervillains. Maybe a secret society really is running the show – in which case the world at least has a weird kind of coherence. Hence, perhaps, the disappointed amazement among some of those who had not expected their protest votes for Brexit to count.

And what happens when the world of ideas really does operate as a marketplace? It happens to be the case that many prominent climate sceptics have been secretly funded by oil companies. The idea that there is some scientific controversy over whether burning fossil fuels has contributed in large part to the present global warming (there isn’t) is an idea that has been literally bought and sold, and remains extraordinarily successful. That, of course, is just a particularly dramatic example of the way all western democracies have been captured by industry lobbying and party donations, in which friendly consideration of ideas that increase the profits of business is simply purchased, like any other commodity. If the marketplace of ideas worked as advertised, not only would this kind of corruption be absent, it would be impossible in general for ideas to stay rejected for hundreds or thousands of years before eventually being revived. Yet that too has repeatedly happened.

As some philosophers have argued, it is not entirely impossible that God created the whole universe

While the return of flat-Earth theories is silly and rather alarming, meanwhile, it also illustrates some real and deep issues about human knowledge. How, after all, do you or I know that the Earth really is round? Essentially, we take it on trust. We may have experienced some common indications of it ourselves, but we accept the explanations of others. The experts all say the Earth is round; we believe them, and get on with our lives. Rejecting the economic consensus that Brexit would be bad for the UK, Michael Gove said that the British public had had enough of experts (or at least of experts who lurked in acronymically named organisations), but the truth is that we all depend on experts for most of what we think we know.

The second issue is that we cannot actually know for sure that the way the world appears to us is not actually the result of some giant conspiracy or deception. The modern flat-Earth theory comes quite close to an even more all-encompassing species of conspiracy theory. As some philosophers have argued, it is not entirely impossible that God created the whole universe, including fossils, ourselves and all our (false) memories, only five minutes ago. Or it might be the case that all my sensory impressions are being fed to my brain by a clever demon intent on deceiving me (Descartes) or by a virtual-reality program controlled by evil sentient artificial intelligences (The Matrix).

The resurgence of flat-Earth theory has also spawned many web pages that employ mathematics, science, and everyday experience to explain why the world actually is round. This is a boon for public education. And we should not give in to the temptation to conclude that belief in a conspiracy is prima facie evidence of stupidity. Evidently, conspiracies really happen. Members of al-Qaida really did conspire in secret to fly planes into the World Trade Center. And, as Edward Snowden revealed, the American and British intelligence services really did conspire in secret to intercept the electronic communications of millions of ordinary citizens. Perhaps the most colourful official conspiracy that we now know of happened in China. When the half-millennium-old Tiananmen Gate was found to be falling down in the 1960s, it was secretly replaced, bit by bit, with an exact replica, in a successful conspiracy that involved nearly 3,000 people who managed to keep it a secret for years.

Indeed, a healthy openness to conspiracy may be said to underlie much honest intellectual inquiry. This is how the physicist Frank Wilczek puts it: “When I was growing up, I loved the idea that great powers and secret meanings lurk behind the appearance of things.” Newton’s grand idea of an invisible force (gravity) running the universe was definitely a cosmological conspiracy theory in this sense. Yes, many conspiracy theories are zombies – but so is the idea that conspiracies never happen.

‘When the half-millennium-old Tiananmen Gate was found to be falling down in the 1960s, it was secretly replaced, bit by bit, with an exact replica’ Photograph: Kevin Frayer/Getty Images

Things are better, one assumes, in the rarefied marketplace of scientific ideas. There, the revered scientific journals have rigorous editorial standards. Zombies and other market failures are thereby prevented. Not so fast. Remember the tongue map. It turns out that the marketplace of scientific ideas is not perfect either.

The scientific community operates according to the system of peer review, in which an article submitted to a journal will be sent out by the editor to several anonymous referees who are expert in the field and will give a considered view on whether the paper is worthy of publication, or will be worthy if revised. (In Britain, the Royal Society began to seek such reports in 1832.) The barriers to entry for the best journals in the sciences and humanities mean that – at least in theory – it is impossible to publish clownish, evidence-free hypotheses.

But there are increasing rumblings in the academic world itself that peer review is fundamentally broken. Even that it actively suppresses good new ideas while letting through a multitude of very bad ones. “False positives and exaggerated results in peer-reviewed scientific studies have reached epidemic proportions in recent years,” reported Scientific American magazine in 2011. Indeed, the writer of that column, a professor of medicine named John Ioannidis, had previously published a famous paper titled Why Most Published Research Findings Are False. The issues, he noted, are particularly severe in healthcare research, in which conflicts of interest arise because studies are funded by large drug companies, but there is also a big problem in psychology.

Take the widely popularised idea of priming. In 1996, a paper was published claiming that experimental subjects who had been verbally primed to think of old age by being made to think about words such as bingo, Florida, grey, and wrinkles subsequently walked more slowly when they left the laboratory than those who had not been primed. It was a dazzling idea, and led to a flurry of other findings that priming could affect how well you did on a quiz, or how polite you were to a stranger. In recent years, however, researchers have become suspicious, and have not been able to generate the same findings as many of the early studies. This is not definitive proof of falsity, but it does show that publication in a peer-reviewed journal is no guarantee of reliability. Psychology, some argue, is currently going through a crisis in replicability, which Daniel Kahneman has called a looming “train wreck” for the field as a whole.

Could priming be a future zombie idea? Well, most people think it unlikely that all such priming effects will be refuted, since there is now such a wide variety of studies on them. The more interesting problem is to work out what scientists call the idea’s “ecological validity” – that is, how well do the effects translate from the artificial simplicity of the lab situation to the ungovernable messiness of real life? This controversy in psychology just shows science working as it should – being self-correcting. One marketplace-of-ideas problem here, though, is that papers with surprising and socially intriguing results will be described throughout the media, and lauded as definitive evidence in popularising books, as soon as they are published, and long before awkward second questions begin to be asked.

It would be sensible, for a start, for us to make the apparently trivial rhetorical adjustment from the popular phrase “studies show …” and limit ourselves to phrases such as “studies suggest” or “studies indicate”. After all, “showing” strongly implies proving, which is all too rare an activity outside mathematics. Studies can always be reconsidered. That is part of their power.

Nearly every academic inquirer I talked to while researching this subject says that the interface of research with publishing is seriously flawed. Partly because the incentives are all wrong – a “publish or perish” culture rewards academics for quantity of published research over quality. And partly because of the issue of “publication bias”: the studies that get published are the ones that have yielded hoped-for results. Studies that fail to show what they hoped for end up languishing in desk drawers.

One reform suggested by many people to counteract publication bias would be to encourage the publication of more “negative findings” – papers where a hypothesis was not backed up by the experiment performed. One problem, of course, is that such findings are not very exciting. Negative results do not make headlines. (And they sound all the duller for being called “negative findings”, rather than being framed as positive discoveries that some ideas won’t fly.)

The publication-bias issue is even more pressing in the field of medicine, where it is estimated that the results of around half of all trials conducted are never published at all, because their results are negative. “When half the evidence is withheld,” writes the medical researcher Ben Goldacre, “doctors and patients cannot make informed decisions about which treatment is best.”Accordingly, Goldacre has kickstarted a campaigning group named AllTrials to demand that all results be published.

When lives are not directly at stake, however, it might be difficult to publish more negative findings in other areas of science. One idea, floated by the Economist, is that “Journals should allocate space for ‘uninteresting’ work, and grant-givers should set aside money to pay for it.” It sounds splendid, to have a section in journals for tedious results, or maybe an entire journal dedicated to boring and perfectly unsurprising research. But good luck getting anyone to fund it.

The good news, though, is that some of the flaws in the marketplace of scientific ideas might be hidden strengths. It’s true that some people think peer review, at its glacial pace and with its bias towards the existing consensus, works to actively repress new ideas that are challenging to received opinion. Notoriously, for example, the paper that first announced the invention of graphene – a way of arranging carbon in a sheet only a single atom thick – was rejected by Nature in 2004 on the grounds that it was simply “impossible”. But that idea was too impressive to be suppressed; in fact, the authors of the graphene paper had it published in Science magazine only six months later. Most people have faith that very well-grounded results will find their way through the system. Yet it is right that doing so should be difficult. If this marketplace were more liquid and efficient, we would be overwhelmed with speculative nonsense. Even peremptory or aggressive dismissals of new findings have a crucial place in the intellectual ecology. Science would not be so robust a means of investigating the world if it eagerly embraced every shiny new idea that comes along. It has to put on a stern face and say: “Impress me.” Great ideas may well face a lot of necessary resistance, and take a long time to gain traction. And we wouldn’t wish things to be otherwise.

In many ways, then, the marketplace of ideas does not work as advertised: it is not efficient, there are frequent crashes and failures, and dangerous products often win out, to widespread surprise and dismay. It is important to rethink the notion that the best ideas reliably rise to the top: that itself is a zombie idea, which helps entrench powerful interests. Yet even zombie ideas can still be useful when they motivate energetic refutations that advance public education. Yes, we may regret that people often turn to the past to renew an old theory such as flat-Earthism, which really should have stayed dead. But some conspiracies are real, and science is always engaged in trying to uncover the hidden powers behind what we see. The resurrection of zombie ideas, as well as the stubborn rejection of promising new ones, can both be important mechanisms for the advancement of human understanding.

Main picture: George Diebold/Getty Images

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.

This article is adapted from Rethink: the Surprising History of Ideas, which is published by Random House on 30 June