Matt Ridley has made a discovery. The natural selection that Darwin described in The Origin of Species is only a particular example of a universal process. As he tells us at the start of this book, Darwinism is “the special theory of evolution”. But there is a general theory of evolution, too, and it applies to society, money, technology, language, law, culture, music, violence, history, education, politics, God, morality. The general theory says that things do not stay the same; they change gradually but inexorably; they show “path dependence”; they show descent with modification; they show selective persistence.

In the course of the book’s 16 chapters, which deal with the evolution of everything from the internet to leadership, Ridley repeats this mantra many times: Darwin’s mechanism of selective survival resulting in cumulative complexity applies to human culture in all its aspects, too. Our habits and institutions, from language to cities, are constantly changing, and the mechanism of change turns out to be surprisingly Darwinian: it is gradual, undirected, mutational, inexorable, combinatorial, selective and “in some sense vaguely progressive”.

It’s curious that Ridley thinks this a new idea. There is nothing at all novel in theories of social evolution. I have a vivid memory of listening to the late FA Hayek, some 30 years ago, lecturing on what he called “the natural selection of religions” – a supposedly Darwinian process in which the religions that survive and spread are those that promote private property and market exchange and thereby support growing numbers of believers. I recall wondering how this account squared with the actual history of religion. The polytheistic cults of Greece and Rome didn’t die out in an incremental process of evolutionary decline; they were stamped out when the emperor Constantine converted to Christianity. If Tibet’s brand of Buddhism disappears from the country, or the Baha’i faith vanishes from Iran, the reason won’t be that these faiths suffer from any evolutionary disadvantage. It will be because state power has been used to destroy them.

Ideas of social evolution pass over the exercise of power – if it is mentioned at all, it’s only as an inconsequential detail in a vast process of evolutionary change. But what is it that supposedly drives evolution in society? The observation that “things do not stay the same” is scarcely a theory. Darwinian natural selection identified a mechanism and – once genes had been discovered, unknown to Darwin, by a Moravian monk called Gregor Mendel – a unit of selection. Theorists of social evolution in the past have never succeeded in specifying either of these. Despite a great deal of waffle about mutational and combinatorial processes and the like, neither does Ridley. There is no general theory of evolution.

What Ridley does is what proponents of social evolution have always done: he fastens on some of the events of the past few decades, suitably bolstered by selective bits of history, and turns these fleeting episodes into unstoppable trends. The 19th-century prophet of early capitalism, Herbert Spencer – described by Ridley in a long footnote as “one of the most unfairly traduced figures of history” – did this when he nominated mid-Victorian laissez-faire as the final state of human development towards which every society was evolving. Eccentric as he was – he wrote his lengthy tomes while wearing ear muffs to block out noise – Spencer was a more interesting thinker than his disciples. He lived long enough (born in 1820, he died in 1903) to see the world moving towards various types of statism rather than the minimal government he expected and desired. Spencer’s last years were spent in baffled gloom. On the basis of this bumptious and tediously repetitive tract, it’s difficult to imagine Ridley displaying a similar capacity for realistic observation or self-criticism.

Talk of social evolution mirrors the fads of the time. With capitalism in retreat in the 1930s, Spencer’s disciple Beatrice Webb nominated Stalin’s Soviet Union as the next stage of social evolution. In the 1970s, the American sociologist Daniel Bell was convinced that capitalism and socialism were evolving towards similar forms of post‑industrial bureaucracy. In the 90s, Francis Fukuyama had no doubt that “democratic capitalism” was evolving to become “the final form of human government”.

The fact that all of these predictions have been overturned by events doesn’t matter in the least. Social evolution isn’t a falsifiable theory but a succession of ideologies – in Ridley’s case, a dated and mechanical version of rightwing libertarianism. He tells us we must abandon the notion that society can be ordered by government from the top down – schooling and health care must be privatised, for example, so that better methods emerge in a process of innovation. There are some familiar problems with this anti-statist ideology. One is that the wholesale privatisation it recommends can only be implemented by the capture of government. So we’re back to power, the missing link in social evolution.

A more fundamental difficulty concerns the role of ideas and values. Why should anyone accept Ridley’s libertarianism? The ideologies of top-down government against which he rails are just as much a product of social evolution. After all, everything is a product of evolution, is it not? This includes the most pernicious ideas and beliefs. If evolutionary processes continue to produce antisemitism – a formidably adaptive meme, which today replicates itself right across the political spectrum – mustn’t this type of racism be “in some sense vaguely progressive”?

Relying on social evolution can be a risky business. Ridley’s career illustrates the point. The fifth Baron Ridley, educated at Eton and Oxford, Conservative member of the House of Lords, popular science writer, author of The Rational Optimist (2010) and Times columnist, he was chairman of Northern Rock (where his father had also been chairman) in the run-up to the financial crisis of 2007, when, after the first serious bank run in British history for more than 100 years, the bank collapsed and had to be nationalised. Ridley mentions the incident only to claim that it was the result of “intrusive regulation” and “top-down interference”. Considering that the financial crisis was preceded by far-reaching banking deregulation, this is risible as an account of events. But it also illustrates the incoherence of the idea of social evolution. Was the crisis that destroyed Northern Rock a part of the evolutionary process? If so, the evil regulations that according to Ridley caused the crisis must also be part of that process. Or was evolution embodied in the person of Ridley, when he resigned as chairman of the bank in October 2007?

If he was a more serious and reflective writer, Ridley might have given some thought to “Evolution and Ethics”, a lecture given in 1893 by TH Huxley, Darwin’s first and greatest disciple. Huxley was concerned to debunk the idea that evolution could teach anything of ethical importance:

Cosmic evolution may teach us how the good and evil tendencies of man have come about; but, in itself, it is incompetent to furnish any better reason why we what we call good is preferable to what we call evil than we had before … The fanatical individualism of our time attempts to apply the analogy of cosmic nature to society … Let us understand, once and for all, that the ethical progress of society depends, not on imitating the cosmic process, still less in running away from it, but in combating it.

Huxley aimed to stop Darwin’s theory of evolution being turned into an ideology. It was a noble effort, but unsuccessful. A confusion between evolution and human betterment has become embedded in our culture. Ridley recommends letting evolution take its course. But any halfway civilised morality involves interfering with evolutionary processes. There is nothing in any theory of evolution that tells us to protect the weak and help them live independent and worthwhile lives.

Ridley and many who think like him will respond that the evolution of society isn’t a blood-soaked struggle for survival; there is also empathy and mutual aid. Morality, they never tire of telling us, has evolved and is still evolving. But as Huxley pointed out more than a century ago, “the immoral sentiments have been no less evolved. The thief and the murderer follow nature just as much as the philanthropist”. Even if there were something that might be described as social evolution, the forces of barbarism would be as much a part of it as civilisation.

If The Evolution of Everything has any value, it’s as a demonstration that, outside of science, there isn’t much progress – even of the vaguer sort – in the history of thought. Bad ideas aren’t defeated by falsification, and they don’t fade away. As Ridley’s book shows, they simply recur, quite often in increasingly primitive and incoherent forms.

• To order a copy for £16 (RRP £20), go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min. p&p of £1.99.