In 1985, historian Barry Mehler had a dream. His research was taking him deep into the murky territory of academia’s extreme right wing. As he worked, he found his waking life beginning to soak into his subconscious, colouring his sleep. In his dream, his son, then two years old, was trapped in a runaway car hurtling down a hill. “The traffic is going in both directions, and I am in the middle of the road desperately waving my hands trying to stop the flow, in order to save the life of my son,” he tells me. “It’s a metaphor for how I felt.”

Mehler had been looking into what happened after the second world war to scientists who, during the conflict, had collaborated with the Nazis, were eugenicists or shared their racial worldview. “I was really focused on the ideological continuity between the old and the new,” he says. He learned that the fear of some kind of threat to the “white race” was still alive in some intellectual circles, and that there was a well-coordinated network of people who were attempting to bring these ideologies back into mainstream academia and politics.

Mehler, who is Jewish, understandably found all this disturbing. He immediately saw parallels between the far-right network of intellectuals and the rapid, devastating way in which eugenics research had been used in Nazi Germany, terrifying him with the possibility that the brutal atrocities of the past could happen once more. It was impossible not to imagine that the ideological heart behind them was still beating. “I felt like I was desperately trying to prevent this from happening again,” he says. “I thought that we were headed for more genocide.” His voice betrays an anxiety that political stability in even the strongest democracies sits on a precipice.

His fear is something I have begun to share. Mehler said of his relatives who survived the Holocaust: “They are prepared for things to cease to be normal very quickly.” His words ring in my ears. I never imagined I might live through times that could also make me feel this way, that could leave me so anxious for the future. Yet, here I am.

I grew up in south-east London – in an Indian-Punjabi household – not far from where the black teenager Stephen Lawrence was killed by white racist thugs in 1993 while waiting for a bus. He was only five years older than I was, and his murder left a mark on my generation. The old British National Party bookshop was in the same town as my secondary school. Racism was the backdrop to my teenage years. But then, for a brief moment, things looked as if they might be changing. My son was born five years ago, when British society seemed to be embracing diversity and multiculturalism. Barack Obama was president of the US. I dreamed that my baby might grow up in a better world than mine, perhaps even a post-racial one.

Things ceased to be normal. Far-right and anti-immigrant groups have once more become visible and powerful across Europe and the US. In Poland, nationalists march under the slogan “Pure Poland, white Poland”. In Italy, a rightwing leader rises to popularity on the promise to deport illegal immigrants and turn his back on refugees. White nationalists look to Russia under Vladimir Putin as a defender of “traditional” values.

‘Let them call you racist. Wear it as a badge of honour’ … Steve Bannon, former chief strategist to Donald Trump. Photograph: J Scott Applewhite/AP

In the German federal elections in 2017, Alternative für Deutschland won more than 12% of the vote. Last year, whistleblower Chris Wylie claimed that Cambridge Analytica, known to be closely linked to Donald Trump’s former chief strategist Steve Bannon, was using ideas of racial difference targeting African Americans to figure out how to stir up support among white conservatives in the 2014 mid-term elections. Since leaving the White House in 2017, Bannon has become a key figure for European far-right movements, and is now hoping to open an “alt-right” academy in an Italian monastery. This echoes “scientific racists” after the second world war, who, when they failed to find avenues in mainstream academia, simply created their own spaces and publications. The difference now is that, partly because of the internet, it’s so much easier for them to attract funding and support. In France in 2018, Bannon told far-right nationalists: “Let them call you racist, let them call you xenophobes, let them call you nativists. Wear it as a badge of honour.”

¶

I have spent the last few years investigating the tumorous growth of this brand of intellectual racism. Not the racist thugs who confront us in plain sight, but the well-educated ones in smart suits, the ones with power. And like Mehler, I’ve encountered tight networks, including academics at the world’s leading universities, who have sought to shape public debates around race and immigration, gently nudging into acceptability the view that “foreigners” are by their very nature a threat because we are fundamentally different.

Within this cabal are those who look to science to shore up their political views. Some describe themselves as “race realists”, reflecting how they see the scientific truth as being on their side (and because calling yourself a racist is still unpalatable, even to most racists). For them, there are innate biological differences between population groups, making entire nations, for instance, naturally smarter than others. These “biological facts” neatly explain the course of history and modern day inequality.

There is no gene that exists in all the members of one racial group and not another. We are all a product of ancient and recent migration

These so-called scholars are slippery – they use euphemisms, scientific-looking charts and arcane arguments. Riding the wave of populism around the world and harnessing the internet to communicate and publish, they have also become bolder. But as Mehler reminds me, they are not new.

This is a story that goes back to the birth of modern science. Race feels so tangible to us now, we have forgotten that racial classification was always quite arbitrary. In the 18th century, European scientists sifted people into human types, inventing such categories as Caucasian, but with scarce knowledge of how others lived. This is why, in the centuries that followed, nobody could ever quite pin down the thing we now call “race”. Some said there were three types, others four, five or more, even hundreds.

It was only towards the end of the 20th century that genetic data revealed that the human variation we see is not a matter of hard types but small and subtle gradations, each local community blending into the next. As much as 95% of the genetic difference in our species sits within the major population groups, not between them. Statistically, this means that, although I look nothing like the white British woman who lives upstairs, it’s possible for me to have more in common genetically with her than with my Indian-born neighbour.

We can’t pin down race biologically because it exists like an image in the clouds. When we define ourselves by colour, our eyes don’t consider that the genetic variants for light skin are found not only in Europe and east Asia, but also in some of the oldest human societies in Africa. Early hunter-gatherers in Europe had dark skin and blue eyes. There is no gene that exists in all the members of one racial group and not another. We are all, every one of us, a product of ancient and recent migration. We have always been in the melting pot together.

‘I never imagined I might live through times that could leave me so anxious for the future’ ... Angela Saini. Photograph: ✎ Gareth Phillips/The Observer

Race is the counter-proposition. In the history of race science, lines have been drawn across the world in many different ways. And what the lines meant changed in different eras. In the 19th century, a European scientist was unexceptional in thinking that white people were biologically superior to everyone else, just as he might assume that women were intellectually inferior. The power hierarchy had white men of European descent sitting at the top, and they conveniently wrote the scientific story of the human species around this assumption.

Because race science has always been innately political, it shouldn’t surprise us that prominent thinkers used science to defend slavery, colonialism, segregation and genocide. They imagined only Europe could have been the birthplace of modern science, that only the British could have built a railway in India. Some still imagine that white Europeans have a unique set of genetic qualities that propelled them to economic domination. They believe, as French president Nicolas Sarkozy said in 2007, that “the tragedy of Africa is that the African has not fully entered into history … there is neither room for human endeavour nor the idea of progress”.

¶

We have not left the past behind. There is a direct line from old ideologies to the rhetoric of the new. Mehler was one person who understood this because this was the line that he was carefully tracing.

After the second world war, race science gradually became taboo. But one of the key people to have kept his racial worldview intact, Mehler learned, was a shadowy figure called Roger Pearson, who is in his 90s today (he declined to speak to me). Pearson had been an officer in the British Indian army and then, in the 1950s, worked as managing director of a group of tea gardens in what was then known as East Pakistan, now Bangladesh. It was around this time that he began publishing newsletters, printed in India, exploring issues of race, science and immigration.

Very quickly, Mehler says, Pearson connected with like-minded thinkers all over the world. “He was beginning to institutionally organise the remnants of the prewar academic scholars who were doing work on eugenics and race. The war had disrupted all of their careers, and after the war they were trying to re‑establish themselves.” They included Nazi race scientist Otmar Freiherr von Verschuer, who before the war ended had run experiments on the body parts of murdered children sent to him from Auschwitz.

One of Pearson’s publications, the Northlander, described itself as a monthly review of “pan-Nordic affairs”, by which it meant matters of interest to white northern Europeans. Its first edition in 1958 complained about the illegitimate children born due to the stationing of “Negro” troops in Germany after the war, and about immigrants arriving in Britain from the West Indies. “Britain resounds to the sound and sight of primitive peoples and of jungle rhythms,” Pearson warned. “Why cannot we see the rot that is taking place in Britain herself?”

His newsletters relied on being able to reach out to marginal figures from all over the world, people whose views were generally unacceptable in the societies in which they lived. Within a couple of decades, Pearson ended up in Washington DC, establishing publications there too, including the Journal of Indo-European Studies in 1973 and the Journal of Social, Political and Economic Studies in 1975. In April 1982 a letter arrived for him from the White House, bearing the signature of President Ronald Reagan, praising him for promoting scholars who supported “a free enterprise economy, a firm and consistent foreign policy and a strong national defense”. Pearson used this endorsement to help raise funds and generate further support.

The public may have assumed that scientific racism was dead, but the racists were always active under the radar

Investigating race scientists at the same time was Keith Hurt, a softly spoken civil servant also in Washington, who was astonished to find “networks and associations of people that were attempting to keep alive a body of ideas that I had associated with at the very least the pre-civil rights movement” in the US, “and going back to the eugenics movement early in the last century. These ideas were still being developed and promulgated and promoted in discreet ways.”

“They had their own journals, their own publishing houses. They could review and comment upon each other’s work,” Mehler tells me. “It was almost like discovering this whole little world inside academia.” These were the people keeping scientific racism alive.

In May 1988, Mehler and Hurt published an article in the Nation, a progressive US weekly, about a professor of educational psychology at the University of Northern Iowa called Ralph Scott. Their report claimed that Scott had used funds provided by a wealthy segregationist under a pseudonym in 1976 and 1977 to organise a national anti-busing campaign (busing was a means of desegregating schools by transporting children from one area to another). Yet in 1985 the Reagan administration appointed Scott to the chair of the Iowa Advisory Committee to the US Commission on Civil Rights, a body tasked with enforcing antidiscrimination legislation. Even after taking up his influential post, Scott was writing for Pearson’s journal.

For those on the political extremes, it’s a waiting game. If they can survive and maintain their networks, it’s just a matter of time before an entry point opens up once more. The public may have assumed that scientific racism was dead, but the racists were always active under the radar. In The Bell Curve (1994), a notorious bestseller, US political scientist Charles Murray and psychologist Richard Herrnstein suggested that black Americans were less intelligent than white and Asian Americans. A review in the New York Review of Books observed that they referenced five articles from Mankind Quarterly, a journal co-founded by Pearson and Von Verschuer; they cited no fewer than 17 researchers who had contributed to the journal. Although The Bell Curve was widely panned (an article in American Behavioral Scientist described it as “fascist ideology”), Scientific American noted in 2017 that Murray was enjoying “an unfortunate resurgence”. Facing down protesters, he has been invited to give lectures on college campuses across the US.

Pearson’s Mankind Quarterly remains in print, published by a thinktank calling itself the Ulster Institute for Social Research, and joined by a slew of newer publications – some of them online – looking at similar topics. Recent articles in the journal include “racism in a world in which racial differences exist” and links between “solar radiation and IQ”. Immigration is a recurrent theme.

A poster for the Alternative für Deutschland party showing ‘The Slave Market’ by Jean-Leon Gerome. It reads ‘For Europe will not become ‘Eurabia’!’ Photograph: Clemens Bilan/EPA

In an email interview with its current editor, a biochemist called Gerhard Meisenberg working in Dominica, I was told matter-of-factly that there are racial differences in intelligence. “Jews tend to do very well, Chinese and Japanese pretty well, and Blacks and Hispanics not so well. The differences are small, but the most parsimonious explanation is that much and perhaps most of this is caused by genes,” he wrote. Meisenberg, like others in this network, condemns those who disagree – in essence, the mainstream scientific establishment – as irrational science-deniers blinded by political correctness.

“I think what we’re experiencing now is a much more threatening environment,” Hurt tells me. “We’re in a much worse situation than we were a couple of decades ago.” Online, these “race realists” have a fierce doggedness about them. Canadian self-styled philosopher Stefan Molyneux, whose YouTube channel has almost a million subscribers, delivers rhetorical monologues so long they seem designed to grind down viewers into submission. “Mother Nature’s the racist,” he has said. “I’m just shining the light.” Former guests on his show include one-time columnist Katie Hopkins and bestselling author Jordan Peterson.

What is worrying is that the thinkers who supply the material being brandished online have begun asserting a presence in other, more credible spaces. Earlier this month Noah Carl, an Oxford-educated social scientist, saw his prestigious fellowship at St Edmund’s College, Cambridge terminated after an investigation confirmed that he had collaborated “with a number of individuals who were known to hold extremist views”. A contributor to Mankind Quarterly, Carl had argued in another publication that, in the interests of free speech, he should be able to say that genes might “contribute to psychological differences between human populations”. According to a statement released by his college, his research activities and connections “demonstrated poor scholarship, promoted extreme rightwing views and incited racial and religious hatred”.

The editors of Mankind Quarterly, which has been called a “white supremacist journal”, have begun to assert a presence in other, more widely trusted scientific publications. Assistant editor Richard Lynn today sits on the editorial advisory board of Personality and Individual Differences, produced by Elsevier, one of the world’s largest scientific publishers, with the Lancet among its titles. In 2017, both Lynn and Meisenberg were listed on the editorial board of Intelligence, a psychology journal also published by Elsevier.

In late 2017, the editor-in-chief of Intelligence told me that their presence in his journal reflected his “commitment to academic freedom”. Yet after my inquiries to both him and Elsevier, I found that Lynn and Meisenberg had been quietly removed from the editorial board by the end of 2018.

What was once unacceptable is gaining a foothold under the banner of “academic freedom” and “diversity of opinion”. Those within academia who might have once kept controversial political opinions to themselves are crawling out of the woodwork. In the last few years, the journal Nature has even, in editorials, urged scientists to be careful, warning them about the rise of extremists looking to abuse their results.

White nationalist writer Jared Taylor, a contributor to Mankind Quarterly. Photograph: The Washington Post/Getty Images

One contributor to Mankind Quarterly who has become a major figure in the white supremacist movement is Yale-educated Jared Taylor, who founded the magazine American Renaissance in 1990. A phrase Taylor uses to defend racial segregation, borrowed from the zoologist Raymond Hall writing in the first issue of Mankind Quarterly, is that “two subspecies of the same species do not occur in the same geographic area”.

For those with a political ideology to sell, the 'science' is simply a way to project themselves as scholarly and objective

Taylor’s American Renaissance Foundation conferences were described by the late American anthropologist Robert Wald Sussman as “a gathering place for white supremacists, white nationalists, white separatists, neo-Nazis, Ku Klux Klan members, Holocaust deniers, and eugenicists”. Male attendees were expected to dress in business suits, to set themselves apart from the thuggish image most people associate with racists. Yet a visitor at one meeting reported that they didn’t “flinch from using terms such as ‘nigger’ and ‘chink’”.

For Hurt, it’s clear that the race science that thrived in Europe and the US at the start of the 20th century, manifesting itself most devastatingly in Nazi “racial hygiene”, had survived by the end of it and beyond. “The election of Trump made it impossible for many people to any longer overlook this stuff,” he says.

Once there was the backdrop of slavery and colonialism, then it was immigration and segregation, and now it is the rightwing agenda of this age. Nativism remains an issue, but there is also a backlash against greater efforts to promote racial equality in multicultural societies. For those with a political ideology, the “science” is simply a way to project themselves as scholarly and objective.

“Why do we still have race science, given everything that happened in the 20th century?” asks US anthropologist Jonathan Marks, who has worked to combat racism within academia. He answers his own question: “Because it is an important political issue. And there are powerful forces on the right that fund research into studying human differences with the goal of establishing those differences as a basis of inequalities.”

A common theme among today’s “race realists” is their belief that because biological race differences exist, diversity and equal opportunity programmes – designed to make society fairer – are doomed to fail. If an equal world isn’t being forged fast enough, it is due to a permanent natural roadblock created by the fact that, deep down, we’re not the same. “We have two nested fallacies here,” Marks continues. The first is that the human species comes packaged up in a small number of discrete races, each with their own different traits. “Second is the idea that there are innate explanations for political and economic inequality. What you’re saying is, inequality exists, but it doesn’t represent historical injustice. These guys are trying to manipulate science to construct imaginary boundaries to social progress.”

Until his death in 2012, one of the most prominent figures in this “race realist” network was Canadian psychologist John Philippe Rushton, whose name is still cited regularly in publications such as Mankind Quarterly. He earned a fawning obituary in the Globe and Mail, one of Canada’s most widely read papers, despite being notorious for his claim that brain and genital size were inversely related, making black people, he argued, better endowed but less intelligent than white people. Rushton felt “The Bell Curve didn’t go far enough”; his work has featured on Stefan Molyneux’s show.

When Rushton’s book Race, Evolution and Behaviour was published in 1995, psychologist David Barash was stirred to write in a review: “Bad science and virulent racial prejudice drip like pus from nearly every page of this despicable book.” Rushton had collected scraps of unreliable evidence in “the pious hope that by combining numerous little turds of variously tainted data, one can obtain a valuable result”. In reality, Barash wrote, “the outcome is merely a larger than average pile of shit”. In 2019, Rushton remains an intellectual icon for “race realists” and for members of the “alt-right”.

• Superior: The Return of Race Science is published by 4th Estate (£16.99). To order a copy go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £15, online orders only. Phone orders min p&p of £1.99.