Last week, Sandeep and Reena Mander were denied the chance to adopt a child. It was not because their local council, Windsor and Maidenhead, thought that they would not have provided a loving family home. Nor because there were no children to adopt. It is rather that the Manders are of Indian Sikh heritage – though both born in Britain – and the only children needing adoption were white. “They took the colour of our skin as the overriding reason not to progress with the application,” Mr Mander said.

Many have seen in the attitude of Windsor and Maidenhead council a straightforward case of racism. The councillors are predominantly white, and overwhelmingly Conservative. They “appear not to appreciate diversity”, claimed Narinderjit Singh, general secretary of the Sikh Federation (UK).

The problem runs much deeper, however, than the attitudes of the undiverse members of one local council. It speaks to a broader confusion about the relationship between race and culture; a confusion that afflicts anti-racists as much as racists. Few people these days claim that whites and Indians are racially incompatible. But many argue that whites – or, more euphemistically, “Europeans” or “westerners” – and Indians (and blacks and Chinese and countless others) belong to distinct cultures and possess discrete identities. Many argue, too, that especially for children, it is important not to undermine their sense of identity or create confusion about their cultural attachments.

Windsor and Maidenhead council has refused to comment on an “ongoing case”, so it is difficult to know precisely its line of reasoning. Adopt Berkshire, the council’s adoption agency, suggests that it would attempt to “identify appropriate prospective adopters for each child who reflect the child’s culture and religion of heritage”. It is plausible the council imagines that to be white is to belong to a particular culture, and that non-whites belong to other cultures. A white child can only be brought up by white parents because, otherwise, he or she would grow up in the “wrong” culture.

Historically, racism has played a major role in shaping adoption practices

Historically, racism has played a major role in shaping adoption practices. From the days when miscegenation was seen as a social sin, and in some countries a crime, racists have baulked at the idea of non-white parents bringing up white children. At the same time, many non-white children were torn from their parents and given up for adoption to white families, a practice promoted as a civilising mission. Perhaps the most shocking example came not from Britain but Australia, where, between 1910 and 1969, thousands of Indigenous children – the “Stolen Generations” – were forcibly taken from their families to be brought up in white households.

More recently, though, the kind of attitude that seems to have swayed Windsor and Maidenhead council has been promoted by anti-racists as much as by racists. In Britain, the pushback against transracial adoption began in the 1980s. In 1983, the Association of Black Social Workers and Allied Professions (Abswap) condemned transracial adoption as “a microcosm of the oppression of black people” and a “new form of slave trade”. What a black child requires above all is “positive black identity”. The black community could not “maintain any dignity in this country … if black children are taken away from their parents and reared exclusively by another race”.

Stripped of its overheated rhetoric, the Abswap view can be seen partly as a response to racism, and to the long, racist history of adoption practices, and partly as an assertion of ethnic pride by a community against which there existed intense hostility.

But, as the sociologist Paul Gilroy pointed out, it is a perspective that, far from challenging racism, simply appropriates the core of the racial thinking. In his seminal 1987 book about race and culture in Thatcherite Britain, There Ain’t No Black in the Union Jack, Gilroy observed that “the definition of race which informs these arguments elides the realms of culture and biology in the same way as the volkish new right preoccupations with ‘kith and kin’”.

Traditionally, a “race” was seen as a group of human beings linked by a set of fundamental characteristics unique to it. In the postwar world, this concept disintegrated. Racial categories were shown to possess little scientific validity, while, after the Holocaust, ideas of racial inferiority and superiority became impossible openly to espouse.

For the left, too, culture has become the key component of its version of identity politics

But if old-fashioned racial science was buried in the postwar world, many of the assumptions of racial thinking survived. It was not just that western societies remained deeply racist. It was also that the fundamental ideas that once sustained racial thinking – that humanity could be divided into discrete groups, each with a set of unique characteristics that shaped an individual’s identity – continued to resonate. These ideas came to be recast eventually not in the language of biology, but of culture.

This new language of culture has taken both rightwing and leftwing forms. The so-called “New Right” that emerged in the 1970s explicitly looked to culture as a replacement for race. Every nation and people, its proponents argued, had its own culture that had to be protected against foreigners.

For the left, too, culture has become the key component of its version of identity politics. Minority groups – African Americans, Indigenous Australians, Muslims or gays – are seen as possessing distinct cultures, identities and ways of thinking. To confront racism and oppression, many argue, requires a defence of each group’s identity – a mirror image of the New Right argument.

Once, culture was seen as providing the tools with which to open up and transform the world. Today, many regard it more as a protective wall to shield its members and keep out unwanted visitors.

Take the immigration debate. Once, hostility to immigration was rooted in racial antipathy – “yellow peril” or “black invasion”. Today, it is more often expressed in terms of cultural differences.

From the other side of the political spectrum come arguments about “cultural appropriation” – the “unauthorised” use by members of a dominant culture of the products of a less privileged culture. The singer Katy Perry recently apologised for wearing her hair in cornrows, regarded by many as a distinctively black hairstyle.

Open Casket by Dana Schutz at the Whitney Museum of American Art in New York. This painting depicts Emmett Till, murdered by a white mob in 1955. Many objected to a white painter showing a traumatic moment in black history. Photograph: Alina Heineke/AP

Other cases are less trivial. When New York’s Whitney Museum displayed Dana Schutz’s painting of the mutilated corpse of Emmett Till, an African American teenager lynched by a white mob in Mississippi in 1955, many objected to a white painter depicting a traumatic moment in black history. Some called for its destruction. When I wrote an article for the New York Times criticising such arguments about cultural appropriation, I was accused of “defending white supremacy”. Such is the confusion about race and culture that criticising the volkish notion of culture that underlies white supremacy is now seen as defending it.

We should defend the Manders’ right to adopt a white child, the right of Muslims to settle in the west, the right of Dana Schutz to depict a black subject. Above all we need to rethink the way we imagine race, culture and the relationship between the two.