Architecture’s endless war between the classical and the modernist has leaked into politics again. This month saw the emergence of a draft of Making Federal Buildings Beautiful Again, an executive order from the desk of Donald Trump, which would mandate a return to the style chosen by the US’s founding fathers for its associations with “democratic Athens and republican Rome”. New buildings would be designed “to endure for centuries”. By banning “plain ugly” modernism, the document suggests, the administration can restore the US’s unsullied past.

Modernist architecture has come under fire in Britain too. A couple of years ago, the then housing minister, Kit Malthouse, celebrated the launch of the Building Better, Building Beautiful commission with a tweet comparing a neoclassical courthouse in Alabama with a glass and steel building in London. “One will stand for centuries, one won’t,” wrote Malthouse.

Buildings can be read as part of an unbroken chain, embodying an uncomplicated continuity between past and present

A big part of the appeal of classical architecture for the right is this perceived ability to endure. Buildings can be read as part of an unbroken chain, supposedly embodying an uncomplicated continuity between the past and present, with little or no “revisionism” getting in the way of our admiration of the way things were.

The commission – chaired by the late philosopher Roger Scruton – concluded this year with a much less antagonistic final report, but not before Scruton swore to protect people from “disciples of Le Corbusier and Mies”, two giants of global, modernist architecutre. It doesn’t matter that architects haven’t been building in the brutalist or Bauhaus style for decades. Their social idealism is still, apparently, a clear and present danger.

Why? Modernism took the advances of the technological revolution, then tried to make a clean break with the styles of architecture that had preceded it. Its practitioners were designing for the masses, imagining egalitarian utopias and rejecting what had come before.

Neoclassical architecture, on the other hand, harks back to a time when European nations were more powerful on the world stage, more homogeneous – and derived no small part of their power from the subservience of racial minorities and women.

Fans of the neoclassical find plenty of material on the internet, where anonymous accounts pump out beguiling images of well-lit courthouses and town halls. To a casual observer these seem wholly innocuous, but it is worth reading between the lines.

The popular Twitter account Architectural Revival tweets tips for identifying elements of the style such as pediments and porticos, paints modernist architects as villains, and repeats affirmations along the lines of beautiful buildings are ones “your ancestors would recognise”. If your ancestors can spot a Doric column at 20 paces, it seems to be suggesting, then they’re probably a white European. Frequent reference is made to the imagined “our” – our heritage, our history, our tradition. In creating an us, however, there must also be a them. The exclusionary nature of this false definition of beauty and ugliness becomes obvious.

In short, the call to return to classical beauty often masks a desire to regress to a uglier place in the recent past where diversity was a vice and subjugation a virtue. Modernity and globalisation, and their material expressions, are the enemy.

The far right operates by inciting fear about forces that threaten to topple white supremacy. Conspiracy theories, such as the “great replacement”, imagine that white Europeans are under threat from Muslim migration and the falling birth rates of emancipated women. Classical architecture, with its invocation of white power, is a visual and psychic balm for those anxieties.

One architecture nerd with a passion for the beautiful ruins of ancient Greece and Rome was, of course, Albert Speer. Hitler’s favourite designer pioneered the concept of Ruinenwert, or ruin value. Together they planned entire cities that would speak of the greatness of the Third Reich, even as relics, just as the Roman empire’s ruins communicated its imperial power down the ages. I hestitate to make a direct comparison, but it’s hard not to be reminded of this on some level reading Trump’s order to return to the glory of “republican Rome”. In his memoir, Speer recalled that Hitler liked to philosophise: “Ultimately, all that remained to remind men of the great epochs of history was their monumental architecture … Periods of weakness are bound to occur in the history of nations, but at their lowest ebb, their architecture will speak to them of former power.”

Traditional narratives depict imperial Rome as a bulwark against invading barbarian hordes, an image that has clear appeal for the contemporary right. It was also a slave-owning society, and one where women’s power was limited to the home, since they were unable to vote or hold public office.

History, both ancient and more recent, is a place of refuge only for a select few. If you’re not white, or not a man, the chances are that the past would not be such a beautiful place to return to. Aesthetic time travel is far more appealing to those who see classical architecture as a comforting symbol of their former power. Eulogising these styles without confronting their complicated history is a mistake.

Diversity – in a society and in architecture – must be defended. Architects should be free to build brighter futures through their work, not forced to mirror a compromised past.

• India Block is assistant editor at Dezeen