At the end of January, Theresa May rushed to Washington to ensure that she would be the first foreign leader to meet the newly inaugurated American president. The pair held a joint press conference – shortly before she was photographed awkwardly holding Donald Trump’s hand, and shortly after she extended an invitation for a state visit to London to meet the Queen – which began with the usual invocations of the sacred bond between the United Kingdom and the United States. “The special relationship between our two countries has been one of the great forces in history for justice and for peace,” Trump declared in his opening statement. “We have one of the great bonds.”

In reply, May said that her invitation to the White House was “an indication of the strength and importance of the special relationship that exists between our two countries, a relationship based on the bonds of history, of family, kinship and common interests”, before reiterating her anxious hope for a trade deal with the US to “cement the crucial relationship that exists between us, particularly as the UK leaves the European Union and reaches out to the world”.

After a question to Trump from an American journalist, it was May’s turn to call on a British reporter. She picked the BBC political editor Laura Kuenssberg, who asked Trump a series of pointed questions about torture, Russia, and banning Muslims, at which point Trump turned to May with mock horror. “That was your choice of question?” he said, to laughter. “There goes that relationship!” Trump’s humour, like everything else about him, is random, unpredictable and often almost incomprehensible. But he does have a disconcerting habit of saying, albeit unpleasantly, things that happen to be true, such as that free trade agreements have been a mixed blessing for working-class Americans, or that the Republican leader in Congress, Paul Ryan, is “very weak and ineffective”, or that the CIA is both self-satisfied and incompetent. Whether “There goes that relationship!” proves to be another true word spoken in jest remains to be seen.

What was unmistakable was the frantic and rather degrading desperation that May’s visit expressed. She had reportedly been dismayed by the length of time it had taken Trump to make contact with her after his election – the eighth of the foreign leaders he rang, or was it ninth? Then she had felt the humiliation of seeing Nigel Farage and Michael Gove meet the president-elect ahead of her: evidence of Trump’s genuine special relationships, with rightwing nationalism and with Rupert Murdoch (who was, we have since learned, sitting in on the interview with Gove).

Not that there is anything new about this neediness, even if Britain’s current position appears especially feeble. Over and over again on her visit to the US, May uttered the hallowed phrase: “the unique and special relationship that exists between us”. The words are intoned repetitiously, over and again, almost as a prayer or incantation, to the point when everyone has long since forgotten what they actually mean.

Even debunking the special relationship now has a long and distinguished history of its own. There is nothing new in pointing out, as the late German chancellor Helmut Schmidt famously quipped, that the relationship is so special that only one side knows it exists – or, at any rate, in observing that dealings between American presidents and British prime ministers have been excruciatingly one‑sided. Tony Blair’s service to George W Bush in the invasion of Iraq was only the most extreme case of this game of give and take, in which the president is given everything and takes everything, offering nothing in return.

Least of all was there anything new in the invocation of one hallowed name and memory. Before her visit, May had sent Trump the text of the broadcast Winston Churchill had given from the White House at Christmas 1941: “I spend this anniversary and festival far from my country, far from my family, yet I cannot truthfully say that I feel far from home … I cannot feel myself a stranger here in the centre and at the summit of the United States. I feel a sense of unity and fraternal association.” Five years later, in a speech at Westminster College in Fulton, Missouri, Churchill upgraded “fraternal association” to something more potent, enunciating for the first time the “special relationship between the British Commonwealth and Empire and the United States”.

Americans have almost never used the phrase, quite apart from the fact that their presidents have so often disproved it in practice. But Americans do invoke Churchill – and how! The American cult of Churchill is a very striking phenomenon, and a very curious one, and Trump is only the latest, and most unlikely, person to flourish that name. Within hours of entering the White House, he fulfilled a pledge to restore a bust of Churchill to a place of prominence in the Oval Office, where it had been proudly displayed by Bush, something The Sun improbably claimed as “a diplomatic coup” for Theresa May.

No one was more delighted than Murdoch’s rightwing channel Fox News. Echoing the famous (if possibly apocryphal) signal sent to the Royal Navy when Churchill returned as First Lord of the Admiralty in September 1939, Fox said that “President Trump has, for his part, signalled his own ‘Winston is Back’ moment, by symbolically returning the bust of Churchill … Far beyond symbolism, Trump is signalling, with Prime Minister Theresa May, his own reset of the special Anglo-American relationship, and it couldn’t have come at a more propitious time … This is a proud and promising moment for both leaders and one that bodes great potential for the nations they lead respectively and the partnership that they have expressed interest in resetting and strengthening.”

Looked at more objectively, it might seem that this was an exceptionally unpromising moment for a British prime minister to be embracing an American president who is not only intensely distrusted and disliked by a large part of humanity, but is also frighteningly unstable and erratic. If ever there was a good time to subject the myth of the special relationship to renewed scrutiny, this would seem to be it.

For what now seems most remarkable about this myth – invented so profitably by Churchill and maintained with blind faith by his many successors – is that on the one hand it has so easily and so often been demonstrated to be false, and yet on the other it refuses to go away.

To call it a myth seems apt, because the special relationship is indeed like some fabulous creature from mythology, which can never be slain. Over and again the sword of hard experience has slashed at it, or the sword of reason has been thrust in – it has been derided by American statesmen and English commentators, and even more often made to look foolish by events. And still the idea doggedly persists.

With the passing of time – and for all the accumulation of evidence that falsifies the idea – prime ministers and party leaders have become ever more insistent on the significance of the special relationship. It is a heresy in British politics to suggest anything other than standing shoulder to shoulder with the US, to the point where our leaders are not simply saying “my country right or wrong”, which is bad enough, but “their country right or wrong”, which is barely sane.

We have just witnessed the House of Commons voting for something that most MPs had originally opposed, with the majority now giving May what she regards as a carte blanche for Brexit. The last time so many voted against their own instincts was more than a decade ago, when in 2003 a majority of Labour MPs voted for the invasion of Iraq, which most of them did not want.

The special relationship is indeed like some fabulous creature from mythology, which can never be slain

Even in this latest case, the special relationship lurks in the background. May repeatedly says, and might even believe, that she can renew the Anglo-American partnership by embracing Trump and forging a trade alliance that softens the economic blow of exiting the European single market – along with the risible pretence that Britain’s wise and calming influence will somehow moderate the erratic extremism of the Trump administration. This too has a long pedigree: “hug them closer” was the fatuous Whitehall slogan for dealing with the Americans even before the Blair government took office in 1997.

Then a new British ambassador was sent to Washington, and Sir Christopher Meyer left with instructions from Jonathan Powell, Blair’s chief of staff, ringing in his ears: “We want you to get up the arse of the White House and stay there.” Having transferred his allegiance from Bill Clinton to Bush, Blair genuinely persuaded himself that it was his duty to support the president come what may, telling his cabinet as early as March 2002 that, “We must steer close to America. If we don’t we will lose our influence to shape what they do.” The Tories agreed.

At the time, William Hague was between jobs, a former leader of the opposition and a future foreign secretary. In his own speech supporting the invasion, he said that “whenever we really need help, we turn to the United States of America”, and “it is also part of our national interest to act in concert with the United States of America”, to which he added the blatantly untrue claim that “every serious attempt to advance peace in the Middle East has been advanced under the auspices of the United States”.

Hague is not an ignorant man, and he has written a couple of serious historical biographies. But, like so many other British politicians, he seems transfixed by this idea of a mystical bond between the two countries, however much that flies in the face of historical fact, or that hugging the Americans in a “special” tight embrace might be gravely damaging to the British national interest.

The sanctity of the special relationship did not merely survive the Iraq disaster; it has somehow been enhanced even further by its repeated invocation in the aftermath of its greatest failure. There is a magical logic to it: the more desperate our predicament, the more we must cling to our very powerful ally; at the same time, the worse the behaviour of our special partner, the more we must hold it close to ensure our continued influence.

May’s visit to Washington, and the consternation at home over her invitation to Trump, has elicited a loud reprise of these contradictory arguments from Tory politicians and the tabloid press. The Daily Mail, which declares unconvincingly that “we have never been a great admirer of Donald Trump”, admonishes his “hysterical critics”, who threaten the stability of the special relationship with their distaste for “the democratically elected president of our greatest and most powerful ally, who has deep affection for our country”.

Never mind the optics, Theresa May’s US dash was mortifying | Jonathan Freedland Read more

After all, the Mail concludes, “the last thing Britain wants to do is make an enemy of the world’s greatest power”.

The possibility that there might be a path to be taken somewhere between making an enemy of the US and fawning on it at all times, even under a volatile and capricious leader, does not seem to occur to exponents of the precious special relationship. Nor do the triumphalist Brexiters pause to ask whether a man of Trump’s personality will actually allow his “deep affection for the country” to get in the way of whatever he may do at any moment as the whim takes him.

We have clung to the special relationship through thick and thin, whatever the changed circumstances and no matter how damning the consequences. Perhaps it should be seen as a kind of comfort blanket for prime ministers, who will not give it up. Or perhaps a supposedly lucky charm, acquired long ago and never discarded, even when experience should have taught that any good luck it may once have brought had long since run out.

When Dean Acheson was secretary of state under President Harry Truman at the dawn of the cold war, he forbade the State Department even to discuss the “special relationship” and ordered officials to stop work on a paper intended to define it. Acheson had come under attack from Senator Joseph McCarthy and the nationalist right – and did not want his enemies to suggest that his country was working to help the interests of another, a concern which has too rarely bothered British politicians the other way round.

When Senator John McCain visited England in 2006, two years ahead of his failed run for the presidency, he was cheered at Tory conference and ingratiatingly interviewed in the Spectator. “The special relationship between our two countries will endure throughout the 21st century,” McCain told the magazine. “I say that with total confidence because it’s lasted for 200 years.”

His total confidence was not matched by any authentic command of history. “Two hundred years” would take us back to the first years of the 19th century, when relations were so warm that the two countries went to war in 1812. After that, they came close to conflict three more times in the 19th century. The last was in December 1895 over an obscure border dispute between Venezuela and British Guiana, when President Grover Cleveland rattled his sabre, while Lord Salisbury, the prime minister, spoke soft words.

It was recorded at the time of that crisis that whereas the English were horrified by the prospect of such a war, in America a war with England would be the most popular of conflicts. One man who knew that very well was Winston Churchill, then a 21-year-old subaltern in the 4th Hussars. He was coming home from Cuba by way of New York that December when he witnessed the war fever, with the New York Times reporting the crisis under the headline “WANT TO FIGHT ENGLAND – Army and Navy Men Profess Great Eagerness to Go to War”. And as he sailed home, Churchill wondered whether he might not soon be returning with his regiment, to fight the Americans.

After the Armistice in 1918, Churchill wrote that “The conclusion of the Great War raised England to the highest position she has yet attained.” This skirted round the awkward fact that England ended that war enormously indebted to Washington. When he was invited to speak at an Anglo-American dinner in London in 1921, Churchill found it “uphill work to make an enthusiastic speech about the United States”, he told his wife, Clementine, “when they are wringing the last penny out of their unfortunate allies”.

Facebook Twitter Pinterest Harry Truman and Winston Churchill. Truman forbade the State Department even to discuss the ‘special relationship’. Photograph: George Skadding/Time & Life Pictures/Getty Image

Six years later, Churchill was chancellor of the exchequer in the cabinet of Stanley Baldwin, and told his colleagues that although they were expected to say that there could never be war with the United States, “everyone knows this is untrue”. And when he grew restless at the Treasury in 1928 and thought of becoming foreign secretary, Clementine reminded him this would be difficult, due to “your known hostility to America”. And this was the man who invented the special relationship.

But Churchill’s personal attitude to America completely changed in 1929, when he was out of office and made a luxurious, and lucrative, American tour, befriending William Randolph Hearst and various financiers, signing valuable contracts for journalism and books, including a projected history of what was then to be called a history of “the English-speaking races”.

Churchill’s version of the English-speaking races, much like the later invocations of the Anglosphere by his Tory admirers, did not make much room for Indian or African speakers of our language. In this as much as in his creation of the special relationship, there was more than a hint of the White Man’s Burden. It is often forgotten that Kipling’s poem of that name, published in 1898, is subtitled “The United States and the Philippine Islands”. He was urging the Americans to accept their new duty as an imperial race, as indeed they did.

But there was another imperative behind Churchill’s “special relationship”. The United States duly, if belatedly, and in the second case involuntarily, entered the two great wars of the first half of the last century, and emerged as the most powerful country on Earth. Meanwhile, Churchill’s country was about to shed its empire and plainly ceasing to be a great power, not to say a near-bankrupt dependency of American largesse, if the Americans felt like it. Those were the circumstances in which Churchill coined the special relationship, as a form of compensation.

The first devastating critique of the special relationship came from Dean Acheson, who had earlier forbidden the phrase at the state department. In a speech to military cadets at West Point in December 1962, shortly after the Cuban missile crisis, Acheson delivered something like a direct rebuke to Churchill – declaring the special relationship all but dead. For what followed his famous statement that “Great Britain has lost an empire and not yet found a role” was a rejection of Churchill’s consoling myth. “The attempt to play a separate power role – that is, a role apart from Europe, a role based on the ‘special relationship’ with the US,” Acheson said, “this role is about played out”.

Its obituary was written again in 1974, when Harold Wilson returned to Downing Street, eager to rekindle the special relationship that had cooled under his predecessor Edward Heath, who took the UK into the European Economic Community while distancing himself from Washington. In August 1974, Richard Nixon became the first president to resign the office amid the disgrace of Watergate. Louis Heren, then the deputy editor of the Times, and its former Washington correspondent, offered his own advice to the prime minister. “Mr Wilson must realise the ‘special relationship’ is dead.” But once again it proved hard to bury.

Why has this tired phrase meant so much to the British, when it was so often and so obviously proved to be so empty? There is something here that goes beyond misunderstood or rewritten history. It was Lord Palmerston, twice prime minister and three times foreign secretary, who enunciated not so much the best as the only rational basis for any country’s foreign policy: England has no eternal friends and no eternal foes, only eternal interests.

Invoking the special relationship is a way of evading that truth, and allowing one prime minister after another to desperately and sometimes abjectly cling to the belief that he or she is the best friend of the most powerful person in the world.

Historians began to scrutinise the wartime dealings of Churchill and Roosevelt, and realised how fraught they often were

All prime ministers since the war have been obliged to pay formal respect to the American alliance. But the most successful premiers have been those who kept it to lip service. Margaret Thatcher has often been appropriated by champions of the Anglo-American alliance – including the current occupant of the White House, who has dubbed Theresa May “my Maggie” – but Thatcher was far more hard-headed in practice. She met Ronald Reagan in April 1975, when she had just deposed Edward Heath as Tory leader of the opposition. Reagan was touring Europe ahead of his first bid for the Republican nomination, five years before he won the presidency.

She took to Reagan, and he to her, with Winston Churchill present by Thatcher’s request to bless their meeting (that is, Winston S Churchill MP, the great man’s grandson). When Reagan died in 2004, the veteran English journalist Sir Harold Evans claimed that “the relationship between Thatcher and President Reagan was closer even than Churchill and Roosevelt”. But that was doubly wrong: it was precisely during the 1980s that historians began to scrutinise and unravel the wartime dealings of Churchill and Roosevelt, and realised how fraught and far from fraternal they often were.

Although Thatcher liked Reagan and shared some of his convictions, she had no illusions about him. Soon after Reagan was inaugurated – quoting Churchill in his speech – she was talking over a drink one evening with Lord Carrington, her first foreign secretary. The conversation turned to the new president, at which Thatcher tapped the side of her skull, and said. “Peter, there’s nothing there.”

Nor did she succumb to another illusion that still persists to this day, six decades after it was hopefully expressed by Harold Macmillan, long before he became prime minister. In 1943 Macmillan was Churchill’s plenipotentiary in the Mediterranean, and in Algiers one day he was talking to Richard Crossman, a former Oxford don and future Labour politician. “We, my dear Crossman, are Greeks in this American empire,” Macmillan said. “You will find the Americans much as the Greeks found the Romans – great big, vulgar, bustling people, more vigorous than we are but also more idle, with more unspoiled virtues but also more corrupt.”

Facebook Twitter Pinterest Ronald Reagan walking his dog, Lucky, with Margaret Thatcher. ‘Although Thatcher liked Reagan and shared some of his convictions, she had no illusions about him.’ Photograph: Bettmann/Bettmann Archive

What should have been clear from the start, and was endlessly demonstrated in practice, was that those vulgar, bustling “Romans” in Washington had no wish or need to be guided by the worldly-wise Greeks in London. Macmillan himself learned that at the time of the Cuban missile crisis, and other prime ministers have learned repeatedly since.

No one has been more hypnotised by this illusion than the first prime minister to be born after the second world war. In 1998, Tony Blair was entertained at the White House by Bill Clinton, the first president to be born after the war. Giving the toast, Blair recalled another dinner, in England during the war, when the host was Winston Churchill and the guest was Harry Hopkins, President Roosevelt’s personal emissary. As he spoke about Anglo-American friendship, Hopkins quoted the Book of Ruth: “Whither thou goest, I will go; and where thou lodgest, I will lodge: thy people shall be my people, and thy God my God.” At that, Blair reminded his audience, Hopkins paused before he added, “even to the end” and at that “Churchill wept”. Hearing that story, Bill Clinton also wept.

That lachrymose occasion was the prelude to something far worse. Most of the reasons Blair gave for the invasion of Iraq were false, but Blair did hold one belief with passionate sincerity: it was his duty to support the American administration at all times, because only by displaying complete loyalty could he expect to influence them.

This was “Greeks and Romans” writ new, and more fantastical than ever. There was never the smallest likelihood that the Bush administration would allow Blair to shape its actions – just as there is today not the slightest reason to expect that President Trump will heed the advice of Prime Minister May if it runs counter to his wishes.

Blair’s support was politically very useful to Bush, but the reciprocal benefits to this country consisted mostly of flattering references to the greatness of Churchill – dubbed the “Man of the Century” by Murdoch’s Weekly Standard, then the most zealous voice of American neoconservatism.

Bush invoked Churchill at every opportunity. Meanwhile, Blair told colleagues that Saddam Hussein was Hitler and that he was Churchill. Visiting a stricken New York, Blair managed to say that “My father’s generation knew what it was like. They went through the Blitz,” when “there was one country and one people that stood side by side with us then. That country was America, and the people were the American people.” During the German bombing of London and other British cities, many countries stood side by side with us, but the United States was conspicuously, and profitably, neutral.

Facebook Twitter Pinterest George W Bush and Tony Blair at Camp David in 2001. Blair believed his only hope of influencing the US president was by displaying complete loyalty. Photograph: Reuters Photographer/Reuters

As we have seen, such flagrant fabrication of history is the bedrock of the special relationship. It is perhaps no wonder that an alliance founded on delusion reached its apotheosis in the unwanted and unlawful catastrophe in Iraq. In fact, it might have been foreseen a century earlier by Mark Twain, who introduced Churchill in 1900 during the newly elected MP’s lecture tour of America. Twain was a critic of both the Boer war, from which Churchill had recently returned, and of his own country’s new age of empire, which had begun with a war against Spain and the annexation of the Philippines.

“England sinned when she got herself into a war in South Africa which she could have avoided, just as we have sinned by getting into a similar war in the Philippines,” Twain said. “And now that we are kin in sin, there is nothing more to be desired.” It serves as a fitting description of the Iraq adventure, which surely should have been the bitter final act of the special relationship.

What Theresa May fails to see should be perfectly obvious. The United States is a sovereign country – that’s the point of the Fourth of July – with its own objectives and interests. These may or may not be wise and virtuous, and they may or may not coincide with those of other countries, Great Britain among them. In his biography of Harold Macmillan, Charles Williams observed that Macmillan persuaded himself that there was some mystical bond between the two countries, quite failing to see that “the United States, like all great powers, would in the end follow – without necessarily much regard for others – what it perceived from time to time to be its own interests”.

For Macmillan’s successors, faith in the special relationship has been in truth a symptom of continuing national decline. When the prime minister made her frantic dash to Washington, she may have thought that this was a display of strength and superiority. In truth, it was the opposite – a display of desperation and yearning that are themselves symptoms of weakness, not to say a national leader in a very tight spot, whether she knows it or not. Perhaps May also deludes herself that the present moment is especially propitious for a renewal of “our special relationship”.

But on any objective view, the moment could not be worse. Whether in regard to British relations with the European Union or British policy on the Middle East or almost anything else, the conjunction of Brexit and Trump is especially perilous, and not only because we are now even more reliant on a nation led by Donald Trump.

For more than 60 years, the United States has strongly supported what is now the European Union, as well as British membership. As Raymond Seitz, the experienced American diplomat who was ambassador in London in the early 1990s, warned his British hosts: “Always remember that the United States is interested in Britain only in so far as it is a player in Europe.”

Perhaps here we may find a clue to the improbable persistence of British hopes for the special relationship – and an explanation for its popularity among those Conservative Europhobes who most loudly champion the “Anglosphere”. A few years after Acheson’s stinging 1962 speech, his position was amplified by Henry Kissinger, not yet in the employ of Richard Nixon, who observed that “many influential Americans have come to believe that Britain has been claiming influence out of proportion to its power”.

As Kissinger said, Washington firmly believed that Britain’s place was in Europe – and that it “should be treated as simply one or other European country”. But this is what Britain will not abide: to accept that the special relationship is a fiction would be to accept, in Kissinger’s words, that we are simply one among many countries in Europe.

Now the British government is engaged in what is likely to be a very difficult divorce that threatens to turn downright nasty, and one in which, whatever ministers or tabloids may say, our erstwhile European allies hold most of the cards. The Daily Mail may harbour the fantasy that May’s friendship with Trump has “foreshadowed a dramatic shift in the balance of global trade” that will send our EU exes into spasms of jealousy. But what has Trump done that might possibly be considered helpful to the British?

And in any case, how can it possibly be in our interests to bind Britain to a man who is so obviously temperamentally unsuited for high office? While Trump governs by tweet, his administration is already in disarray. Staffers “get up in the morning, read Trump’s Twitter posts, and struggle to make policy to fit them,” the New York Times reported this weekend.

It is not Trump’s repellent personal conduct that should worry Theresa May, or even his hateful attitudes, so much as the glaring fact that he is the most unstable and irrational leader of a great power for perhaps a century. It is now more than 70 years since Churchill first concocted the “special relationship”, more than 50 years since Acheson declared it “played out”, and more than 10 years since the calamitous invasion of Iraq. What further grave national misfortune must befall us before our rulers grasp the truth?

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.