Once there was a president who warned the world about conduct his government would not tolerate. And when this “red line” was crossed, or seemed to be, he took the US to war. Though this might sound like America’s involvement in Iraq, or Afghanistan, or Belgrade, or Libya, and what may yet become a wider war in Syria, this story began 50 years ago, on 4 August 1964.

That was when Lyndon Johnson interrupted TV broadcasts shortly before midnight to announce that two US ships in the Gulf of Tonkin had come under fire in international waters, and that in response to what the president described as this “unprovoked” attack, “air action is now in execution” against “facilities in North Vietnam which have been used in these hostile operations”.

The Americans launched 64 bombing sorties, destroying an oil depot, a coal mine and a significant portion of the North Vietnamese navy. Three days later, both houses of Congress passed a joint resolution authorising “the president, as commander-in-chief, to take all necessary measures to repel any armed attack against the forces of the US and to prevent further aggression”. Within three years the US would have 500,000 soldiers in Vietnam. Even today, the Gulf of Tonkin resolution remains the template for presidential war-making.

That 4 August, Daniel Ellsberg was starting work at the Pentagon. A young mathematician who had served as a captain in the marines, then gone on to graduate study at Harvard and a job as a civilian analyst for the Rand Corporation, where he had helped shape America’s response to the Cuban missile crisis, Ellsberg was among the first to receive the classified “flash” signal from the USS Turner Joy, the destroyer that claimed to be under attack.

At the time a devout cold warrior, Ellsberg told me his initial reaction was “We must strike back”. Yet within a few days, when Johnson repeated his accusation that “the attacks were deliberate. The attacks were unprovoked”, and assured the world that “we seek no wider war”, Ellsberg knew “all of those statements were false” – the beginnings of a disillusion that would eventually lead him to leak the top secret Pentagon Papers seven years later. What he didn’t know, and what remained for decades one of America’s most tightly guarded secrets, is that the attack on 4 August may never actually have happened.

I recently interviewed Ellsberg for the BBC. His memory of the Gulf of Tonkin incident was vivid, detailed and totally consistent with the documents – many of them classified “top secret” – which historians have spent years tracking down. But what struck me most forcefully in talking to him and to other witnesses and historians about those events were the remarkable parallels with our world today.

Like Barack Obama, Lyndon Johnson was a president who felt “the fierce urgency of now” to address the glaring inequalities of American society. Just a month earlier, with Martin Luther King Jr standing at his side he had signed the civil rights act, ending racial segregation. And as the Pulitzer prizewinning historian Frederik Logevall told me, “Johnson apparently said in the spring of ’64, ‘I don’t think we can win in Vietnam and I don’t think we can get out.’ You can have all the military power in the world, but if you can’t win the thing politically then you’re not going to succeed.”

Reading headlines from Syria, or watching the news from Iraq – where an army which had been trained and equipped at enormous expense simply laid down their weapons and ran away, abandoning territory that had cost British and American troops their lives –it has been impossible to resist the sensation, in the words of the great Yankee catcher Yogi Berra, that this was “deja vu all over again”. Listening to Obama and David Cameron respond to the debacle in Iraq, I kept hearing echoes of President Kennedy declaring in September 1963: “I don’t think that unless a greater effort is made by the government to win popular support that the war can be won out there.”

Thanks to Edward Snowden and the Guardian we know a great deal more about how Britain and America view the world – and their own citizens – than was even suspected in 1964. But we still may have to wait decades to find out what George Bush said to Tony Blair about Iraq, or what Obama told David Cameron about Syria. We can, however, finally tell the full story of what happened – and didn’t – in the Gulf of Tonkin.

War, Lies and Audiotape, an episode of BBC Radio 4’s Archive Hour, will be broadcast on Saturday 1 August at 8pm

• This article was amended on 6 August 2014. An earlier version described the destroyer Turner Joy as a battleship.