The chief of Australia’s domestic spy agency, Asio, defied a direct order from then Labor prime minister Gough Whitlam in 1974 to sever all ties with America’s Central Intelligence Agency.

Whitlam – hostile to US spy bases in Australia and angy with the CIA’s undermining of leftwing administrations, including Chile’s Allende government in 1973 – effectively forced the Washington-Canberra intelligence relationship underground until the dismissal of his government in late 1975.

The decision by the director general of the Australian Security Intelligence Organisation, Peter Barbour, to ignore Whitlam’s directive is revealed in the latest volume of Asio’s official history by historian and former army officer John Blaxland.

Blaxland’s book, The Protest Years: The Official History of Asio 1963-1975, also discloses the depths of disquiet in the Nixon administration at the appointment in 1974 of Jim Cairns as Australian deputy prime minister, due to the access this would afford him to classified United States intelligence.

While the book acknowledges a circumstantial case of CIA involvement in Whitlam’s dismissal by the governor general, John Kerr, on 11 November 1975, Blaxland writes that the Asio files – to which he had unfettered access – offer no evidence to support the “deeper conspiracy theories” of American involvement.

He also concludes that there is no evidence that Asio was working in response to US intelligence direction against Whitlam, instead of on behalf of the Australian government.

The book lays bare the animosity between Whitlam and Barbour – who quit under pressure in September 1974 – over US-Australia intelligence ties. It was a time of existential uncertainty for Asio, after Whitlam – who had long been suspicious of the agency’s scrutiny of some public officials and their staff – announced a royal commission into intelligence and security.

“Whitlam was so unhappy with the closeness of Asio’s ties with its US partners that he gave instructions to Barbour to sever them. But Barbour felt this would be harmful to the nation, causing damage to critical intelligence links with the United States,” Blaxland writes.

“Barbour decided, therefore, to maintain informal contacts with the United States Government. He selected a mid-ranking officer who despite having reservations about the instructions nevertheless carried them out. This ... bizarre experience reflected the unusual nature of ASIO ‘business’ in a period of dramatic and significant change across the Australian political landscape.”

After the 1974 double dissolution election, socialist left Labor former policeman and academic economist Jim Cairns emerged as deputy prime minister, defeating Lionel Bowen in a caucus vote.

In 1972 Whitlam had ended Australia’s involvement in the Vietnam war. Two years later he publicly questioned the role of US spy bases in Australia and its continued presence in Vietnam, prompting the US president, Richard Nixon, to request a state department review of American-Australian relations.

After the promotion of Cairns, US embassy officers in Canberra explained to Barbour Washington’s negative reaction to Whitlam’s new deputy.

Blaxland writes: “Barbour was pressed to say whether he could give a security clearance to Cairns enabling him to access top-secret and sensitive material. Barbour told them that in Australia decisions concerning ministerial access were made by the Prime Minister personally. Barbour advised that they urge their superiors, as far as they could, not to react precipitately in a matter of such importance and to wait the course of events.”

After Cairns was sworn in as deputy PM, a senior US embassy official visited Barbour in his office at America’s request. The American official said secretary of state Henry Kissinger and defence secretary James Schlesinger viewed Cairns as “a radical with strong anti-American and pro-Chinese sympathies”.

“The American wanted to know whether the elevation of Cairns entailed him being granted access to American intelligence and if so, whether he could be trusted with its security.”

By early 1975 America had become even more concerned about Australia as an ally.

“US embassy officials confided to Asio that the ‘maintenance of the ALP Government in power is essential to Soviet planning for this area and their activities in Australia would be tempered by this consideration’,” Blaxland writes.

On 8 November 1975, as pressure mounted on Kerr to sack the Whitlam government over the continuing Senate impasse on budget supply, Asio’s senior liaison officer in Washington was summoned to see the CIA’s East Asia division chief, Theodore Ted Shackley.

There was heightened concerned in the agency and the Nixon administration that Whitlam had named specific purported CIA officers’ connections to the Pine Gap tracking station near Alice Springs.

Blaxland writes that the liaison officer was “given a message to pass to Asio’s interim director general, [Frank] Mahony”.

The message was conveyed to Canberra that Washington was “perplexed as to what all this means” amid concern that Whitlam would “blow the lid off those installations in Australia where the persons concerned have been working and which are vital to both of our services and countries particularly the installation at Alice Springs”.

Blaxland says that according to one assessment, “the Shackley cable was probably the most serious note passed to Australian authorities in the history of bilateral relations between Australia and the United States – a virtual ultimatum to Mahoney as director general of ASIO to do something”.

Blaxland interviewed Malcolm Fraser, who was hastily sworn in as interim Liberal prime minister after Whitlam’s sacking and won the subsequent election on 13 December 1975.

In later life Fraser was an extreme sceptic about elements of the US-Australia relationship.

Blaxland writes that “reflecting on the rumours of US destabilisation of Whitlam and the aspersions cast upon Kerr, Fraser later maintained that the stories were ‘crap, total crap’”.

Blaxland, whose book is to be launched at the new Asio headquarters in Canberra on Friday, told Guardian Australia: “This is a story about Australia during one if its most turbulent periods – when anti-Vietnam war and anti-conscription protests were in full swing and the baby boomer generation was rewriting the rule books. It is an exciting and fresh look at the country in transition through the eyes of an organisation trying to catch up.”