George Brandis was frank in his farewell speech to the Australian Security Intelligence Organisation, delivered a day before he left parliament earlier this month.

Amplifying fears already present in the agency, the outgoing attorney-general underscored the view that changes to Australia’s security structures could place operations at risk of political interference.

Since before the Department of Home Affairs was created late last year, members of the intelligence community have been privately expressing concerns that having so much power vested in a single portfolio could see agencies such as ASIO – which will fall under it as soon as legislation is passed – vulnerable to political pressure.

The Saturday Paper has obtained details of a private address Brandis gave to hundreds of intelligence and security officers at ASIO’s fortress-like headquarters in Canberra on February 7, in which he acknowledged those concerns and effectively endorsed them.

Brandis quoted from Pezzullo’s speech directly, describing it as sounding like an urtext for a manifesto to rewrite the nation’s whole security framework.

The attorney-general’s portfolio has maintained oversight of an otherwise independent ASIO since its postwar inception in 1949. In the lead-up to last year’s restructure announcement – and since – Brandis has made it known to cabinet colleagues he believes it should stay that way.

On the eve of his retirement as chief law officer, he reportedly used his ASIO speech to suggest that the future direction of the new mega-department, which is headed by Peter Dutton, could expose the agency to the kind of ministerial and bureaucratic direction from which it has always been – and, he said, should be – protected.

Brandis is understood to have said the agency’s statutory independence was of “utmost importance” and that Australians would lose confidence in ASIO if they saw it as doing the bidding of “political or bureaucratic masters”.

Sources told The Saturday Paper that Brandis acknowledged the creation of the new department had been “unsettling” for the agency and that while there was still some “discomfort” about the new structure, he was confident it would strive to make the changes work.

In his private remarks to ASIO, Brandis focused not on Dutton but the secretary of the Home Affairs Department, Michael Pezzullo.

He is understood to have said there was “an obvious potential tension” between the operations of a domestic intelligence agency built on a culture of upholding the rule of law and the kinds of ambitions for Australian security that Pezzullo had outlined in a speech of his own last year.

Some senior members of the intelligence community – within ASIO and other agencies – are understood to share those concerns.

The tone and content of Pezzullo’s October 17 speech to a Canberra conference of the Trans-Tasman Business Circle alarmed some officials, with its reference to globalisation having created a “dark universe” which must be attacked to avoid “the end of days”.

The security agencies did not endorse the idea of the mega-department when Dutton and Pezzullo proposed it after the 2016 federal election.

Before that, then prime minister Tony Abbott had rejected a similar idea. But Abbott’s successor, Malcolm Turnbull, accepted Dutton’s proposal, announcing the change in July last year, to take effect on December 20.

Dutton, a former Queensland police officer, is the conservatives’ preferred leadership candidate and is touted as a potential future rival for the prime ministership.

In his speech in October, Michael Pezzullo told his audience that along with the creation of Home Affairs, he foresaw the need to restructure traditional models of policing, intelligence, customs, immigration, transport security and border protection.

Pezzullo said: “How we do things today is not going to be … how we will need to do things tomorrow.”

He argued that old models of having the state at the apex of the community would no longer work: instead, it had to be silently inserted everywhere.

Pezzullo illustrated his arguments with cultural and historical references, including to the AC/DC song “Dirty Deeds Done Dirt Cheap” and J. R. R. Tolkien’s Lord of the Rings trilogy. Speaking about the need to anticipate risks and prepare for them, he praised television presenter Rebecca Judd for wearing flat shoes to the AFL grand final in case there was a terrorist attack and she needed to run.

He said “home” could no longer be assumed as a safe place.

“Home has to be pushed out in this globalised world because security is a task, it’s not an end-state,” Pezzullo said.

He envisaged Home Affairs as “the centre of excellence of figuring out ‘how does Australia work?’ ”

He said: “We have to be careful about how we write this down, because when you then write the manual, ‘How you take Australia down’, there’ll be one copy of that and I’m not going to tell you where I’m going to keep that, because that’s going to be a very dangerous book.”

Pezzullo joked it would be disguised in a set of J. K. Rowling novels, so people would just think he was a Harry Potter fan.

Some in the intelligence community are seeing a less funny side.

Pezzullo advocated massively increased surveillance and intervention and the re-engineering of the security architecture and shape of government.

“The state has to embed itself invisibly into global networks and supply chains, and the virtual realm, in a seamless and largely invisible fashion, intervening on the basis of intelligence and risk settings, increasingly at super scale and at very high volumes,” Pezzullo said.

Some fear that sounds a lot like it could tip easily from keeping Australia safe to routinely spying on its own people.

In his own recent address, Brandis is reported to have quoted from this section of Pezzullo’s speech directly, describing it as sounding like an urtext – an origin text or blueprint – for a manifesto to rewrite the nation’s whole security framework.

He noted ASIO was placed and had remained under the jurisdiction of the attorney-general’s department because that was the portfolio responsible for maintaining the rule of law. He is understood to have said that preserving ASIO’s operational autonomy, at present protected by law under the ASIO Act, was crucial.

The Saturday Paper contacted the former attorney-general but he had no comment.

Brandis’s remarks are likely to fuel what has become a public feud between him and Dutton, after the outgoing attorney-general used his valedictory speech in parliament that same day to mount a thinly veiled attack on the Home Affairs minister.

“It is for the attorney-general always to defend the rule of law, sometimes from political colleagues who fail to understand it or are impatient of the limitations it may impose upon executive power,” Brandis told the Senate.

His remark was interpreted as being directed at Dutton, who had attracted criticism from law bodies for remarks about gang violence in Victoria and “lenient” judges.

Dutton hit back on Sydney’s Radio 2GB, saying Brandis thought himself superior to ordinary people and always wanted to be “the smartest person in the room”.

A fortnight after Brandis left politics, Dutton has now announced he plans to toughen the foreign-allegiance laws that Brandis amended. The changes had been made when concerns were raised that the legislation as it stood could leave people stateless, in breach of international law.

Dutton argues the laws are not working because they have failed to strip dual-national convicted terrorists of their citizenship.

Under the changes, the Home Affairs minister will now oversee ASIO and other security agencies, including the Australian Federal Police.

But the new attorney-general, Christian Porter, will still have to sign off on ASIO applications for warrants.

Despite the hesitation of some in the intelligence community about the new structure, there are others endorsing it.

The policy director at the Australian National University’s National Security College, Jacinta Carroll, is backing the change.

“I do think it’s a good thing,” Carroll told The Saturday Paper. “This is an extremely significant development in Australia’s national security architecture – of course the early stages of forming the new arrangements won’t be without bumps and I do think it will take a few years for the agencies to bed down in the new Home Affairs portfolio … If you look at how to position our agencies for the future, this is a no-brainer.”

She says any resistance within the agencies may simply be following a pattern in team development outlined by change management specialist Bruce Tuckman: “forming, storming, norming and performing”.

Carroll believes agencies including ASIO and the AFP, which have long worked closely together anyway, will maintain their autonomy and benefit from having a joint strategy.

“You’re talking about agencies with unique statutory roles and great capability that are fiercely independent. They’ve been through a lot and they’ve got very proud histories.”

Carroll says the reconfiguration can provide strategic direction and oversight.

“I’m pretty sure all of the agency heads would agree with that and think they could benefit from it,” she says.

Those who are more apprehensive fear ASIO’s new minister will seek to weaken that autonomy by amending the statute that gives it a direct line of reporting to the prime minister and precludes a departmental secretary from giving it instructions.

Under law, while the portfolio minister has legal authorisation and oversight of ASIO operations, he or she cannot direct them and nor can the departmental head.

Section 8 of the ASIO Act sets out the lines of authority in controlling the organisation, stating that a minister can’t override the opinion of ASIO’s director-general concerning the nature of advice it should give.

The minister can override whether collecting or communicating intelligence on a particular individual is justified on security grounds, but that has to be done in writing.

A copy of any direction to override must be also provided to the inspector-general of intelligence and security and if it relates to intelligence on an individual, also to the prime minister. The director-general is required to ensure a record is kept.

Issues relating to directives to ASIO surfaced recently with the publication of a document included in a collection of cabinet files obtained by the ABC.

The October 2013 document from the Immigration Department to its then minister and now treasurer Scott Morrison revealed the then secretary, Martin Bowles, had written to ASIO to ask it to slow down the processing of asylum seekers’ security assessments.

The document indicated the requested go-slow was so some applicants missed the deadline to qualify for permanent visas.

The minister had issued a direction to other agencies over which he had authority, ordering them to rearrange their processes. While not bound by the direction, ASIO was being asked to voluntarily abide by it. The agency’s response is not known.

In a statement when the document was published, Morrison said: “As minister for Immigration and Border Protection, it was my policy and practice to put Australia’s national security interests first.”

This week, Peter Dutton gave his first public speech as Home Affairs minister, telling the National Press Club there had been similarly named departments since 1901 but none with the focus, nor facing the challenges, of this one.

“This reform is the most significant change to Australia’s law enforcement and domestic security arrangements in decades and Australians expect their government to keep them prosperous, safe and secure,” Dutton said. “It is as far-reaching as it is necessary.”

He said governments were too frequently forced to restructure in the midst of – or immediately after – a security crisis, but with the creation of a single agency this government was taking preventive action.

“In contrast, the Turnbull government is acting now to protect Australians and their interests, in a security environment few imagined 20 years ago,” Dutton said.

He explained the new department was modelled on the British version, also drawing on some experience from the Department of Homeland Security in the United States.

“Past reviews of Australia’s domestic security and law enforcement arrangements have called for better coordination and cooperation between agencies,” Dutton said.

“In establishing the Home Affairs portfolio, the government is doing just that – seeking to develop and entrench a more integrated approach to our nation’s security and prosperity. We are bringing together the agencies responsible for our domestic security, law enforcement and border security, as well as those that facilitate trade and the legitimate travellers and migrants on whom our economy depends.”

Dutton said that for the new portfolio to succeed it had to “be more than a bureaucratic construct”.

“It must make a real difference to the prosperity, safety and security of Australians.”

It also has to manage the apprehension of those about to move their boxes in under its roof.

And as Australia’s next high commissioner to London, George Brandis faces his own challenge: hosting the Home Affairs minister Peter Dutton whenever he visits Britain.