The retired defence chief Angus Houston has warned Australia should not participate in freedom of navigation operations within 12 nautical miles of China’s artificial islands in the South China Sea.

He has also criticised the push to blockade China’s artificial islands, saying that would invite a “sharp response” from Beijing.

But he has reminded China that all countries must abide by the protocols of international law, saying a global, rules-based order should apply in the South China Sea “as it applies everywhere else”.

“To have a sort of environment where everybody does their own thing is clearly not on for the future,” he said.

Appearing at the National Press Club on Tuesday, Houston said diplomacy would be the best response to current problems in the South China Sea.

He said it was too late to stop the Chinese building artificial islands in the region but “we may be able to ... convince our Chinese friends, with the assistance of all the other players in the region who have the same concern as we do, that they do not militarise those artificial islands”.

“I think any militarisation of those islands is deeply destabilising to the region,” he said.

Tensions have increased in recent years as China has increasingly claimed sovereignty over almost the entire South China Sea, building seven artificial islands on reefs and rocks, and outfitting them with military-length airstrips and anti-aircraft guns.

In July last year, the permanent court of arbitration in The Hague ruled that China had no historical title over the South China Sea, increasing global diplomatic pressure on Beijing to scale back military expansion in the area but China rejected the ruling.

The US secretary of state, Rex Tillerson – during his confirmation hearing in Washington last month – argued China should be denied access to its artificial islands, saying its control and construction of artificial islands in waters claimed by neighbouring countries was “akin to Russia’s taking of Crimea”.

“We’re going to have to send China a clear signal that, first, the island-building stops and, second, your access to those islands also is not going to be allowed,” he said.

But Houston said on Tuesday that a blockade would be a bad idea.

“I wouldn’t support that,” he said. “I think that that will invite a sharp response from the Chinese.”

He also said Australia should not take part in freedom of navigation operations within 12 nautical miles of the artificial islands, warning: “I think that could provoke a response, a military response, and I don’t think that that would be a good idea.

“I think it’s all about diplomacy in the first instance and there’s no need to provoke any military response.”

But he did say Australia needed to ensure that overflight was not a problem, or that transit by ships through the South China Sea was constrained “in any way.”

Australia has previously drawn criticism from China for running surveillance flights over disputed islands in the South China Sea and supporting US freedom of navigation exercises there.

Labor’s defence spokesman, Richard Marles, argued last year that Australia’s navy and airforce should be “fully authorised” to conduct freedom of navigation exercises in the South China Sea, including within 12 nautical miles of artificial islands built by China.