The decision by the Australian government to turn asylum seeker boats back into Indonesia’s territorial waters and its ports was always a high-risk game. It is no surprise that it has ended in a serious confrontation with Indonesia, particularly following the admissions that Australian naval vessels have made incursions into Indonesian territorial waters.

Indonesia retaliated with some heavy rhetoric from foreign minister Marty Natalegawa and other officials, and dispatched its own naval vessels to the areas where incursions have taken place. Indonesia’s media have been full of these events, together with the recent spying scandals. Frantic telephone calls and requests for meetings by Australian ministers have drawn cold responses from their Indonesian counterparts.

This may be a simple case of huffing and puffing. But coming hard on the heels of the acrimonious exchanges over spying by Australia’s intelligence agencies, it would seem that the relationship between the two countries is in for a rough ride.

Why, then, despite recently confirming that the Indonesia relationship is Australia’s most important, did the Australian government press ahead with its “stop the boats” policy in this form?

At one level, this has been driven by domestic politics, the result of pressures to fulfil populist electoral bargains.

At the same time, we must assume that the government hopes the “shock” of the new policy will put a quick stop to the flow of asylum seekers across the region and into Indonesia itself, and that this ongoing problem will just go away.

There may have been some thought that by ramping up the stakes, Indonesia would be forced to take a more serious approach to co-operation on this issue. However, there is not much evidence of this having occurred so far.

More important are signs of an increasing impatience with Indonesia. These were initially evident in the uncompromising responses of Australian prime minister Tony Abbott and others in government – and in the conservative wings of the Australian media – to Indonesia’s demands for an apology over the spying incidents.

This willingness to take a firm position may simply indicate a judgement that there is not much Indonesia can do about a tougher policy in any case, and that this can be insulated from other elements of the relationship.

Floating below the surface, though, are some indications that a new mood is emerging. This is one that is more circumspect about Indonesia and its capacity – or willingness – to give much in return.

In other words, Australian leaders might be asking the question: is Indonesia worth the trouble?

Such a view has a long history and is not confined to conservative politicians. It has been most recently and forcefully put forth by former Labor leader Mark Latham, who argued last December that:

The best approach for Australia is to keep Indonesia at arm’s length, neutralising difficulties but not expecting goodwill or co-operation.

Attempts at co-operation, including in the control of people smuggling, will – in Latham’s view – always be undermined by a lack of will on the part of the Indonesians and the pervasive corruption of its officials. He refers to the flow of asylum seekers through Indonesia to Australia as an act of “political bastardry”.

There is no doubt that Indonesia has responded with indifference to many of Australia’s bilateral and regional concerns. For example, Australia’s attempts to revive its live cattle exports in the face of widespread evidence of animal cruelty were undermined both by nationalist efforts to restrict imports and by the reality that no-one seemed to have the will or the authority to regulate the industry in Indonesia.

Predatory fishing and environmental practices and uncertain and unpredictable legal and regulatory frameworks for investors have also long been points of irritation in the relationship.

But does this mean that Australia should retreat and keep Indonesia at arm’s length, expecting neither co-operation nor goodwill?

Such a retreat is not a serious prospect. In fact, it is the very weaknesses and uncertainty cited by Latham as reason to disengage that drive Australia’s relationship with Indonesia and make it so intense.

The Indonesia policies of previous Australian governments have always been motivated less by the prospect of lucrative opportunities than by the fear that Indonesia could become an incubator for political and social instability and terrorism, and for various forms of illegal trade.

AAP/Jon Faulkner

These factors are only going to get more intense in coming decades as Australia faces a vastly expanded set of problems. As they become increasingly globalised, flows of refugees and labour, the spread of criminality, problems of over-fishing, climate change, environmental degradation, breakdowns in biosecurity and public health epidemics will intensify.

Australia will not be able to deal with these if Indonesia is part of the problem.

Yet the very attempt to mobilise Indonesia within these policy agendas also provides much of the basis for the ongoing irritation between the two countries. Indonesia does not always share the anxieties and priorities of Australian policymakers.

In the people smuggling case, for example, the flow of a few hundred thousand asylum seekers through a nation of almost a quarter of a billion and into its disorganised cities and societies is not the same issue of public concern as it is in Australia. Nor have populist politicians tried to mobilise this issue. So, it is unreasonable to expect Indonesia to react with enthusiasm to our priorities.

And the frustrations of Australia’s policymakers are also stoked by the fact that government in Indonesia often possesses a limited capacity to enforce policy even where it exists.

It is doubtful, for example, that the Indonesian government has the authority to mobilise the complex networks of police, military and officials behind the effort to stop people smuggling. This is especially so given some officials were deeply implicated in the trade and were its beneficiaries.

Finally, in their attempts to strengthen the capacity of the Indonesian government – including that of the judiciary and regulatory agencies dealing with issues of corruption, criminality and various forms of illegal trade – reformers must engage with an architecture of power in which the very forces that are inimical to these reforms are prominent.

Powerful politico-business oligarchies, embedded in systems of political rents and predatory politics, have little interest (to say the least) in tackling corruption and creating honest courts or providing environmental regulation. For them, Indonesia works.

Can the cycle be broken? It is true that progressive forces within the bureaucracy and in the middle classes exist and have made progress. The problem, for both Australia as well as Indonesia, is that their grip on the commanding heights of Indonesia’s political and economic life remains tenuous.