China is not the first country to build military outposts on tiny rocks and reefs in the South China Sea. Several other countries laying claim to these contested flyspecks have done so before. But as with so many things, China has now gone much further, and faster, than anyone else. Just in the past few years it has built six substantial new islands in the Spratly Islands (which lie between the Philippines and Vietnam), some of them big enough to accommodate a military airfield. It’s quite an impressive feat of engineering, but no one paid much attention until just a few months ago, and when they did it wasn’t the structures but their strategic significance that grabbed attention. The startling aerial photos of new military bases being created out of dredged sand conveyed, more concretely than anything else, the image of China’s power on the march.

This increased focus did not happen by accident. Since earlier this year Washington has mounted a deliberate campaign of drawing attention to what China has been doing in these waters. The aim has been specifically to stoke fears about China’s growing power, and encourage other governments to support the US in pushing back against it. Australia has been one of the key targets of this campaign. Indeed, the first salvo was fired in Canberra, when the US naval commander in the Pacific, Admiral Harry B Harris Jr, speaking at a naval conference in March, described China’s new islands as a “great wall of sand”, and denounced them as a threat to regional peace and stability.

“Great wall of sand” is a clever phrase. It conjures a telling image of these lonely outposts as both formidable and fragile. Physically there is not much to them, and it will be interesting to see how they fare in the region’s notoriously savage typhoons, or how long they would survive as military bases in the first hours of an escalating conflict. But they are also, as Harris implied, a formidable symbol of China’s growing power and ambition in Asia. And in the months since his speech the new islands have also become a critical focus of the American response to China’s rise, and of the huge questions that now hang over US–China relations, and hence over the future of the Asia-Pacific region. These few acres of dredged sand have become key chess pieces in the new power politics of Asia. They are now tokens in a titanic contest over regional leadership. A contest of a kind and on a scale we have not seen for more than a generation.

The key to understanding what is happening now in the South China Sea, and what it means for Australia, is to recognise that it has almost nothing to do with the disputed reefs and rocks themselves, with the economic riches that may lie under the waters around them, or even with the security of the major shipping routes that run close by. It has everything to do with the deadly serious rivalry between the world’s two strongest powers, determining who sets the rules in Asia and who enforces them. This is old-fashioned power politics of the most raw and dangerous kind, and it is all the more dangerous because most of us still haven’t realised it is happening.

How do such inherently worthless specks of rock and coral become so central to Asia’s power politics and the region’s future? The answer lies first in Beijing, where decades of remarkable economic growth have transformed China’s view of its place in Asia. When Richard Nixon met Mao in 1972, China recognised that it could not grow wealthy and strong as long as it confronted both Soviet and American power. So Mao decided to accept America’s place in Asia and acknowledge its regional leadership. That won China US support against the Soviets, but even more importantly provided access to US capital, technology and markets to build China’s economy, and ensured that its Asian neighbourhood would remain stable and secure. China accepted US leadership as a means to achieve that end. However, China never accepted US regional primacy as anything more than a regrettable but expedient accommodation of the then current realities of power.

To Beijing its primacy now seems within reach, probably much sooner than anyone imagined. China’s rise over the past 35 years is the primary strategic event of our time, and yet it is already so familiar that we lose sight of just how remarkable it is, and how far it still has to go. Measured in terms of purchasing power parity, which factors out differences in prices and exchange rates to compare how much is actually produced, China has already overtaken the US to become the world’s largest economy. Even measured in the more conventional market exchange-rate terms, China’s economy will overtake America’s within a decade. And though it faces many problems, the odds are that China’s economy will keep growing fast enough to pull far ahead of the US over coming decades. In July the Economist estimated that by 2050 the US economy will be only 70% the size of China’s in market exchange-rate terms.

Ultimately wealth is the foundation of national power, and just as power shifted to the US a century ago as its economy grew, power is shifting to China today. Already China is far richer, relative to the US, than the Soviet Union ever was at the height of the Cold War. But for a long time, and until quite recently, China stuck to Deng Xiaoping’s injunction to bide its time and hide its power, until the time came that its claim to leadership would be irresistible. Over the past few years, under President Xi Jinping, it seems that that time has come. In Beijing they call this “a time of strategic opportunity” to challenge US leadership in Asia and build “a new model of great power relations” in Asia, with China at its head.

As Xi told Australia’s parliament in November last year, China now sees itself as “the big guy” in Asia, and wants others to see it the same way. The “new model” it seeks is one in which the US has much less power and influence in Asia, and China has much more. The rest of us might not like this, because the old US-led order has worked so well for us, but we can hardly be surprised that China feels differently. It is just doing what every other rising power in history has done.

It’s easy for China to say it wants to displace the US from sole regional leadership, but how can Beijing do this? We can see the answer by looking at its recent actions. China’s strategy is based on the fact that because the US is an outside power its leadership in Asia depends on formal and informal alliances with countries in the region. Beijing appears to have decided that the best way to undermine US leadership is to weaken those alliances. By applying carefully graduated degrees of pressure to US-aligned countries like Japan and the Philippines over long-running territorial disputes, China is trying to show that the US is no longer willing to confront China on their behalf. The more Beijing succeeds, the more US leadership in Asia dwindles, and the further China’s power and influence will grow.

The construction of the new islands in the Spratlys is just one more step in a strategy to present Washington with the invidious choice between confronting China and showing its weakness by letting down its allies. Of course, this whole strategy is based on a gamble that the US will choose the latter. If Beijing bets wrong, and the US does stand firm, then it would be China that has to choose between confrontation and backdown. But this is a risk China is willing to take, because it’s pretty sure the US will back off. Many people are bewildered that China would think this, knowing the US’s record for tough talk and tougher action, but so far China has been broadly proved right. Time will tell whether this will remain true, but China seems confident that it will, and it is worth asking why.

Part of the answer lies in the pattern of US strategic stumbles elsewhere in the world over the past decade: Barack Obama’s muddles in Libya, caution against Islamic State, inaction over Syria and refusal to act more decisively against Vladimir Putin in Ukraine. Beijing also notes that the US has also failed to stop North Korea getting nuclear weapons, mediate between Israel and the Palestinians, and do more than hit the pause button on Iran’s nuclear program. And then there are the bigger failures in Iraq and Afghanistan. It is not surprising that the Chinese doubt that the US is really serious about defending its leading role in Asia when it has apparently failed to do so in the Middle East and Europe.

But there are deeper reasons why Beijing is so confident that the US will back off from Asia. As China has become wealthier and stronger over the past 20 years, the costs and risks to the US of a conflict with China have grown. Back in 1996, when they last eyeballed each other (over Taiwan, after the Chinese carried out intimidating missile tests), it was China that backed down. But a clash at that time would have cost China far more than the US, both economically and militarily. China’s economy would then have been hit much harder than America’s if trade and investment links had been shattered by war. Today America would be hit just as hard, because China’s huge economy is now at least as important to America as America’s is to China.

Likewise, 20 years ago the US could send aircraft carriers and marine forces into combat against China with relative impunity, but since then China has focused its massive military build-up on its ability to target US forces operating in the western Pacific close to China. The People’s Liberation Army is still no match for the US military anywhere beyond its own backyard, but there it can do much more than before to stop the US projecting its power. If a conflict flared today, the US would no longer score a rapid and decisive victory. It would most likely face either a costly and humiliating stalemate or a dangerous escalation towards a bigger conflict. Beijing believes the risks and costs will be much more evenly balanced next time the two countries square up. This means China has the psychological and strategic edge, because it cares more about the outcome. It knows that Washington is determined to remain Asia’s primary power, but China believes it is even more determined to take America’s place. The simple facts of geography mean that primacy in Asia means more to China, so it is more willing to accept the costs and risks of confrontation. China assumes the US understands this. The problem is that American policymakers, even now, do not agree. They still think that if the US stands firm China will back off and go back to accepting US leadership in Asia.

Early this year the Obama administration decided it was time to show China, and everyone else, that US resolve in Asia remains firm. And it chose China’s new islands in the South China Sea as the place to do it. Obama wanted to demonstrate his determination to resist China’s challenge, to reassure Asian friends and allies about US strength and resolve, to stoke anxieties about Beijing’s growing assertiveness, and to shore up its support against China. It may be no coincidence that this decision came so soon after Washington suffered a humiliating diplomatic setback when it tried and failed to dissuade its friends and allies from joining China’s new Asian Infrastructure Investment Bank. US policymakers assumed that countries would willingly sacrifice the economic opportunities offered by China’s initiative in order to deny Beijing a geopolitical win at Washington’s expense. They underestimated China’s pulling power.

This mistake fits a pattern. US policymakers and pundits have consistently misjudged the seriousness of Beijing’s strategic and political challenge. They have been convinced that China is still too poor, too fragile politically, too weak militarily and too friendless diplomatically to seriously contend with the US. And many have found it hard to believe that China really wants to try, because American primacy has been good for China. Why wouldn’t the Chinese want it to last? So when Beijing started to challenge the US overtly a few years ago, the consensus in Washington was that China was just trying it on. The American pundits convinced themselves that America’s post–September 11 preoccupation with the War on Terror had misled Beijing into thinking that the US was no longer interested in Asia, and it could just step into America’s shoes, but they believed Beijing would back off as soon as Washington made it clear it was wrong.

This was the thinking behind Obama’s 2011 Pivot to Asia speech, given while he was in Canberra. The Pivot was Obama’s declaration that the US was not stepping back. He said he was determined to use “all the elements of America’s power” to maintain US leadership in the region. But there was nothing more to it than that. The Marine deployment to Darwin that was announced at the same time was one of a handful of symbolic gestures that were designed to lend the Pivot some substance, but did absolutely nothing to redress the real shift of power away from the US in Asia. It would only have been enough to make China back off if Washington had been right that China was just “trying it on”. But the US was wrong about that.

Everything China has done since the Pivot speech has been designed to show that Washington was wrong. Five months later, China launched the first of its direct challenges to US resolve in the East China and South China seas. It used armed ships to muscle the Philippines out of disputed waters around the Scarborough Shoal, which had traditionally been under Philippine control. When Manila asked for military support, Washington refused as it was not willing to risk a confrontation with China over an uninhabitable reef. Beijing won its point.

Since then, as we have seen, China has maintained continual shifting pressure on US friends and allies, including Vietnam and Japan, over disputed maritime claims. China’s aim is to show the weakness of America’s response and the hollowness of Obama’s pledge. But only in the past few months has Washington started to realise what is happening and think about how to respond. The US thought the answer might lie in China’s new islands.

It is far from clear that the US was right. Appealing as it must have seemed to use China’s island-building as the place to draw the line, the choice has turned out to be highly problematic. First, it’s hard to explain what exactly the Chinese are supposed to be doing wrong. Their island-building is not in itself illegal. The wider implications for maritime jurisdiction of these new bits of land are unclear, but these are hardly questions of fundamental geopolitical importance. Above all, and despite Washington’s efforts to suggest the contrary, there is no reason to fear that these new islands pose any fresh threat to freedom of navigation in the waters through which so much of the world’s trade passes. The island-building program is clearly intended as an intimidating display of China’s growing wealth and power, which is hardly likely to win it any friends. But that does not make it as plainly wrong as Washington would wish.

The second problem is what exactly the US can do to make China stop. Tough talk alone just plays into China’s hands if it is ignored. To score a win, Washington has to find a way to convince or compel China to take a visible step backwards. Someone in Washington must have thought about this question before they decided to make this a test of US strength and leadership, but it is not clear they found an answer. Admiral Harris’ “great wall of sand” speech in Canberra launched a carefully planned campaign to draw the world’s attention to China’s new islands, and promise a tough and effective US response. The campaign included the now-notorious and quickly revoked announcement that B-1 bombers would be deployed to Darwin as part of Washington’s response to China’s island-building, and the highly publicised flyover of the new islands by a US Navy surveillance plane with CNN aboard. It culminated at the annual Shangri-La Dialogue regional security conference in Singapore at the end of May, where even the US secretary of defence, Ashton Carter, declared that China’s actions were unacceptable.

But this is all just talk, and Beijing has simply ignored it. Shortly after the Singapore meeting, China’s foreign ministry coolly announced that the construction of new islands was almost complete, but that work would continue fitting them out for both civilian and military purposes, directly defying US demands. Washington has hinted that it may launch new patrols to assert its right to enter air and sea space around the new islands, but that won’t stop China building and equipping them. Instead, Beijing may fulfil its threat to retaliate by declaring a new Air Defence Identification Zone over the area, as it did in the East China Sea last year. And what would Washington do then?

The reality is that the US can do nothing to stop China building islands or turning them into military bases unless it is willing to take the kind of decisive action that would most probably lead to a military clash, risking escalation to a wider war. And, as Beijing predicted, that is something the US is not willing to do. So the past few months’ tough talk has only underlined Washington’s weakness, just as Beijing hoped. It seems that, even now, people in Washington still do not understand the challenge they face from China. Perhaps that is because these issues are unprecedented for today’s policymakers. Not since 1972 has the US faced a serious rival in Asia, and not since the 1880s has it confronted an economic peer. Almost everyone in the US political and foreign-policy establishment takes it for granted that the US will always lead in Asia, and that its leadership can be sustained at a price it is willing to pay. That is why they are so perplexed by the stark reality that, when the crunch comes, they find themselves compelled to risk an armed clash with China to affirm US power and leadership in the South China Sea. And they find they aren’t willing to accept that risk.

Of course, the flyspecks of the Spratly Islands and second-tier allies like the Philippines matter much less than a major ally like Japan, so it is tempting to think that Washington would be much less risk-averse in a higher stakes stand-off involving Tokyo. Everyone agrees that the US position in Asia depends absolutely on the credibility of its commitment to Japan’s security. That is why, after much prevarication, President Obama last year promised that the US would stand by Japan if push comes to shove with China over ownership of the Senkaku Islands, which lie between China, Taiwan and Japan. But what would happen if that promise were put to the test?

Imagine the White House Situation Room as the president weighed advice about whether or not to send US armed forces into action in an escalating Japan–China clash over the Senkakus. From one side he would be told that the US simply had no choice but to support Japan even if that meant war with China, because to do anything less would be to abandon US leadership in Asia. But from the other side he would hear that there was no realistic prospect of a swift, localised military victory over China. Even before it could deploy forces to the Senkakus, the US would first have to launch a massive campaign of strikes against military bases throughout China, and China would be sure to retaliate against US bases around the Pacific. There would be a real risk of a full-scale Pacific war, and a credible risk that it would become a nuclear war, as well as the certainty of catastrophic economic disruption. Few people believe that any president would be willing to pay those costs, so unless the president were willing to gamble that the Chinese would back off, he or she would have to back off first, whatever the costs to US leadership.

It is by no means inevitable that a US president will face a decision like this over the coming years, but it is far from improbable. This is the reality of power politics in Asia today. The world’s two most powerful states are contending over diametrically opposed visions of Asia’s future order and their roles in it. The US seeks to remain the primary power in Asia, and China seeks to replace it. And both see the outcome as essential to their future prosperity, security and identity. The stakes could not be higher. Such fundamental differences are not always resolved by war, but wars – major wars – have most often broken out over differences such as this. Reducing the risk that this will happen in Asia must be the primary imperative for policymakers everywhere.

For the US, that means doing whatever can be done to minimise the risk that a president will one day face the choice described above. Instead of wishing the problem away by assuming that China is neither willing nor able to confront US power, it means understanding that China is at least as committed to changing the Asian order as the US is to preserving it.

Today there are signs that American policy analysts are at last starting to think about these things. We can see the beginnings of a more realistic debate about the nature of China’s challenge and how to respond to it.

In the past few months a number of essays and books have acknowledged that the US needs to take China more seriously, and they propose new approaches to dealing with the situation. Predictably, these proposals fall into two categories. Some, such as a paper sponsored by the highly influential Council on Foreign Relations, advocate a hard line. They argue that the US should stick to the Pivot’s objective of preserving US primacy in Asia, but recognise that this will require a much bigger, tougher effort than has so far been understood. They say that the US should accept that escalating strategic rivalry with China is inescapable, and should commit itself to do whatever it takes to confront and contain this new rival, just as it did with the Soviet Union, and hopefully with the same result. It is helpful to get these arguments on the table, because they give Americans a more realistic idea about what that would involve.

The second group of new ideas heads in the opposite direction. A number of experts argue that Washington should try to avoid escalating rivalry with China, by sitting down with Beijing to negotiate agreements on the many issues of contention between them. Interestingly, the most discussed of these new arguments comes from our own Kevin Rudd, who now holds a prestigious position at the heart of America’s China debates as head of a think tank sponsored by the Asia Society of New York. His paper, prepared while he was with the Harvard Kennedy School, shows Rudd at his analytical best. In a few pages he sets out why the US must be willing to make room for China and suggests how to start the process. But Rudd is not alone: a number of prominent American scholars have started making similar arguments. At last, a serious debate about China is underway in the US.

And what about Australia? To understand our position we have to go back to 1996, when John Howard came to office determined to bind Australia even closer to the US and maximise Australia’s economic opportunities in China. He quickly confronted a problem when his robust support for Washington in the Taiwan crisis of that year froze contact with Beijing for months. To resolve this, Howard made China a promise: Australia would always remain an ally of the US, but nothing Australia did as a US ally would be directed against China. This was good enough for Beijing, and the relationship flourished accordingly.

It turned out to be easy for Howard to keep his promise, because while he was prime minister Washington remained almost as keen as Canberra to avoid friction with Beijing. After September 11, US attention was seized by the War on Terror. Beijing trod softly, so President George W Bush could largely ignore China’s rise while Howard burnished Australia’s alliance credentials in Iraq and Afghanistan. No one in Washington sought Canberra’s support against Beijing, so Howard could truthfully say, as he so often did, that Australia did not have to choose between the US and China.

But Labor found things harder soon after it took office in 2007, when Beijing became more assertive just as the newly elected President Obama began to swing US attention away from the Middle East. Obama soon focused on China, and by the time of his 2011 Pivot speech it was clear that he expected Australia’s support to help convince China to abandon its challenge to US regional primacy. The simultaneous announcement that US Marines would become the first US combat forces to routinely operate through Australia since World War Two was just as clearly intended to show that Australia was willing to give that support. Since then, as Washington has struggled to respond to China’s tactics, its expectations of Canberra’s support against Beijing have grown. Meanwhile, China has become ever more important to Australia’s economy. The policy dilemma that Howard managed to evade after 1996 has now become inescapable. Australia can no longer assure China that nothing we do as a US ally is directed at China, because for Washington the alliance today is all about acting together to resist China’s ambitions.

Canberra has simply gone into denial about this, maintaining that nothing has changed since Howard’s day. Over the past five years both Labor and Coalition governments have enthusiastically supported the Pivot while pretending to believe that it is not aimed at China. Julia Gillard welcomed the Marines to Darwin while energetically talking up the relationship with Beijing, and Tony Abbott, after a few wobbles, has learned to follow her example. “We don’t have to choose” remains the slogan on both sides of Australian politics, even as the pressure mounts from both China and the US for us to do exactly that.

This pressure was very clear late last year, when Obama decided at the last minute to make a big speech about China in Brisbane, just two days before President Xi Jinping was to make his own big speech in Canberra and finalise the China–Australia Free Trade Agreement (FTA). The headlines from Obama’s speech centred on criticisms of Tony Abbott’s climate policy, but its real message was aimed at China, and at the Australia–China relationship. In a blatant attempt to undercut Xi’s visit and the FTA hoopla, Obama delivered the most direct and sustained criticism of China’s strategic aims in Asia by any US president since before Nixon’s 1972 visit. Tony Abbott simply ignored him, and so did Xi Jinping, but the problem will not go away. It reappeared in May with the injudicious announcement that US B-1 bombers would be deployed to Darwin in reaction to China’s island-building. Tony Abbott himself was swift to contradict him, denying specifically that US military deployments to Australia were directed against China. Channelling John Howard’s promise to Beijing 19 years ago, Abbott declared that our US alliance “is not directed against anyone”.

For Washington today this is simply no longer true, or acceptable. The US may have no clear strategy for dealing with China, but it is absolutely clear that it expects unqualified Australian support in whatever it decides. That means that, unless it simply withdraws, Australia is going to find it harder and harder to avoid choices between damaging our alliance with the US and damaging our relationship with China. It is unclear just how far Abbott and his colleagues are deluding themselves, as well as the rest of us, when they deny this. It seems incredible that they should have missed the perfectly clear evidence of escalating rivalry over recent years, or that they could still cling to the hope that China can be persuaded to accept US leadership in Asia indefinitely. It may, of course, be a bit of both.

Of course, Abbott might well say that we have got away with it so far. We have signed our FTA with China while keeping our place as America’s trusted ally, so why worry? But this ignores the trend. Unless something happens to change that trend, we must expect strategic rivalry between the US and China to keep escalating. As that happens, they will draw closer and closer to the kind of break in relations that forces us to a choice that Canberra’s evasions and half-truths cannot bridge.

What might change the trend of escalating tension? If, like me, you put little faith in the idea that China will back off completely, then there seem only two practical possibilities. One is for the US to step back and abandon any major role in Asia, which would be a very bad outcome indeed for Australia. The other is for the US and China to seek agreement on a new order in Asia, which accommodates some of China’s ambitions but preserves a major role for the US. Given that the golden era of uncontested US primacy is now irrevocably gone, this is surely the best credible outcome for Australia. But today Canberra is doing nothing to make it more likely. On the contrary, by pretending to the US that we will support it against China, and pretending to China that we will stay neutral, we encourage both to believe they can get what they want without accommodating the other. That makes any kind of agreement between them less likely, and only encourages the escalating rivalry we need, above all, to avoid.

This leaves Australia with some big questions. What could we do to help avert the risk that the US and China will drive us towards having to choose between them? What would we do if events conspire to force this choice anyway? And how would we fare if, as seems increasingly likely, the Chinese are right and Americans decide that the game in Asia is not worth the high stakes, and choose to step back from Asia in the Asian century? It is clear enough that the answers to these questions would have revolutionary implications for our foreign policy, but they present more pressing issues for our defence policy.

For 40 years Australian defence planning has assumed that the US would remain both predominant and unchallenged as the leading strategic power in Asia. That assumption has determined the military force we have built and the amount we have spent on it, because it has set a low ceiling on the kinds of support we might need to provide the US in Asia, and the kind of war we might need to fight by ourselves. Now when we look, as defence planners must, decades ahead to the middle of the century, these assumptions are no longer a credible basis for deciding the kinds of forces we might need. There is a real prospect that we might find ourselves without a great and powerful friend in an Asia riven by major-power rivalry. Our political leaders have not even begun to consider what that means for our defence strategy. It is all too hard.