The latest, dangerous standoff between Japan and China, which brought violent protests across China, was sparked by Tokyo's decision to "nationalise" a chain of disputed, gas-rich islands in the East China Sea. But the row is just one manifestation of a deeper, multi-faceted, and almost institutionalised confrontation pre-dating the second world war. Coincidentally, Tuesday marks the anniversary of an infamous event that raised the curtain on the modern era of Sino-Japanese tensions.

The Mukden, or Manchurian Incident of 18 September 1931, became the pretext for Imperial Japan's invasion of northern China. The dynamiting of a section of track belonging to the Japanese-owned South Manchuria railway near Mukden (now Shenyang) was blamed by Japan on Chinese insurgents. In truth, the Japanese military had itself deliberately staged the explosion, a tactic copied by Hitler before he attacked Poland in 1939. Within months, Japan had created a puppet state in northern China.

A long, sorry history of conflict ensued, and many decades after Japan's defeat in 1945 Chinese commentators continue to complain that, unlike Germany, Tokyo has never fully come to terms with its wartime actions, let alone shown genuine remorse. Rows erupt periodically over the Yasukuni shrine in Tokyo, which, in Shinto belief, houses the spirits or souls of soldiers who died fighting for the emperor. Past visits to the shrine by senior Japanese politicians invariably aroused Chinese fury.

The sharpening dispute over the Senkaku islands, known as Diaoyu in China, is the most recent product of this old narrative of violence, hatred, fear and grief that continues, sporadically, to obstruct both nations in their efforts to forge a more stable, trusting relationship. But understanding its roots does not render it less potent. The Americans, who still view themselves as the Asia-Pacific's leading power, are increasingly nervous.

Leon Panetta, the US defence secretary, currently visiting Japan, pointedly warned that "provocative behaviour" by either side could lead to misjudgments, violence and, potentially, open warfare. "It is in everybody's interest … for Japan and China to maintain good relations and to find a way to avoid further escalation," Panetta said. In this he echoed Hillary Clinton, the US secretary of state, who this month stressed the need for restraint and dialogue between China and several south-east Asian countries with which Beijing has territorial disputes.

China's unhappy legacy of foreign occupation and its sense that an unrepentant Japan is party to a US-orchestrated geo-strategic conspiracy to contain or limit its development as a great power also form part of the backdrop to the Senkakus standoff. While claiming he was not taking sides, Panetta confirmed in Tokyo that the islands were covered by the Japan-US security and defence treaty. That means, in theory at least, that Washington is bound to help Japan defend territory that China says has been illegally seized.

The Obama administration's pivot towards Asia, its concern to defend itself and its friends against nuclear-armed North Korea, China's ally, and its active pursuit of expanded security and commercial alliances embracing countries such as Vietnam, the Philippines and Indonesia are all factors influencing the way China's leaders think about their country's security and relations with Japan, Washington's key regional partner and military springboard. When Clinton declared two years ago that freedom of navigation in the South China Sea was a US "national interest", Beijing viewed it as an outrageous provocation, just as it now views Tokyo's "intolerable" decision to purchase the Senkakus.

When this or similar disputes have flared in the past, days of rage, such as were suffered by Japanese nationals living in China's cities at the weekend, have invariably been followed by gradual de-escalation and diplomatic fence-mending. The Chinese government's apparent ability to turn street protests on and off has often given an air of artificiality to such episodes. Hard-headed self-interest and pragmatic considerations, such as $340bn-worth of annual bilateral trade and the dreaded possibility (for the Beijing leadership) that Chinese street protesters might turn on their own government, usually prevail in time, and tempers cool.

This time it may be different, mainly because of political shifts inside Japan. The Senkaku dispute has been close to the boil for months. The prime minister, Yoshihiko Noda, took the decision to buy the islands to head off a more destabilising but popular proposal not only to acquire them but also to begin their active development. China's reaction, including unleashing mass protests, sending ships to the area and threatening trade sanctions, was harsh but not unusual.

Under pressure from nationalist opinion, Noda has taken a surprisingly inflexible public position: "There can be no doubt that the Senkaku islands are part of Japanese territory, both under international law and from a historical point of view. The Senkakus are under the effective control of our nation, and there is no territorial issue with any country over the islands," he told the Yomiuri Shimbun in July. He is now calling on China to return to "calm" dialogue, without offering any concessions.

"The confrontation between Japan and China … has escalated to a truly dangerous level," wrote the former US diplomat Stephen Harner in Forbes. "Objectively it must be stated that it has been Japan that has done the most to raise tensions. Further escalation cannot be in the interests of either side. While his [Noda's] leadership in domestic policy matters has generally been laudable, even brilliant, in relations with China on this issue he seems captive to interests that would lead Japan into a trap."