By Nobuhiro Kubo, Linda Sieg and Phil Stewart

TOKYO/WASHINGTON (Reuters) - As Japan and the United States start talks on how to respond to armed incidents that fall short of a full-scale attack on Japan, officials in Tokyo worry that their ally is reluctant to send China a strong message of deterrence.

Military officials meet this week in Hawaii to review bilateral defense guidelines for the first time in 17 years. Tokyo hopes to zero in on specific perceived threats, notably China's claims to Japanese-held islands in the East China Sea, while Washington is emphasizing broader discussions, officials on both sides say.

Washington takes no position on the sovereignty of the islands, called the Senkaku by Japan and the Diaoyu by China, but recognizes that Japan administers them and says they fall under the U.S.-Japan Security Treaty, which obligates America to come to Japan's defense.

But even as Asia-Pacific security tensions mount, U.S. officials have made clear they do not want to get pulled into a conflict between the world's second- and third-biggest economies.

Prime Minister Shinzo Abe's government is alarmed at China's rapid military buildup. Beijing in turn accuses Tokyo of being a regional threat, citing Abe's more nationalist stance, his reversal of years of falling military spending and his visit to a shrine that Asian countries see as glorifying Japan's wartime past.

"Japan wants to prioritize discussions on China and clarify the respective U.S. and Japanese roles in the event of a 'grey zone' incident," said a Japanese government official, referring to less than full-scale, systematic military attacks backed by a state but still representing a threat to Japan's security.

Tokyo wants Washington to join in drafting scenarios for how the two allies would respond in specific cases, he said.

But Washington is worried about provoking China by being too specific, say Japanese officials and experts.

"The United States is certainly ambivalent about this because they think it would drag them into a confrontation and possibly a conflict with China," said Narushige Michishita, who was a national security adviser to the government of Junichiro Koizumi from 2001-2006.

Story continues

A U.S. defense official rejected the idea that Washington worries about antagonizing China but stressed that the guideline review is a broad exercise, including the Korean peninsula and global contingencies.

"There is a tendency to distil all this back to the Senkaku islands," the official said. "It's not about any particular contingency. It's about making the U.S.-Japan alliance more flexible and responsive to a security environment that's not as black and white as we were thinking about in 1997."

Singling out China, the official said, is "too simplistic a narrative".

GREY ZONE

Underlying Tokyo's concerns are worries that Washington might one day be unable or unwilling to defend Japan, despite President Barack Obama's strategic "pivot" toward the Asia-Pacific region. This fear is adding momentum to Abe's drive to beef up Japan's forces while loosening constitutional limits on military actions overseas.

If Washington does not get involved in specifically addressing the China threat, "it would undermine the credibility of the alliance and might end up encouraging China to be bolder," said Michishita, a professor at the National Graduate Institute for Policy Studies in Tokyo.

"U.S. policy makers will have to walk a thin line and try to strike a balance between maintaining credibility and deterrence, and preventing excessive involvement in the situation."

High on the agenda in Hawaii will be "grey zone" incidents. Japanese government officials offer such hypothetical examples as a landing of Chinese special forces disguised as fishermen on the disputed islands.

When the guidelines were last updated in 1997, North Korea's missile and nuclear programs were seen as the biggest threat. Japan was less nervous then about China's military expansion, and issues such as cyber-warfare barely existed.

The old guidelines "are too binary," said the U.S. official. "We're either in peacetime or we're on full contingency." This is "far too inflexible and rigid a framework" for today's threats, the official said.

Tokyo's strategic planners have become increasingly concerned about grey-zone incidents since Sino-Japanese tensions over the tiny uninhabited islands increased in 2012. Japanese and Chinese vessels and aircraft regularly play cat-and-mouse in the disputed areas, with Tokyo often scrambling fighter jets against what it says are incursions of its air and sea territory.

China's Foreign Minister Wang Yi said on Saturday there was "no room for compromise" with Japan on questions of history and disputed territory, "each inch" of which it would defend from its Asian neighbor.

DYSFUNCTIONAL DETERRENCE

The guidelines to update the U.S.-Japanese defense alliance, agreed to in October, coincides with Abe's push to bolster Japan's military and ease the constraints of the post-war, pacifist constitution on the country's armed forces. That includes his plan to lift a self-imposed ban on giving military aid to an ally under attack.

The update, which the two sides aim to wrap up by the end of the year, also follows years of Washington urging Japan to take on a bigger role in the alliance, the core of Tokyo's post-war security policy.

But American voters are weary of foreign wars after Iraq and Afghanistan and wary of being entangled in any new conflicts, experts say.

"U.S. public opinion is more negative toward involvement in foreign wars than even during the Vietnam War," said former senior Japanese diplomat Yoshiji Nogami, now president of the Japan Institute of International Affairs. "If the alliance is firm, then the chance of (America) being drawn in will be less, but this point is not fully understood by ordinary Americans."

Japan has its own headaches over grey-zone incidents. Government officials and many security experts say the authorities must close loopholes between situations where only Japan's Coast Guard and police can act and those where the military can be mobilized.

Examples, Abe recently told parliament, could include a foreign submarine lurking in Japanese waters despite repeated warnings to surface and identify itself or leave, and aggression against remote islands to which police or the Coast Guard could not promptly respond.

"A legal gap like that at a time when the security environment surrounding our country is getting tougher would render deterrence ... dysfunctional and put the people in grave danger," Abe said. A panel of Japanese security experts is expected to recommend revising laws to close that gap.

Washington wants to know how far Japan's military "can expand its roles, missions and capability," said another Japanese official.

U.S. involvement in grey-zone incidents could include intelligence, surveillance and reconnaissance, the U.S. official said. The guideline review will likely focus in part on strengthening cooperation in those areas, as well as "maritime domain awareness ... early on, possibly in a grey-zone kind of situation," he said.

That would be a more likely outcome than more direct military action by U.S. forces, said ex-diplomat Nogami.

Where grey-zone tensions are rising, joint intelligence, surveillance and reconnaissance are vital, he said. "The reason that is needed is to prevent the grey zone from becoming black."

(Writing by Linda Sieg; Editing by William Mallard and Jeremy Laurence)