By Stuart Grudgings

KUALA LUMPUR (Reuters) - The submerged reef would be easy to miss, under turquoise seas about 80 km (50 miles) off Malaysia's Borneo island state of Sarawak.

But two Chinese naval exercises in less than a year around the James Shoal have shocked Malaysia and led to a significant shift in its approach to China's claims to the disputed South China Sea, senior diplomats told Reuters. The reef lies outside Malaysia's territorial waters but inside its 200 nautical mile exclusive economic zone.

The latest incident in January, in particular, prompted Malaysia to quietly step up cooperation with the Philippines and Vietnam, the two Southeast Asian nations most outspoken over China's moves in the region, in trying to tie Beijing to binding rules of conduct in the South China Sea, the diplomats said.

Beijing's growing naval assertiveness could also push Malaysia closer to the United States, its top security ally, thus deepening divisions between Southeast Asia and China over the potentially mineral-rich waters.

Malaysia has traditionally played down security concerns in pursuit of closer economic ties with China, its biggest trade partner.

The James Shoal, which China calls Zengmu Reef, is 1,800 km (1,100 miles) from mainland China. It is closer to Malaysia, Brunei, Singapore, the Philippines, Vietnam and Indonesia - nearly all of Southeast Asia - than it is to China's coast.

Nevertheless, Beijing regards those waters as its southernmost territory, the bottom of a looping so-called nine-dash line on maps that comprise 90 percent of the 3.5 million sq km (1.35 million sq mile) South China Sea.

Pictures from China's state media on January 26 showed hundreds of Chinese sailors standing to attention on a warship's deck, backed by two destroyers and a helicopter that was reported to be at James Shoal.

Malaysia's navy chief denied the Chinese media reports at the time, telling state news agency Bernama the ships were far from Malaysian waters, which are rich in the oil and gas that power the nation's economy. He may have been able to deny the incursion because Malaysian forces did not monitor or sight the Chinese flotilla, security analysts said.

Story continues

But diplomatic and naval security sources have told Reuters the exercise by three warships, which included an oath-taking ceremony to defend China's sovereignty, almost certainly took place at or close to James Shoal.

"It's a wake-up call that it could happen to us and it is happening to us," Tang Siew Mun, a foreign policy specialist at Malaysia's Institute of Strategic and International Studies who advises the government, said of the recent incidents.

"For some time we believed in this special relationship ... James Shoal has shown to us over and again that when it comes to China protecting its sovereignty and national interest it's a different ball game."

MORE URGENCY ON MARITIME CODE

Neither Malaysia's Foreign Ministry nor the prime minister's office responded to requests for comment.

While Malaysia's public response to the January incident was typically low key, senior diplomats from other Southeast Asian nations said their Malaysian counterparts had been far more active since then in pushing for a common stance in talks with China over a code of conduct for the South China Sea.

Officials from the 10-member Association of Southeast Asian Nations (ASEAN) and China will resume negotiations in Singapore on March 18 after agreeing to accelerate talks last year that have made little headway so far.

The code is intended to bind China and ASEAN to detailed rules of behavior at sea, reducing the chance of an escalation in tensions that could lead to conflict. China says it is sincere in trying to reach an agreement.

Malaysia, the Philippines, Vietnam, Brunei and Taiwan claim parts of the sea. All are members of ASEAN except Taiwan.

Less than a week after the January incident, Malaysian Foreign Minister Anifah Aman made a previously unannounced private visit to Manila to meet his Philippine counterpart, the Philippine Foreign Ministry said. The South China Sea issue was discussed, a ministry spokesman said.

Then on February 18, officials from the Philippines, Malaysia and Vietnam held a meeting to coordinate policy towards China on the maritime dispute and code of conduct, a diplomat with knowledge of the talks in Manila said.

"In the past it was only the Philippines and Vietnam that were pushing for this meeting, but now we see Malaysia getting involved," said the diplomat.

At the unannounced talks, the officials agreed to reject China's nine-dash line, push for an early conclusion to the code of conduct negotiations and ask Brunei to join a meeting with the three countries in Kuala Lumpur in March, the diplomat said.

Malaysia's change in tack comes ahead of visits to Kuala Lumpur by Philippine President Benigno Aquino this week and U.S. President Barack Obama in April.

U.S. officials have also hardened their stance toward China over the South China Sea in recent weeks. On February 13, the commander of the U.S. Navy, Admiral Jonathan Greenert, said Washington would come to the aid of the Philippines in the event of conflict with China over the disputed waters.

Those sorts of comments could embolden some countries, said Hong Nong, deputy director of the Research Centre for Oceans Law & Policy at the National Institute for South China Sea Studies on China's Hainan Island.

"That will have an influence on ASEAN. In the past the U.S. never made it clear it was going to stand by whom," said Hong.

NEW MALAYSIAN NAVAL BASE

In March 2013, a similar exercise at the James Shoal by a four-ship Chinese amphibious taskforce rattled Malaysia and prompted it to make a rare private protest to Beijing.

"These two developments are very worrying for Malaysia's national security establishment," said Ian Storey, a senior fellow at Singapore's Institute of Southeast Asian Studies.

"We can anticipate there will be more of this kind of incident in the future. The PLA (People's Liberation Army) will show the flag in Malaysian waters and this will require Malaysia to recalibrate its policy."

Malaysia already appears to be doing that.

In October, it announced plans to build a navy base in Bintulu on Sarawak, the closest major town to the James Shoal, where a new Marine Corps, modeled on the U.S. version, will be stationed. Without mentioning China, the defense minister said the aim was to protect oil and gas reserves in the region.

In unusually blunt language, Prime Minister Najib Razak said in New York last September that China was sending "mixed signals" and could not afford to alienate its Asian neighbors.

Washington is expected to give advice and possibly training to help Malaysia set up its Marine Corps, Malaysian security analysts said.

"This is a very important development," said Tang, the foreign policy specialist, adding it could significantly deepen U.S.-Malaysia military ties.

U.S. naval commander Greenert told reporters he had discussed the formation of the new Marine Corps with his Malaysian counterparts during a visit to Malaysia this month, but said details on the new force were sketchy.

WORRY OVER CHINA REACTION

Despite its shifting stance, Malaysia will likely stop short of risking any chill in ties with China, which routinely says its ships patrol the region to protect the country's sovereignty.

Sources close to Malaysia's government said it is not considering joining a legal challenge the Philippines has lodged against China over South China Sea claims.

Manila has taken its dispute to arbitration under the U.N. Convention on the Law of the Sea and its lawyers say the tribunal in The Hague has the power to allow other states to join the action. China is refusing to participate in the case.

Malaysia has given the impression of seeing the South China Sea dispute as a hitch in an otherwise thriving and historic relationship. Najib's father, who was also prime minister, established Malaysia's formal ties with Beijing in 1974, the first ASEAN country to do so.

Malaysia offers a "more sober and highly nuanced way of resolving regional conflicts", the pro-government New Straits Times said last October before Chinese President Xi Jinping visited the country.

Economic ties have surged, with Najib and Xi setting a goal last year to triple two-way trade to $160 billion by 2017.

One senior Western diplomat said he expected no major shift in Malaysia's overall policy of balancing its alliances with Beijing and Washington.

"In principle they are committed to the ASEAN position and the code of conduct. But at the same time they worry about a China reaction," the diplomat said.

"They think they can cut a deal. China will not cut a deal. You can see that China is getting step by step more aggressive."

(Additional reporting by Manuel Mogato in Manila and John Ruwitch in Shanghai. Editing by Jason Szep and Dean Yates)