China is likely to agree to new or tightened economic sanctions on North Korea or possibly curb its own assistance as its frustration with its ally grows, experts believe.

China summoned the North Korean ambassador and delivered a stern protest, and as after previous tests, the foreign ministry called for a calm reaction and denuclearisation talks. However, it stopped short of the harsh criticism it unleashed in 2006 when it described the North's first nuclear test as "brazen".

Beijing agreed to extend United Nations sanctions after the North launched a rocket in December, signing up to a security council resolution that warned of "significant action" against further missile or nuclear tests. It had signalled its unhappiness with Pyongyang in the runup to Tuesday's test, which was felt on the Chinese side of the border.

Even so, it will not cut off its neighbour and is likely to oppose the toughest measures sought by the US and others, analysts say.

China is North Korea's main trading partner and supplier of aid, providing an estimated half a million tonnes of oil annually as well as much-needed food supplies.

But analysts say it has become increasingly annoyed by the North's intransigence, particularly after its attempts to increase engagement.

"China tried to help the North to focus on economic growth rather than the military-first policy. Its logic was that if we could help them shift from military and ideological to national interests first, they would become more reasonable. That was China's basic logic: to make them negotiable," said Jin Canrong, a foreign policy scholar at Renmin University. "This time, after the third test, I think China's attitude will see some change."

Jin pointed to the speed with which the security council had convened a meeting. "In the past [China] would debate with the US about a meeting for several days, but today it will have one within 12 hours [of the test]," he said.

Jin said China would probably support economic measures but would shun security-related action such as signing up to the Proliferation Security Initiative.

North Korea and China were once said to be as close as "lips and teeth", but they have long had an ambivalent relationship weighted with mutual suspicion.

After Pyongyang tested missiles in 2009, He Yafei, then China's vice-foreign minister, told a US diplomat that North Korea was acting like a "spoiled child" because it wanted to engage the attention of the "adult" – meaning the US – according to cables obtained by WikiLeaks.

Analysts say Beijing fears that a collapse of the regime could lead to instability – with refugees trying to escape over the border or dangerous conflicts within the political elite – and, with reunification, the arrival of US troops on its doorstep.

It is also reluctant to lean too heavily on Pyongyang, particularly if that alienates the North further and encourages its pursuit of direct dealings with the US. Su Hao, of the China Foreign Affairs University, told the Global Times recently: "Most of the time the whips for North Korea actually fall on the back of China."

At the same time, its influence over the regime, though limited, has proved a useful diplomatic card to play. "In the last 10 years, China has felt frustrated about the North's behaviour many times and will have concluded from Kim Jong-un's behaviour that he is even more volatile and provocative than his father," said Shi Yinhong, another international relations expert at Renmin University.

He added: "It's still in a dilemma. I don't think it will abandon North Korea. It will probably support new sanction resolutions but not as strong as the US [would like]. It thinks they are counter-productive."

Rory Medcalf, director of the international security programme at the Lowy Institute, said a "very short, concentrated economic signal, such as cutting off oil for a couple of days" would send a signal to the North without destabilising it.

He added: "The constant debate within China was temporarily resolved in 2010 in favour of largely unqualified tolerance of North Korea's provocations … Since then, there has been such an accumulation of provocations. Secondly, China does not want multiple international crises on its hands at a time when its leadership should be focused on domestic problems. They already have a crisis with Japan [over the islands in the East China Sea]."

He said the new leadership under Xi Jinping might find it easier to change course because they were not invested in the old policy.

China's foreign ministry made its frustrations unusually visible last month when its spokesman Hong Lei, discussing North Korea's military, rocket and nuclear spending, told reporters: "We would also like to actively encourage the relevant country to develop economy and improve people's living conditions."

The nationalist tabloid Global Times, a state-run newspaper often used to float controversial ideas, said last week that the North should pay a heavy price for another nuclear test, adding: "The various kinds of aid it receives from China will be decreased for good reasons."