Though Mr. Wei insists that creating the Communist Party shrine was his own idea, locals have expressed concerns — with no direct evidence — that Mr. Wei is acting as a proxy for the Beijing government, a suspicion that has been amplified by recent revelations about China’s efforts to influence the domestic affairs of countries like Canada, Australia and New Zealand.

That possibility, Mr. Chen said, is alarming.

“While freedom of speech is important, this is also about the national security of Taiwan,” he said. “The government and people in Taiwan should consider this more seriously.”

The demolition comes as Beijing has been ramping up pressure on Taiwan, which China’s Communist government sees as part of its rightful territory but has never ruled. Under President Tsai Ing-wen, whose Democratic Progressive Party is skeptical of closer relations with China, cross-strait relations have worsened. This month, Beijing and Taipei exchanged new accusations of espionage.

In a statement on Wednesday morning, An Fengshan, spokesman for China’s Taiwan Affairs Office, said the demolition of the temple addition showed that “some people in the Democratic Progressive Party not only tolerate and ignore separatist behavior promoting ‘Taiwan independence,’ they also attack and persecute Taiwanese who favor unification.”

“Such an action must be subject to strong opposition and condemnation from people on both sides of the strait,” he added.

Reached by telephone on Wednesday, Lee Pai-jan, Mr. Wei’s wife, said it was “not convenient” for Mr. Wei to talk.

For now, though, there is relief in the village. Since 2012, after Mr. Wei took over the temple and evicted the nuns of Biyun Temple, the nuns have been living in a makeshift encampment adjacent to the temple property and making their case to move back to the temple, to no avail.