A former top U.S. diplomat on East Asia indicated Wednesday that Prime Minister Shinzo Abe should exercise restraint over a possible visit to the controversial war-linked Yasukuni Shrine in Tokyo, saying it could severely damage Japan’s relations with its Asian neighbors.

The remarks by Kurt Campbell, former assistant secretary of state for East Asian and Pacific Affairs, in a video message to a forum in Tokyo, came amid speculation that Abe may visit the shrine by Dec. 26, the first anniversary of his premiership.

Past visits by prime ministers and Cabinet members to Yasukuni, which enshrines convicted Class-A war criminals along with the war dead, have angered China and South Korea, both of which suffered Japan’s wartime aggression.

While noting that it is not the business of the United States to provide advice on the matter, Campbell said, “My own personal advice would be to go very carefully here. . . . Going to Yasukuni risks putting Japan’s relations with South Korea, China and other states at risk.”

“An inadvertent message sent by a senior leadership going to Yasukuni could set back, very substantially, Japan’s soft-power achievements in Asia and raise concerns about where Japan is heading,” he added.

On the ongoing discussions within the Abe government over allowing Japan to exercise the right to collective self-defense, Campbell said Japan’s willingness to work with other countries in preserving peace and security is a “natural development.”

“I think it’s logical and it’s in response to a dramatically changing security environment around Japan,” said Campbell, who resigned from his State Department post in February.

But he also stressed that Japan needs to explain to other countries what has motivated it to consider the changes now being envisioned within the government.