Six members of a local citizens’ watchdog group have filed a civil suit against the city of Osaka, demanding that Mayor Toru Hashimoto repay the nearly ¥700,000 bill the city footed after he canceled a trip to the U.S. last June.

Hashimoto scrapped the visit to San Francisco and New York following an international uproar in May when he said that Japan’s wartime “comfort women” system had been necessary at the time, and suggested that U.S. Marines in Okinawa could curb sexual violence by making use of “fuzoku” establishments, a euphemism for paid sex clubs.

Hashimoto’s remarks were condemned by the U.S. State Department and the San Francisco City Council, while the city’s mayor said he was disappointed in Hashimoto’s remarks. Nobel laureates and human rights’ groups in Japan and South Korea also criticized him.

But Hashimoto refused to apologize, and spent several weeks attempting to deal with the backlash, which finally damaged him and his party in the July Upper House election.

In October, members of Mihariban, a local watchdog group, asked city auditors to probe if Hashimoto’s comments violated local laws and whether he should be forced to repay a ¥690,000 cancellation fee for the U.S. trip that had been drawn from city coffers.

The auditors turned down the request to investigate, saying Hashimoto’s comments could not be said to be illegal. But they added the mayor needed to sufficiently recognize his responsibility for the cancellation. Nevertheless, Hashimoto did not repay the money, prompting Mihariban to file the lawsuit on Tuesday.

“If you continue to cause trouble by making crude statements like advising U.S. forces in Japan to use (paid sex) establishments, you can foresee they’ll invite a strong reaction from the U.S. It’s clear that, with his comments, the city was forced to pay damages in the form of a cancellation fee, and that this was illegal,” the lawsuit charges.

For his part, Hashimoto vowed to fight the lawsuit, saying that he did everything by the book and that free speech is being threatened.

The U.S. trip in mid-June was intended to celebrate Osaka’s sister-city relationship with San Francisco, and to discuss administrative reforms with New York officials.