TOKYO—The Japanese government has watched largely helplessly as protesters in China in recent days have damaged Japanese property and products. Officials are now wrestling with a more nettlesome prospect: how to respond if Chinese activists, possibly escorted by Chinese government boats, take their challenge closer to home by entering waters near disputed islands controlled by Japan.

The first line of response remains a small coast guard with no legal authority to expel foreign ships—a situation that has prompted calls from...