TOKYO—As Japanese nationalism is fueled by friction with neighbors over territories and World War II legacy issues, hostile demonstrations against the country's Korean residents are gathering steam, raising concerns among political leaders and setting off soul-searching among Japan's largely homogeneous population.

While attendance at the rallies is small and such extreme actions are far from entering the mainstream of Japanese politics, the demonstrations of nationalist activists using hate speech and intimidation have grown in size and frequency in recent months. One target has been the central Tokyo neighborhood of Shin-Okubo, known for Korean restaurants and shops selling South Korean pop-culture goods. Starting in February, groups of 200 or so demonstrators have descended on its busy weekend streets, waving Japanese flags and carrying signs that read "Roaches" and "Go Back to Korea." They shouted in unison: "Let's Kill Koreans," language that passersby told local television they found shocking.

Similar, though smaller, rallies have been held every weekend across Japan. While the demonstrations have raised tensions, there have been no reports of violence, beyond a handful of minor scuffles.

Alarmed, some lawmakers have started calling for new regulation to ban hate speech, a term unfamiliar to most in Japan where immigration is tightly controlled and racial and ethnic minorities—mostly descendants of Koreans brought to Japan before and during World War II—account for less than 1% of the population.

"When they started shouting 'Kill Koreans' on the streets early this year, I knew they had crossed the line," said Yoshifu Arita, an opposition lawmaker leading a debate in parliament along with a dozen colleagues. "This is something we can't overlook," he added in an interview.