KOLKATA—They hang around Sudder Street, a backpacker ghetto, or loiter outside Mother Teresa’s Missionaries of Charity house—both hunting grounds for their quarry: Japanese tourists.

These groups of Japanese-speaking Indian men take advantage of the visitors’ cultural idiosyncrasies and lack of English-language skills to isolate and then steal from them, according to Japanese residents of Kolkata and some of the men themselves.

“Two...