They are worried about crime. They don’t want to pay any more taxes. And they really, really miss Michael Bloomberg.

But to understand why many white voters are so down on New York City Mayor Bill de Blasio, consider that some of them said they believed the feeling was mutual.

“He’s so down on me,” said Gene Reilly, a 71-year-old Democrat...