"God, it's just a very vexing problem," he says. "I don't want to feel like I'm a victim of the SPLC. I'm not happy about [the listing], but it's what they do, and it's their right."