"I face challenges every time I run for office, it's nothing new," he said. "They might have four or five people in there. They will have to face their own primary. I can't worry about that, I can only worry about what we think we can do."