Washington (CNN) Rep. Steve King on Saturday opened his first town hall since sparking outrage over his latest racially insensitive remarks by insisting he isn't racist.

The Iowa Republican was removed from his committee assignments in the new Congress earlier this month for defending the term "white supremacist," the latest among numerous public statements widely seen as racist or insensitive.

"It's stunning and astonishing to me that four words in a New York Times quote can outweigh 20 some years of public service, 20 some years of giving you my word every day," King said in Primghar, Iowa, at an event where he addressed the partial government shutdown.

He said his experience made him think of Supreme Court Justice Brett Kavanaugh's contentious confirmation proceedings over allegations of sexual assault, where "at least he had accusers."

"Not one soul has stood up that said, 'Steve King has ever acted in a racist fashion, that he's ever discriminated against anybody,'" he added. "There's plenty of evidence out there to the contrary."

Read More