Story highlights Rep. Reid Ribble is tearing into Donald Trump

The Wisconsin Republican is a longtime critic of the now-presumptive Republican nominee

(CNN) A Republican congressman is accusing his party's presumptive presidential nominee of being a "racist," ratcheting up the criticism over Donald Trump's remarks about an Indiana-born federal judge of Mexican heritage.

Three-term GOP Rep. Reid Ribble of Wisconsin told CNN on Wednesday Trump's comments -- that federal judge Gonzalo Curiel could not rule fairly in a Trump University case because of his Mexican heritage -- is a "racist" remark, putting him in line with House Speaker Paul Ryan's criticism.

But Ribble took the matter one step further.

"His comments over the weekend are authenticating what I believe is the man's character," Ribble said. "Something that walks like a duck, talks like a duck, is likely to be a duck. If you continue to say what I believe are racist statements, you're likely to be a racist."

Ribble, who is retiring, said he doesn't plan to vote for Trump and is weighing whether to back the Libertarian Party ticket.