Americans are "scared to death of the IRS," and the agency should be severely downsized to keep tax authorities in check, Rep. Roger Williams told CNBC on Wednesday.

"I'm one of those that would love to do away with the IRS. It's a rogue agency. It scares the heck out of people," the Texas Republican said on "Squawk Box."

"The IRS does a pretty good job of going after people that really have done nothing wrong," Williams asserted. He said he personally had a tough time with the agency over inheritance taxes, often referred to as the death tax.

Williams said the IRS came knocking "three days" after the 1990 death of his father who ran a car dealership. He said he worked out a plan to save the business but spent 20 years paying off the government.

The inheritance tax is double taxation, and "it's not fair," said Williams, a member of the House Financial Services Committee.

The House Republican blueprint for tax reform proposes to eliminate the inheritance tax and aims to transform the IRS into what it calls an "agency focused on customer service."

"The IRS, along with some other agencies, should be downsized and basically made a collection agency," Williams argued. "Collect what we're due. Not what we're not due."