You know tax scams are getting seriously out of hand when a criminal posing as an IRS agent calls one of the country's top tax-crime fighters, hoping to scare him into sending money.

Yet that's exactly what happened to Timothy Camus, the deputy inspector general for investigations at the agency that oversees the IRS.

The episode really drove home a key point Camus made to a Senate panel Thursday when he called the scam one of the agency's "top priorities."

"The criminals do not discriminate; they are calling people everywhere, of all income levels and backgrounds," he said.

His office, the Treasury Inspector General for Tax Administration (TIGTA), receives as many as 12,000 reports of these scam calls a week.

Since October 2013, when the agency started tracking complaints, more than 3,000 people have fallen victim to the scam, sending criminals $15.5 million, Camus said. The average heist is $5,000 per victim, but one individual actually lost $500,000.

Related: 12 biggest tax scams to avoid

The criminals typically call people unsolicited, claim to be from the IRS and assert that they owe taxes. The scammers tell the victims they will be criminally prosecuted if they don't make a payment immediately and threaten them with arrest, loss of their driver's license, deportation or some other seriously negative consequence.

They typically demand payment through prepaid debit cards, which are hard to trace. Once victims load up a card with their money, they are instructed to read the numbers from the prepaid card to the scammers over the phone.

While these phone scams have hurt victims in almost every state, Camus said the top five states in terms of total dollar losses so far are: California ($3.84 million), New York ($1.35 million), Texas ($795,884), Florida ($760,000) and Virginia ($648,363).

Related: IRS has $1 billion in unclaimed refunds

Phone scams top the IRS's so-called Dirty Dozen list of tax scams this year and Camus called them "the largest, most pervasive impersonation scam in the history of our agency."

Camus said the scam was the subject of "an ongoing multi-agency investigation."

As he told the criminal who called him, "Your day will come."

If you get what you suspect is a scam call, report it to TIGTA through its web site or call 800-366-4484.