Just because the filing season is over doesn't mean scammers have stopped preying on taxpayers.

The Internal Revenue Service said Wednesday that fraudsters continue to impersonate IRS agents and demand bogus payments, despite warning of this scam last October.

The Treasury Inspector General for Tax Administration, which oversees the IRS, has received 90,000 complaints about this scam so far, and about 1,100 victims have been duped out of $5 million.

Related: 4 ways scammers can steal your tax refund

Some fraudsters tell victims they owe money and will be in big trouble if they don't pay immediately, while others tell taxpayers they are owed big refunds and they simply need to provide their personal information -- like a bank account number -- to claim those sums of money.

They may use a method known as spoofing so that the number on the caller ID appears to be the toll-free IRS number, and then use bogus names and fake IRS badge numbers to introduce themselves. To make the call seem more official, they often send an email to victims after the call to confirm the conversation.

Related: 'A scammer stole my tax refund'

Some threaten to send a victim to jail or revoke their license, hang up in a rage and then call back moments later pretending to be the police or DMV.

As convincing as they are, urgent and threatening phone calls are the first sign that this is a scam, the IRS says.

"[The] first contact with the IRS will not be a call from out of the blue, but through official correspondence sent through the mail," IRS Commissioner John Koskinen said in a statement. "A big red flag for these scams are angry, threatening calls from people who say they are from the IRS and urging immediate payment."

Koskinen said IRS agents will never ask for credit card or bank account information by phone, will never threaten enforcement action if a payment isn't immediately made, and they will not require a consumer to use a specific payment method -- like a prepaid card.

The IRS advises people to hang up and call its office or the inspector general directly to report the incident.