Every tax season, scammers go after taxpayers on the internet, by email and over the phone to trick them into giving out their financial details. These frauds rake in billions of dollars each year.

The Internal Revenue Service estimates that it prevented $22.5 billion in attempted identity-theft tax fraud, but paid out $3.1 billion in fake refunds in 2014, the most recent year available.

Researchers from the Government Accountability Office said those estimates could be understated and the IRS should do more to strengthen its efforts to protect taxpayer information. (See chart below.)

The U.S. Treasury Inspector General for Tax Administration recently projected that victims collectively have paid more than $50 million to scammers posing as IRS officials since October 2013. The average amount lost is $5,200.

The IRS recently warned taxpayers about the most prevalent impersonation schemes that can affect them: Requesting fake tax payments. Scammers use automated messages to demand quick payment from taxpayers by threatening legal action. However, the IRS will never call to demand immediate payment, nor would the agency ask people to use a specific payment method, such as a prepaid debit card or wire transfer, as these scams do. "If you get a call from the IRS requesting you meet them with a cashier's check at 12th Street and Main, it's not the IRS," said Bill Smith, managing director of accounting firm CBIZ MHM. Smith has grilled scammers over the phone on behalf of his clients. Paying the "federal student tax." There is no such tax, but hucksters hound students and parents to pay this fictitious tax. The IRS has reported that some aggressive scammers threaten to report students to the police if they don't pay. Call their bluff, because real IRS agents never threaten to immediately bring in local law enforcement to have you arrested for not paying your taxes.

Sending fake notices related to the Affordable Care Act. Scammers are using confusion about Obamacare to get people to fork over money. The scam involves sending a false CP2000 notice to unsuspecting marks. People do receive CP2000 notices from the IRS when they fail to report any income, payments or credits or if they overstate certain deductions on an income tax return. However, a real CP2000 notice is not a bill. It only informs taxpayers of the proposed adjustments to income, payments, credits or deductions that IRS wants to make. "Verifying" tax return information over the phone. Con artists don't always trick you into sending them money. All they really need are critical details, such as Social Security numbers and bank account information, to steal your identity for a big payoff.

"Safeguard your Social Security number," Smith said. Never give out personal financial information to anyone who calls you claiming to be from the IRS. Generally, the IRS will first mail you a bill if you owe any taxes. Pretending to be a tax preparer. Official-looking emails are designed to fool taxpayers into thinking they're communicating with the IRS, tax preparers or tax software companies. Known as phishing, these emails and text messages are used by scammers to seek information that will help them file a false refund in your name. "Phishing scams are getting more sophisticated," Smith said. "To avoid them, just don't click on links and emails from places you don't know."

What to do if you're targeted by a tax scam