(CNN) They are very convincing when they call. They have a Washington phone number and can cite your financial history down to the cent.

They say you're under investigation, in danger of losing your home, or worse, your freedom -- unless you pay thousands of dollars on the spot.

But they're not real. And you're not in trouble. Not unless you take it seriously. This is a scam.

A big one. Federal authorities say it's the largest IRS impersonation scam they've ever seen -- swindling victims out of more than $15 million since it began in 2013.

"They have information that only the Internal Revenue Service would know about you," said Timothy Camus, deputy inspector general for investigations with the Treasury Department. "It's a byproduct of today's society. There's so much information available on individuals."

Using identity theft technology, the thieves have successfully victimized more than 3,000 people in the past two years, although the Treasury Department cautions that number is only documented cases and the true number might be higher. Camus said they've recorded more than 366,000 reports of contact with the scammers, and it's increasing at a rate of 10,000 to 12,000 a week.

Authorities believe the thieves are operating out of India, using phishing technology to make it appear they're IRS agents in Washington.

The Federal Trade Commission, which goes after scammers like these from a civil standpoint, and the Treasury Department, which leads the criminal probe, both have open investigations.

The largest loss reported was a staggering $500,000, Camus said. Most have lost about $5,000.

One of those victims was former NFL player Frank Garcia, who is now a sports radio host in Charlotte, North Carolina. When he got the call, it sounded so authentic, he left the radio station in a panic, scramming to get the money they wanted.

"The only thing running through my head is, I'm going to jail. I'm gonna be on television, in handcuffs, for tax evasion," he recalled. "I had to follow specific steps not to be arrested. That the authorities had been contacted and in fact, they are on the way and will be there in 30 minutes."

Garcia says he spent five hours driving to various stores around Charlotte, depositing $500 each time into a PayPal account set up by the woman on the phone. He ended up losing about $4,000.

He, and other victims, told CNN the swindlers never let their victims hang up the phone.

"I have never been arrested in my life and was very scared," said Kin Ko, a New Jersey resident who lost about $5,000 after he says the impersonator told him he was facing five years in prison, and the IRS was about to confiscate his assets: His car, his house and all the money in his bank account. The person had a badge number, read him an arrest warrant from a nearby police jurisdiction.

The thieves are incredibly smart and convincing. They harness stolen identities and use programs such as Google Earth to identity locations where their victims can transfer money.

"It sounded as legitimate as could be," said Al Cadenhead, a pastor in North Carolina who also fell victim. "They knew where I was. He told me where to go -- to the Rite Aid, up two streets turn left to the Rite Aid. The names of the streets, it was really just incredible."

Cadenhead told CNN he didn't come to his senses until he'd signed over $16,000.

"It was like I came out of a coma and realized this was not normal and this is not how you do business," he said. "I know other people who've heard the story say 'How did a guy with a Ph.D. fall victim?' I was the perfect victim. I've never been audited, never paid a traffic ticket. I don't know how to pay fines. How do I know they aren't stern and serious about everything?"

Camus said many people are afraid when they hear from the IRS, so they do whatever the caller says.

In December, federal authorities found and arrested two U.S.-based "runners" who admitted to transferring almost a million dollars from pre-paid cards to foreign bank accounts.

In many cases, victims are warned and stopped from sending more money by bank tellers or clerks at money wiring locations. Ko said that it was a bank teller who told him this was a scam when the impersonator asked him to go through a second round of depositing money.

How to get help If you think you have been a victim of an IRS impersonation scam, you can report it to TIGTA's web site or call 800-366-4484.

For Frank Garcia, the light came on when the woman on the phone asked him for another $8,000 after he thought his debt had been paid.

He says he decided to hang up and await what the woman promised would be an arrest -- which, of course, never came.

"I felt taken advantage of. I felt small. And I was naive," he said. "I wasn't aware. I didn't understand the system. And didn't blame anybody but myself for not taking more time to understand those things."

Often, Camus says, immigrants are targeted and threatened with deportation. The elderly are also a popular target, although the scam has grown so large, people of all ages, income levels and status are getting these calls.

Treasury Department officials say if you get one of these calls, the best and simplest way to handle it is to hang up.

Camus himself, a Treasury Department investigator, even got one. He told them, "Your time is coming."