Uncle Sam needs some help collecting unpaid taxes, so the Internal Revenue Service will now use private companies to tackle this growing backlog of debt.

The IRS announced earlier this month that it’s hired four debt collection agencies to round up outstanding payments from taxpayers who’ve been contacted numerous times and still haven’t paid.

The new private debt collection program is starting off slowly: At first, just a few hundred taxpayers a week will receive mailings and subsequent phone calls. That will grow to thousands of people a week later in the spring and summer.

These debt collectors won’t simply call people out of the blue. Taxpayers with overdue tax bills will always receive several collection notices from the IRS through the mail before their accounts are turned over to the private collectors. The collection agencies will then send a letter of their own, informing the taxpayer that their account has been transferred to them. These companies must clearly identify themselves as working for the IRS in all communications.

“The IRS is taking steps throughout this effort to ensure that the private collection firms work responsibly and respect taxpayer rights,” said IRS Commissioner John Koskinen in a statement. “The IRS also urges taxpayers to be on the lookout for scammers who might use this program as a cover to trick people.”

The IRS says it will not assign accounts to private collection agencies involving certain types of taxpayers, including: minors, those in combat zones, victims of tax-related identity theft, accounts that are subject to installment agreements or classified as an innocent spouse case.

Related: IRS Warns Tax Refunds May Be Delayed for Some Americans

Like all debt collectors, those contracted to recover back taxes must follow the Fair Debt Collection Practices Act which spells out when they can call, whom they can call, and what they can and cannot say. The IRS told NBC News these private collectors will not use robocalls to contact taxpayers.

The IRS did not decide to do this on its own. Congress required the agency to use private-sector debt collectors as a way to help fund road improvement projects, when it passed the Fixing America’s Surface Transportation Act in 2015.

Some IRS employees are opposed to private debt collection. Tony Reardon, president of the National Treasury Employees Union, said in a statement that he expects the result will be “collection agents getting paid to harass taxpayers, many of whom need assistance, not threats.”

Let our news meet your inbox. The news and stories that matters, delivered weekday mornings. This site is protected by recaptcha

Consumer Groups Worried

“There are so many reasons why it’s a bad idea that the IRS has been forced to use private debt collectors,” said Chi Chi Wu, staff attorney with the National Consumer Law Center. “They’re the most complained about industry to the Federal Trade Commission and the Consumer Financial Protection Bureau. All too often, consumers are being mistreated by debt collectors and now taxpayers are at risk of that in the collection of tax debt.”

Wu told NBC News she is very concerned that most of the taxpayers who will be contacted by these private collectors are well below the poverty level. A lot of these people may be eligible for special IRS programs, such as offers in compromise or not collectible status, Wu said. And she worries private collectors won’t tell people about these options.

“The collectors don’t have any incentive to do that because they get paid a commission for every dollar they bring in. Their main incentive is to collect money, come hell or high water,” Wu told NBC News. “We’re concerned that some of these vulnerable taxpayers will agree to pay more than they can afford and more than they should be paying given the availability of these programs.”

Suzanne Martindale, staff attorney at Consumers Union, the policy and action division of Consumer Reports, has similar concerns.

“When you outsource the work of the government to private companies without simultaneously having a strong oversight program to make sure these firms are treating people fairly, you run into problems,” Martindale said. “The incentive there could be strong to be very aggressive. They’re not working for you, they’re working for the government and the government wants its money and that’s a big part of the problem.”

NBC News contacted the four collection companies hired by the IRS: CBE, ConServe, Performant and Pioneer Credit Recovery. We wanted to ask them about these concerns and what procedures they have in place to prevent abuses.

Only Pioneer responded. The corporate communications department pointed us to a page on their website that tells people to expect “fair treatment and quality service.” The company goes on to say that a collection agent will “confirm you are aware of your rights as a taxpayer under the Fair Debt Collection Practices Act, as we comply with debt collection rules and consumer protections.”

Related: Student Loan Collector Navient Sued for Overcharging Borrowers

Pioneer is a subsidiary of Navient, the nation’s largest servicer of student loans. In January, the Consumer Financial Protection Bureau sued both companies, charging that they “illegally cheated many struggling borrowers out of their rights to lower repayments, which caused them to pay much more than they had to for their loans.” Navient insists the allegations are false, and said it vows to vigorously defend them.

A Golden Opportunity for Scammers

Consumer advocates warn that the use of private debt collectors will lead to confusion and could make it easier for scammers — who already extort money by pretending to be IRS agents — to steal even more money.

“From a fraud standpoint, I’m very worried about this,” said Susan Grant, director of consumer protection at the Consumer Federation of America. “We’ve always warned not to believe anybody who calls you claiming to be from the IRS because the IRS doesn’t call trying to collect delinquent taxes. And now, people will be calling.”

The IRS is well aware of the fraud problem and it promises to do everything it can to help taxpayers avoid confusion and understand their rights.

The Federal Trade Commission told NBC News it plans to update its warnings about IRS imposter scammers because of the new debt collection program. But there are still some simple ways to tell if the call is a legitimate private debt company or a fraudster. It’s a scam if the caller:

Is very aggressive or threatens you in any way with arrest or someone coming to your house

Tries to pressure you to make immediate payment

Asks for your credit or debit card information

Requests payment via gift cards — including Amazon and iTunes — prepaid debit cards or a wire transfer.

“A legitimate private debt collection company will instruct you to send a check made out to the U.S. Treasury directly to the IRS, “said Janice Kopek, an attorney with FTC’s Bureau of Consumer Protection. “You should not pay that debt collection company. Go to the IRS payments page. Never send a payment for federal taxes to anyone other than the IRS.”

You can find more information about your debt collection rights, spotting fake debt collectors and tax scams online. Anyone who has an issue with a private debt collector working for the IRS can file a complaint with the Treasury Inspector General for Tax Administration or with the Federal Trade Commission.

Herb Weisbaum is The ConsumerMan. Follow him on Facebook and Twitter or visit The ConsumerMan website.