Swipe your card at a local store and you might be creating some serious tax headaches for the business owner.

The Internal Revenue Service has sent out letters to 20,000 small businesses since fall 2012, notifying them of "possible income under-reporting."

Small Business & Obamacare

The IRS says it is trying to identify businesses that get "an unusually high portion" of their reported sales through credit card transactions. The thinking is that a lot of cash transactions might be going unreported.

To decide who gets letters, the IRS compares a business's credit card and cash receipts with industry averages.

One recipient was the owner of a baking equipment supply company (who preferred to remain anonymous). The IRS sent her a letter on May 28 saying that 80% of her $549,955 in annual revenue came from credit card swipes.

"A larger amount of noncard revenue would be expected," the IRS letter warned.

Related: Defunding Obamacare won't stop it entirely

The business owner was given 30 days to review her records and respond in writing with an explanation. Her accountant, Steve Schneider, wrote back to the IRS this month explaining that the numbers were accurate, but the agency's assumptions were wrong.

"Over the years, the business model has switched to more online sales," Schneider told CNNMoney. "These types of customers tend to pay with credit cards."

Schneider takes issue with the IRS's tactics. He said the agency, by relying on industry averages, doesn't account for how a particular business operates.

"I just don't think that the data they have is sufficient for them to send these letters out," Schneider said.

In the case of an Italian restaurant in Harrisburg, Penn., accountant Steve Gift had to explain to the IRS that his client's credit card data had been reported incorrectly by the payment processor.

Since 2011, the IRS has required payment processors to file 1099-K forms, a record of all credit card transactions on their systems.

The numbers were off, Gift said, because the business owner had changed his federal identification number mid-year. Plus, Gift doesn't think the IRS is looking at restaurants the right way.

"This client was targeted because he has a restaurant and the IRS believes he's getting substantial cash payments," he said. "But everybody nowadays charges on their credit card."

Cracking down on small businesses could make sense. The IRS found that some $450 billion is owed in taxes that goes uncollected. Under-reporting by small businesses accounts for about $140 billion of that tax gap.

In a written statement, the IRS said the aim is to "ensure that people who are non-compliant don't get an unfair advantage over those that play by the rules and follow the law."

The IRS also said its approach is "measured and equitable in several ways, including giving taxpayers the opportunity to explain and fix errors."

Some accountants say the IRS is just doing its job, and that not many business are affected. There are 6 million small employers and 20,000 makes up less than half of one percent.

"This is nothing new," said Van Ballantyne, a small business accountant in Greenland, New Hampshire. "I think it's another attempt to try to get us all to sit a little straighter in our chairs and be more honest in our reporting. It is fairly innocuous."