The Internal Revenue Service’s targeting of conservative groups has revived old fears about the agency’s vast taxing and auditing powers, so easy to abuse. But the IRS isn’t alone in holding those powers. Across the country, states and municipalities have endowed thousands of revenue and audit bureaucracies with similar capabilities. Critics complain that officials use these entities to harass enemies and help allies. The evidence makes clear just how well-founded those concerns are—especially since these agencies typically receive far less scrutiny than the IRS does.

Under the administration of Democratic governor Bill Richardson, New Mexico’s labor department sparked controversy in 2006 for auditing the state’s Republican Party. The audit, launched shortly after the party criticized the governor harshly, was meant to examine whether it was complying with state laws on employment taxes. After initially claiming that a computer had randomly chosen the GOP for scrutiny, the state admitted that an employee of the labor department had selected the party. Under fire from state newspapers, the Richardson administration turned the audit over to a private firm. The controversy faded after the firm found the Republican Party “squeaky clean,” as the Santa Fe New Mexican put it, though the paper noted that the audit was “more harassment than just due vigilance on the labor department’s part.”

Taxing and auditing to silence critics can backfire. In 1997, when the Puerto Rican newspaper El Nuevo Día ran a tough article about the island’s governor at the time, Pedro Rossello, the Rossello administration pulled $6 million in government advertising from the paper and conducted a tax audit of a construction company owned by El Nuevo’s publisher. The newspaper sued and the administration settled, promising that its future actions would be “based on legitimate, non-retaliatory reasons unrelated to news coverage.”

Often, however, the use of taxing powers for political purposes becomes known only through happenstance. In Bristol, Pennsylvania, a private firm that handled tax matters for the city hit a mayoral critic with an audit in 2000. One of the firm’s employees found a manager’s note indicating that the mayor himself had requested the audit. She turned the note over to the audit’s target, who then refused to cooperate. The town fined him, but a judge later threw out the penalty after hearing about the circumstances. A similar case took place in 1990, when Florida’s director of revenue had to resign after a whistleblower said that she had directed her staff to audit a political opponent of her husband, at the time the president of the state’s AFL-CIO. The official, Katie Tucker, told staff that she believed that the man was laundering money and involved with organized crime, though an investigation found no basis for the suspicions. Tucker pleaded no contest to official misconduct.

Elected officials sometimes use these agencies to help friends. Between 2008 and 2010, former auditors with the Idaho Tax Commission accused the agency of cutting favorable deals for some individuals and firms, either forgiving their tax bills or reducing amounts owed. In one case, the ex-auditors alleged, the commission’s chairman gave a friend confidential tax data about a local business with which the friend had a dispute. The local prosecuting attorney’s investigation found insufficient evidence to bring charges, but the commission’s chairman resigned his position anyway, saying, “I know there needs to be change.”

At other times, bureaucrats abuse their tax and audit powers to protect those very powers. Oregon business interests led a movement in 1995 to remove tax appeals from the jurisdiction of the state’s Department of Revenue, saying that the agency shouldn’t decide appeals of its own cases. Shortly after testifying in favor of the reform, one of the movement’s leaders had to go through a state income-tax audit. The Department of Revenue said that the audit was routine, but the man’s charge that the agency lacked independence gained some credibility after its chief hearings officer testified that he frequently faced pressure to rule in favor of the state and that he had been demoted after speaking out in favor of reform. The legislature eventually passed a law placing tax appeals under the purview of the Oregon Tax Court.

These cases underscore the power of taxing agencies to harass the public. So while the IRS scandal has stoked understandable outrage, it should also prompt an equally troubling question: Who is watching all the other tax and audit authorities around America?