When Americans go to the polls this November, they better have some ID. If they don’t, millions of them won’t be casting a ballot. Over the past two years, Republican legislators from Texas to Florida to Wisconsin and beyond have enacted new restrictions making it significantly more difficult for citizens to exercise their right to vote. These laws, which could disenfranchise more voters than at any time since the 1960s, exist because of one widely held conservative belief: that our elections are plagued with fraud.

National Republican chairman Reince Priebus echoed this view on MSNBC recently when discussing the new Wisconsin law that requires citizens to produce photo ID at the polls or be shown the door. He argued that the state’s election system was “absolutely riddled with voter fraud.” Priebus may be correct, but only if his standard for “riddled” is 0.0002 percent. A nonpartisan study on voter fraud in Wisconsin after the 2004 election found just seven ineligible votes—all of which were cast by ex-felons who were ineligible to vote despite being released from prison—out of 3 million ballots cast.

Yet the notion that voter fraud is rampant appears to have become a core belief among conservatives. Since the 2010 election, dozens of anti-voting measures have been enacted, from requiring photo identification to cutting early voting periods to mandating that citizens present a birth certificate or passport in order to register.

If you challenge conservatives with just how rare voter fraud is in the United States, you usually get one of three responses: It’s easy to do, it’s hard to catch, and they’ve heard of it happening. Let’s take these arguments one at a time.

First, conservatives often note that even if widespread fraud hasn’t occurred, the ease with which voters could misrepresent themselves at the polls warrants stringent preventive measures. Sting artist James O’Keefe has released a number of videos in which people show up at the polls with hidden cameras claiming to be someone they’re not. The fact that most poll workers trust these individuals and hand over a ballot proves, in O’Keefe’s estimation, that we need voter ID laws so people can’t misrepresent themselves.

This view ignores an important point: Regardless of how “easy” it may be to trick an unsuspecting poll worker, it rarely happens. And there is a simple explanation why: Voter fraud is a felony that carries a federal sentence of five years in prison and a $10,000 fine. If I show up at the polls and pretend to be Michael McDonald, at best, I gain a single vote for my preferred candidate; at worst, I get sent to prison until 2017. Stealing an entire election one vote a time (and risking significant penalties for doing it) defies common sense.

So does the logic that we must prevent any crime that’s “easy” to do. It’s really easy to dump a bucket of water on a policeman’s head. In reality, it doesn’t happen because people don’t want to go to jail. Yet by O’Keefe’s logic, the best way to prevent it would be to ban buckets.

Second, many conservatives contend that the reason studies show so little voter fraud is because it’s really hard to catch. It’s also hard to catch murderers, but that doesn’t mean we start locking up everyone in order to prevent people from killing one another. Americans have long subscribed to the notion of innocent until proven guilty; that, as Benjamin Franklin said, “it is better 100 guilty persons escape than that one innocent person suffer.” Without proof, or even credible evidence that widespread voter fraud exists, saying that it’s difficult to detect is not enough reason to justify stripping the right to vote from millions.

Finally, those who suspect rampant fraud sometimes admit they are just relying on their gut; it simply feels like it exists. Of course, we’ve all heard the stories of dead voters in Chicago or suitcases of absentee ballots in the Deep South. Conservatives capitalize on these familiar chestnuts to make the case for restrictive new measures, often parading these old stories as modern facts. For example, South Carolina Attorney General Alan Wilson went on Fox News earlier this year to declare that “we know for a fact there are deceased people whose identities are being used in elections in South Carolina.” Conservative outlets parroted the story endlessly, yet after a state investigation turned up zero evidence of fraud, they have gone silent.

In truth, concerns about voter fraud usually stem from other conservative fears. The right believes that undocumented immigrants are menacing our country; therefore, they must also be trying to vote illegally. The poor, who liberals want to assist with food and health care, are being driven from poll to poll to ensure Democrats stay in office. If you’re impoverished, the thinking goes, you must lack the moral fortitude that prevents the wealthier among us from voting more than once.

Some Republicans admit they have seen no evidence of fraud. Last year, Pennsylvania State Sen. Charles McIlhinney conceded as much. Yet in his mind, the mere prospect that it could exist outweighed the possibility that 700,000 citizens may lose their vote in November because of the state’s new voter ID law, which passed two months ago.

Proponents of tough new voting restrictions often argue that voter fraud cancels out honest votes, effectively disenfranchising you and me. The irony is that by trying to stamp out a fraud menace that doesn’t exist, millions of honest voters will be turned away from the polls. If widespread voter fraud existed, we should confront it. In the meantime, we shouldn’t make policy out of tall tales and paranoid fears.

