Next November more than 5 million Americans will not be allowed to vote because of a criminal conviction in their past. Nearly 4 million of these people are not in prison, yet they remain disenfranchised for years, often for decades and sometimes for life.

States vary widely on when they restore voting rights after a conviction. Maine and Vermont do not disenfranchise people with convictions; even prisoners may vote there. People with felony convictions in Florida, Iowa, Kentucky and Virginia are disenfranchised for life, unless they are granted clemency by the governor. The rest of the country falls somewhere in between.

These laws trace their roots through the troubled history of American race relations. In the late 1800s criminal disenfranchisement laws spread as part of a larger backlash against the adoption of the Reconstruction Amendments – the 13th, 14th and 15th Amendments – that ended slavery, granted equal citizenship to freed slaves and prohibited racial discrimination in voting. Criminal disenfranchisement laws followed in their wake. They were employed right alongside poll taxes and literacy tests as part of an organized effort to design supposedly race neutral laws that were in fact intentional barriers to African-American voting. According to historian Alexander Keyssar’s “The Right to Vote: The Contested History of Democracy in the United States, between 1865 and 1900,” 27 states enacted laws restricting the voting rights of people with criminal convictions.

It’s a simple equation: communities with high incarceration rates have fewer votes to cast. The whole community suffers the result.

Historical records memorializing state constitutional conventions reveal some astounding rhetoric in support of these laws. During the Virginia Constitutional Convention of 1901, delegate Carter Glass (later a prominent senator, the Glass of the Glass-Steagall Act) described the suffrage proposal which included felony disenfranchisement as a plan that would “eliminate the darkey as a political factor in this state in less than five years.” The strategy was not limited to southern states. In New York, African-American suffrage was the subject of much debate at the 1821 and 1846 constitutional conventions. In a refrain that echoes throughout the century-long suffrage debate, New York delegate Samuel Young implored: “Look to your jails and penitentiaries. By whom are they filled? By the very race whom it is now proposed to clothe with the power of deciding upon your political rights.”

When states enacted criminal disenfranchisement laws, they also expanded their criminal codes to punish offenses that they believed targeted recently freed slaves. In an 1896 decision, Ratliff v. Beale, the Mississippi Supreme Court confirmed that the new state constitution narrowed the disenfranchisement provision to target certain crimes such as theft, perjury, forgery and bigamy of which blacks were then more often convicted. The court explained:

Within the field of permissible action under the limitations imposed by the federal constitution, the convention swept the circle of expedients to obstruct the exercise of the franchise by the negro race. . . This race had acquired certain peculiarities of habit, of temperament, and of character, which clearly distinguished it as a race from that of the whites . . . and its criminal members given rather to furtive offenses than to the robust crimes of the whites. Restrained by the federal constitution from discriminating against the negro race, the convention discriminated against its characteristics and the offenses to which its weaker members were prone.

Today, Mississippi’s constitution still denies the right to vote based on the same “furtive offenses.”

Nationwide, 13 percent of black men have lost the right to vote, a rate that is seven times the national average. But the ripple effects of large-scale incarceration now extend well beyond the individuals who are imprisoned, and as a result minority communities throughout the country have lost political influence. It’s a simple equation: communities with high incarceration rates have fewer votes to cast. The whole community suffers the result.

Who Votes? The first in a series about the complexities of voters and voting.

Despite this history and continuing impact, legal challenges to criminal disenfranchisement laws have largely failed. In 1974 the Supreme Court issued an opinion in Richardson v. Ramirez, a challenge to California’s disenfranchisement law. Writing for the majority, the late William Rehnquist relied on an obscure clause of the 14th Amendment itself to sanction criminal disenfranchisement laws. The Court ruled that Section 2 of the amendment, which reduces a state’s representation in Congress if the state has denied the right to vote for any reason “except for participation in rebellion, or other crime,” distinguishes felony disenfranchisement from other forms of voting restrictions (which must be narrowly tailored to serve compelling state interests in order to be constitutional).

But state legislatures have stepped up where court challenges have failed. Since 1997, 23 states have either restored voting rights or eased the restoration process. However, this year two states key to the electoral map – Florida and Iowa – rolled back recent reforms. (More on that in future posts.)

Representative John Conyers Jr. recently reintroduced into the House the Democracy Restoration Act (which I worked on), a bill that would go a long way toward ending disenfranchisement. A broad coalition of law enforcement, religious leaders and civil rights groups support the measure, which would restore voting rights in federal elections to all Americans who are out of prison.

For all of the history and all of the politics, the right to vote is a very personal piece of empowerment. If as a society we want those who have done wrong to do right, then this one act can signify a life change. As one New York resident, reflecting on voting for the first time in 2008 after having her rights restored, put it: “I felt like I was finally a productive member of society. I’ve never before felt like I could make a difference in terms of what happens around me. But I walked out of the polling place on Election Day feeling like I mattered.”

Erika L. Wood is an associate professor of law at New York Law School. This is the first in a series of articles —Who Votes?— that will appear occasionally as part of Campaign Stops.