When the founders of the U.S. Constitution in 1787 considered whether America should let the people elect their president through a popular vote, James Madison said that “Negroes” in the South presented a “difficulty … of a serious nature.”

During that same speech on Thursday, July 19, Madison instead proposed a prototype for the same Electoral College system the country uses today. Each state has a number of electoral votes roughly proportioned to population and the candidate who wins the majority of votes wins the election.

Since then, the Electoral College system has cost four candidates the race after they received the popular vote — most recently in 2000, when Al Gore lost to George W. Bush. Such anomalies and other criticisms have pushed 10 Democratic states to enroll in a popular vote system. And while there are many grievances about the Electoral College, one that’s rarely addressed is one dug up by an academic of the Constitution: that it was created to protect slavery, planting the roots of a system that’s still oppressive today.

“It’s embarrassing,” said Paul Finkelman, visiting law professor at University of Saskatchewan in Canada. “I think if most Americans knew what the origins of the Electoral College is, they would be disgusted.”

Madison, now known as the “Father of the Constitution,” was a slave-owner in Virginia, which at the time was the most populous of the 13 states if the count included slaves, who comprised about 40 percent of its population.

During that key speech at the Constitutional Convention in Philadelphia, Madison said that with a popular vote, the Southern states, “could have no influence in the election on the score of Negroes.”

Madison knew that the North would outnumber the South, despite there being more than half a million slaves in the South who were their economic vitality, but could not vote. His proposition for the Electoral College included the “three-fifths compromise,” where black people could be counted as three-fifths of a person, instead of a whole. This clause garnered the state 12 out of 91 electoral votes, more than a quarter of what a president needed to win.

“None of this is about slaves voting,” said Finkelman, who wrote a paper on the origins of the Electoral College for a symposium after Gore lost. “The debates are in part about political power and also the fundamental immorality of counting slaves for the purpose of giving political power to the master class.”

He said the Electoral College’s three-fifths clause enabled Thomas Jefferson, who owned more than a hundred slaves, to beat out in 1800 John Adams, who was opposed to slavery, since the South had a stronghold.

While slavery was abolished, and the Civil War led to citizenship and voting rights for black people, the Electoral College remained intact. Another law professor, who has also written that the Constitution is pro-slavery, argues that it gave states the autonomy to introduce discriminatory voting laws, despite the Voting Rights Act of 1965 that was built to prevent it.

In 2013, the Supreme Court freed nine states, mostly in the South, from the stipulation in the Voting Rights Act that said they could only change voter laws with the approval of the federal government.

“A more conservative Supreme Court has been unwinding what the [other] court did,” said Juan Perea, a law Professor of Loyola University Chicago. “State by state, that lack of supervision and lack of uniformity operates to preserve a lot of inequality.”

In July, a federal appeals court struck down a voter ID law in Texas, ruling that it discriminated against black and Latino voters by making it harder for them to access ballots. Two weeks later, another federal appeals court ruled that North Carolina, a key swing state, had imposed voting provisions that “target African Americans with almost surgical precision.”

And for this presidential election, 15 states will have new voting restrictions, such as ones that require government-issued photo identification at the polls or reduce the number of hours the polls are open.

“The ability of states to make voting more difficult is directly tied to the legacy of slavery,” Perea said. “And that ability to make voting more difficult is usually used to disenfranchise people of color.”

The National Popular Vote Interstate Compact has gained traction, but for reasons more related to the anomaly of the Gore-Bush election. Assemblyman Jeffrey Dinowitz championed legislation in New York that brought the state into the compact and was asked by the NewsHour Weekend why the movement is important.

“We are the greatest democracy on the planet, and it seems to me that in the greatest democracy, the person who gets the most votes should win the election,” said Dinowitz. “We’re one country, North, South, East and West. One country. The votes of every single person in the country should be equal. And right now, the votes are not equal. Some states your vote is more important than in other states.”

New York overwhelmingly agreed on his bill in 2014, joining nine other states and Washington, D.C. Together, they have 165 electoral votes. If they gain a total of 270 — the majority needed to elect a president — the nation will move to a popular vote.

Not all academics agree that slavery was the driving force behind the Electoral College, though most agree there’s a connection. And both Perea and Finkelman say they know it is not the most prominent argument for the push toward a popular vote.

“But it is a vestige that has never been addressed,” Perea said.

This article has been updated to reflect the fact that Jefferson won the presidency in 1800.