When we think about the framers of the Constitution and how they handled the issue of race, we conjure up the extremes: the hypocrites and the heroes. At one end is Thomas Jefferson, who wrote that “all men are created equal” but believed Africans were inferior and fathered children with an enslaved woman. At the other end is Alexander Hamilton, who, at least as depicted by admirers like the biographer Ron Chernow and the playwright Lin-Manuel Miranda, was an ardent abolitionist.

This framing, however, is simplistic and misleading. It is simplistic because it overlooks harder-to-categorize positions like that of James Madison, the lead drafter of the Constitution, who genuinely rejected the idea of racial inferiority yet still failed to put his beliefs in equality and liberty into practice. And it is misleading because it implies that as long as we avoid having racist attitudes, we can succeed in avoiding racist policies. We think that if we’re not Jefferson, we must be Hamilton. But this is not the case.

In this respect, Madison is the founding father who can teach Americans the most about our present contradictions on race. Madison insisted that enslaved Africans were entitled to a right to liberty and proposed that Congress purchase all the slaves in the United States and set them free. Yet not only did he hold slaves on his plantation in Virginia and fail to free them upon his death, but he also originated the notorious three-fifths compromise in the Constitution, which counted a slave as three-fifths of a person for purposes of legislative representation.

The tension between Madison’s aspirational beliefs and his highly constrained actions continues to be America’s own tension. Like Madison, contemporary United States society rejects racial inequality in principle. But also like Madison, a majority of Americans — as reflected in our democratic institutions — are ultimately unwilling or unable to make the costly changes that would be necessary to achieve equality in practice.