States can voluntarily swap out their statues for new ones at will, thanks to a 2000 amendment to the original federal law authorizing the collection. But Congress is ultimately responsible for what can and can’t be kept within the Capitol; the senators and representatives who condemned the marchers in Charlottesville have the power to clean their own house by banning Confederate statues.

The collection’s origins date back to 1864, although the Capitol has been home to sculptures since its construction. The idea was elegant in its simplicity: Under the original authorizing law, each of the states could commission two statues “in marble or bronze … of deceased persons who have been citizens thereof.” The bill called on the states to accordingly honor citizens who were “illustrious for their historic renown or for distinguished civic or military services such as each state may deem to be worthy of this national commemoration.” Once completed, the chosen statues would then be placed in the old House chamber in the Capitol building, which was renamed National Statuary Hall.

States responded, albeit slowly, by commissioning the likenesses of a variety of eminent Americans in the ensuing decades. Connecticut, for example, sent marble sculptures to Washington in 1872 of Roger Sherman, who helped draft the Declaration of Independence, and Jonathan Trumbull, the only colonial governor to join the revolution when it broke out. Massachusetts contributed one of the famed revolutionary leaders Samuel Adams in 1876 for the nation’s centennial; Pennsylvania proffered the inventor Robert Fulton, who made steamboats a viable means of transportation, in 1896.

But many Southern states took a different approach. At the turn of the 20th century, the Democratic Party’s counterrevolution against Reconstruction-era reforms had reached its apex. “Redeemed” states drafted new constitutions to exclude black Americans from political life and restrict their civil rights. Violence had also returned to the forefront of Southern political life. White supremacists in North Carolina overthrew a multiracial local government in Wilmington in 1898 in the only successful coup d’etat in American history. After Theodore Roosevelt invited Booker T. Washington to dinner at the White House in 1903, South Carolina Senator Ben Tillman remarked that the president’s invitation “will necessitate our killing a thousand niggers in the South before they learn their place again.”

Against this perilous backdrop, legislators in Virginia’s redeemed General Assembly began to discuss sending a statue of Robert E. Lee to take up an honored place in the nation’s capital. The state reserved one of its two allocated statues for George Washington, an obvious and universally hailed choice. But for the second slot, the legislature rejected efforts to commission statues of Thomas Jefferson, James Madison, James Monroe, and other famous Virginians. Only Lee would be allowed to join the father of the nation in the heart of the republic.