Earlier this past week a judge ruled that the city of Louisville, Kentucky can proceed with the removal of a Confederate monument near the campus of the University of Louisville. Arguments against removing Confederate monuments over the past year have often claimed that in doing so communities run the risk of erasing history. What has been universally overlooked, however, is that the push to establish monuments to the Confederacy during the postwar years helped to erase the history of those white and black southerners who remained loyal and were willing to give their lives to save the Union.

Southern Unionism took many forms during the Civil War. Some disagreed with the right of a state to secede from the Union at the war’s outset while others grew weary of the Confederacy in response to a number of factors, including a Conscription Act in 1862 that exempted large slaveowners, the impressment of horses or mules for the army, and a “tax-in-kind” law that allowed the government to confiscate a certain percentage of farm produce for military purposes. Others in places like Appalachia and other highland regions that included few slaves saw little value in supporting a government whose purpose was the creation of an independent slaveholding republic.

Resistance to the Confederacy also took many forms throughout the war. The release of the movie, The Free State of Jones, starring Matthew McConaughey next month, will introduce audiences to Newton Knight, who led an armed rebellion against the Confederacy in Jones County, Mississippi. Some joined clandestine political organizations such as the Heroes of America, which may have contained upwards of 10,000 members. Networks of communication kept resistors in touch with one another and their activities throughout the region. Unionists risked arrest by Confederate officials, ostracism from within the family, and violent reprisals from the community.

It is impossible to know just how many white southerners remained loyal to the Union during the war given disagreements over its very definition, but we do know that somewhere around 100,000 southern white men from Confederate states, except for South Carolina, served in the U.S. military. East Tennessee supplied somewhere around 42,000 men, but other Confederate states yielded significant numbers, including 22,000 from Virginia (and West Virginia) and 25,000 from North Carolina. The First Alabama Cavalry, which was considered one of the toughest units in General William Tecumseh Sherman’s army, took part in his “march” through Georgia and the Carolinas in 1864-65.

The decision to express one’s loyalty to the Union by joining the army was often a painful one to make from the lowliest private to some of the highest-ranking officers. While the story of Robert E. Lee’s decision to resign his commission in the U.S. army, rather than betray his home of Virginia, is often told and re-told in tragic prose, others grappled with the same decisions and yet chose to remain loyal. The man who offered Lee command of the U.S. army in 1861 was another Virginian by the name of Winfield Scott. Scott, whose military career stretched back to the War of 1812—including a failed presidential bid in 1852—was the highest-ranking general at the beginning of the war. Scott’s decision was no less difficult than Lee’s and yet he remained loyal and although too old to take command in the field, he helped formulate military policy that ultimately proved successful in subduing the rebellion.

General George Henry Thomas, also from Virginia, became one of the most successful generals in the war and saved the Union army from being completely routed on September 19, 1863, earning him the nickname the “Rock of Chickamauga.” His loyalty to the nation cost him his family, who refused to speak with him ever again and even turned his picture against the wall. Very few monuments to the service of these men and others like them, who defied family, friends, and community for the sake of the nation, can be found in the former Confederate states. And yet the removal of some Confederate monuments has caused some to worry about erasing history.

The other significant Southern bloc that voiced their loyalty to the Union and commitment to crushing the rebellion was the region’s slave population. From the beginning of the war, and in the shadow of a Supreme Court that as recently as 1857 ruled that free and enslaved blacks could not be citizens of the United States, African Americans offered their services to the military. Beginning in 1862 along the Sea Islands of South Carolina and Georgia, former slaves rushed into the first all black regiments. By the end of the war roughly 150,000 former slaves fought and died to save this nation. They did so under the most harrowing conditions. Black soldiers were massacred on battlefields and even sent back into slavery at places like Fort Pillow in Tennessee and at the Crater in Petersburg, Virginia by Confederates, who refused to treat them as legitimate soldiers. As if that wasn’t enough, their own government refused to pay them what white soldiers earned. Only sustained protests that lasted more than a year and continued demonstrations of bravery on the battlefield led Congress to correct this injustice in the summer of 1864.

Southern Unionists, both black and white, may have celebrated Confederate defeat, but they continued to be persecuted owing to their wartime beliefs and actions by terrorist groups such as the Ku Klux Klan. Life was especially difficult for former slaves, who fought for the Union and now hoped to exercise the right to vote, own land, or run for public office. Their sacrifice for the Union ended in the rise of Jim Crow state governments by the turn of the 20th century.

After the war, as white Southerners erected monuments to their Confederate dead they also erected monuments to their former slaves, only they recalled not brave men who fought to preserve the Union, but their loving former “servants” who remained loyal to master and their Lost Cause. The very act of monument erection helped to erase this history for much of the 20th century.

The removal of Confederate monuments need not result in the erasure of history. In fact, it may for the first time create the intellectual and physical space to commemorate and remember a new narrative of the past, one that corresponds more closely to the long and rich history of service and sacrifice to this nation that is recalled each year on Memorial Day.

Kevin M. Levin is a historian and educator based in Boston. He is the author of Remembering the Battle of the Crater: War as Murder (2012) and is currently at work on Searching For Black Confederate Soldiers: The Civil War’s Most Persistent Myth. You can find him online at Civil War Memory and Twitter @kevinlevin.