On Wednesday, a progressive shared a passage from Hillary Clinton’s 1996 book It Takes a Village in which she discussed her discomfort with Arkansas employing prisoners to tend to the governor’s mansion. Clinton recoiled at the thought of murderers serving at the house, but she tolerated the practice during her time in the governor’s mansion.

Now lefties on social media are enraged by Clinton’s supposed immorality or effete, out-of-touch liberal demeanor. They have even gone so far as to compare her to a modern-day slave owner. Yet they fail to grasp how this form of criticism only serves to undermine their own progressive values.

“2016 was a choice between a white woman benefitting from black prison labor and a white man campaigning on sending black people to prison,” said the co-founder of Campaign Zero, Samuel Sinyangwe, on Twitter.

Using prisoners as free labor is a troubling practice that needs to be stopped. And the fact that states across this country, and especially the South, have employed this practice for more than a century does not make it any more acceptable. However, a century-long process of pseudo-normalization and institutionalization of an abhorrent practice does make it both harder to undo and easier to sully the reputation of those who interact with it but fail to fix the problem.

This new revelation represents one more opportunity for those on the left to criticize Clinton and the authenticity of her progressive credentials. This type of progressive criticism has long undermined the progressive movement and enabled its detractors. This predicament, which frequently befalls progressives in the South, represents what I like to call the Dilemma of the Carpetbagger. And Clinton frequently falls victim to this Catch-22.

Carpetbaggers originally were northern Union supporters and former abolitionists who moved to the South after the Civil War to implement Reconstruction. They were businessmen, educators, military men, and politicians. Also, many northern women, who were unable to serve in the Union army, relocated to the South and set up schools, hospitals, and other services for freed blacks. In many ways, these people were the best the North had to offer—and the systematic and pervasive racist structures of the South left most of them gobsmacked.

During Reconstruction carpetbaggers failed countless times in their attempts to remake the South. Since they were not from the region, they all had theoretical notions about the day-to-day goings on and about relationships between blacks and whites. Frequently, these theories resulted in failed experiments, and carpetbaggers were regularly astonished by white Southerners’ commitment to return freed blacks to a life akin to slavery by any means at their disposal. Southern oppression has consistently elicited a befuddled naivete from carpetbaggers as they struggled to cope with the scale of the South’s systemic racism.

Many carpetbaggers eventually relocated back to the North after their theories either failed or were undermined by Southerners. Throughout Reconstruction, Southerners demonized and stigmatized carpetbaggers as good-for-nothing outsiders intent on ruining their way of life. They also seized every opportunity to depict them as failures.

Following the end of Reconstruction, when the South prevailed by driving out Northerners despite losing the presidential election of 1876, the Southern perspective of carpetbaggers became the de facto definition of the word. For more than a century a carpetbagger has been described as an untrustworthy outsider who invades a community instead of as a progressive following the ideals of Abraham Lincoln and working to instill racial equality across America. In 12 years, carpetbaggers and progressives passed the 13th, 14th, and 15th Amendments.

But today, too many progressives have totally succumbed to Southern propaganda about the term carpetbagger. What’s more, we are also too quick to direct blame toward the progressives who fail to bring about the change they desire in the regressive parts of America.

Hillary Clinton was born and educated in the North, relocated to the South, and worked to empower families, minorities, and the poor to the chagrin of her conservative detractors. She’s a modern-day carpetbagger—but I mean that in the good sense (and by the way I don’t mean it the way she was first called it in New York, when it just meant “someone who isn’t from here”).

Now, on this matter of prisoners at statehouses and the like: In the 20th century blacks all across the South were imprisoned for vagrancy and loitering and sentenced to chain gangs and various forms of prison labor. Louisiana still uses prison labor at court houses and the governor’s mansion. Mississippi just outlawed this practice in 2012 following the controversy that ensued when former Governor Haley Barbour pardoned five inmates, including four with murder convictions, who worked at the mansion. When that controversy went national, Southerners were concerned about the pardoning of murderers, while the rest of America was shocked about the practice of using prison labor.

Many Americans prefer to gloss over the structural, racial injustices that are the norm in the South, and today, we are still shocked when we encounter them. Just last month Sinyangwe insightfully wrote on Twitter about his shock at Louisiana’s prison system describing it as “slavery, by another name,” and acknowledging that this is just the reality in this area. I recommend that everyone read his Twitter feed because it is very informative. But we shouldn’t view his shock and outrage over that passage in Clinton’s 22-year-old book as any more noble or moral than Clinton’s predicament. They both were shocked. But Clinton had to live there. As the wife of a governor, no less.

Over four decades ago Clinton uprooted her life and moved to a part of America whose structures still shock the nation. She certainly failed to make all the changes that she aspired to, and she’s definitely had some bad ideas and put her foot in her mouth. However, carpetbaggers, almost by definition, seek to change the worst parts of America by fully committing themselves to the endeavor and living within the nation’s flawed structures.

This quest may sully their progressive purity, and their light may shine less brightly than Northern progressives like Bernie Sanders or Elizabeth Warren, who come from states where no such complexities exist. But without carpetbaggers, progressives would largely be unable to make the changes we hope to bring to America. Our demonization of this integral aspect of American progressivism only demonstrates a naïveté that will continue to impede the movement.