Following the church shooting this week, South Carolina Gov. Nikki Haley offered powerful words of healing and said the “people of South Carolina need us to come together and be strong.”

But at the same time the state’s citizens are being urged to “come together,” the state continues to display a divisive symbol — the Confederate flag — outside the state Capitol.

Reacting to calls to remove it, Sen. Lindsey Graham said the flag “is part of who we are.”

His answer makes me wonder just who Graham means by “we.” Does he mean the white people of South Carolina? Does he mean the neo-Confederates of South Carolina? Surely he doesn’t mean the African Americans in his state whose ancestors were enslaved, tortured and murdered under the regime represented by that flag. Surely he doesn’t mean older African Americans who still recall the terrorism of Klansmen who adopted the flag as their standard during Jim Crow. Surely not.

Meanwhile, Haley danced around the question in the wake of the racist attack on the Emanuel AME Church by a young man who displayed an image of the Confederate flag on his vehicle. Her bottom line was that “right now, I’m not doing that to the people of my state.”

She seems blind to the fact that she’s doing something to a significant portion of the state every day, confronting them with a symbol of the state’s ugly past.

Like Graham, many white Southerners argue that the Confederate flag merely symbolizes family heritage or regional pride, and that their use of it isn’t intended as a hateful act. Displaying the flag in public places, they say, simply acknowledges its historic place in the cultural fabric of the South. But the time has come for South Carolina to take down the flag. Whatever its benign significance to some people, it historically signifies an era of slavery and oppression, and it has been appropriated as a symbol of hate by other groups. Those are facts, not an interpretation.

In his 1861 “Cornerstone” speech, Alexander H. Stephens, vice president of the Confederate States of America, left no doubt about what the Confederacy represented when he rejected the idea that slavery was a moral wrong: “Our new government is founded upon exactly the opposite idea; its foundations are laid, its corner-stone rests, upon the great truth that the negro is not equal to the white man; that slavery subordination to the superior race is his natural and normal condition. This, our new government, is the first, in the history of the world, based upon this great physical, philosophical, and moral truth.”

Let me be clear: I don’t think that everyone who reveres the flag is racist. Surely all of the people who apply for state-issued license plates bearing the flag do not believe in the hatred with which some people display it. For many South Carolinians, the flag at this point may well represent heritage without the taint of past racism.

But they should ask themselves, Whose heritage are they celebrating?

Is the Confederate flag really a symbol around which people, to paraphrase Gov. Haley, can “come together and be strong?”

To truly honor those who were slain, the Confederate flag in every state, in every form, must come down.

J. Richard Cohen is president of the Southern Poverty Law Center.

Sign up for Inside TIME. Be the first to see the new cover of TIME and get our most compelling stories delivered straight to your inbox. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Contact us at letters@time.com.