Charleston, South Carolina (CNN) A day after thousands mourned the slayings of nine black worshippers at Charleston's Emanuel African Methodist Episcopal Church, leaders of diverse ages, races and backgrounds came together to say a symbol of segregation has no place at the state Capitol.

Remove the Confederate flag that has flown at the state Capitol complex, they demanded. Put the relic of history where it belongs, in a museum.

On Monday, three of the state's most prominent politicians -- Gov. Nikki Haley and Sens. Tim Scott and Lindsey Graham -- said they agreed.

"Today we are here in a moment of unity in our state without ill will to say it is time to remove the flag from our Capitol grounds," Haley said.

Earlier in the day, local leaders announced they would hold a rally at 11 a.m. Tuesday in front of the State House, just before the General Assembly goes into session, asking lawmakers to remove the controversial flag from the Capitol grounds.

"The time has come for the General Assembly to do what it ought to have done a long time ago, which is to remove this symbol of division and even of terrorism to some," the Rev. Nelson B. Rivers said at a news conference. He said the flag symbolizes the worst of South Carolina's history and belongs in a place where it can be viewed as history.

Charleston shooting victims

Removing it, he said, would honor the people killed at Emanuel AME. Dylann Roof, a white 21-year-old, has said he shot the black worshippers in hopes of sparking a race war, an official told CNN.

Charleston Mayor Joseph P. Riley echoed Rivers.

"The Confederate battle flag, years and years ago, was appropriated as a symbol of hate," he said, and having it fly at the state Capitol "at best sends mixed messages to those who want to understand it as a part of history."

The flag is "so often used as a symbol of hate," he said. "It has been (used in) defiance to civil rights and to equal rights and equality among races, a symbol used by the Klan, and a symbol you saw at every protest event during times of integration."

Mentioning Roof by name, Riley said, "For (people who have) those kinds of feelings -- and I hope there are very few -- it sends the wrong message" to fly the flag in a place symbolic of the rule of law.

Later in the day, in Mississippi, that state's House speaker said his state's flag similarly needs to change.

"We must always remember our past, but that does not mean we must let it define us," Speaker of the House Philip Gunn said in a statement. "As a Christian, I believe our state's flag has become a point of offense that needs to be removed. We need to begin having conversations about changing Mississippi's flag."

The Mississippi flag has a Confederate flag incorporated into the top left corner.

Photos: Worshippers return to Charleston's Emanuel AME Photos: Worshippers return to Charleston's Emanuel AME People line up to enter the Emanuel African Methodist Episcopal Church in Charleston, South Carolina, before a worship service on Sunday, June 21. It was the first service at the church since a racially motivated shooter killed nine people there on Wednesday, June 17. Hide Caption 1 of 11 Photos: Worshippers return to Charleston's Emanuel AME Church member Kevin Polite helps members into the church for the service. Hide Caption 2 of 11 Photos: Worshippers return to Charleston's Emanuel AME The Rev. Norvel Goff prays at the empty seat of the Rev. Clementa Pinckney, who was killed in the shooting. Hide Caption 3 of 11 Photos: Worshippers return to Charleston's Emanuel AME Parishioners fill the pews for the service. Church elders decided to hold the regularly scheduled Sunday school and worship service as they continue to grieve. Hide Caption 4 of 11 Photos: Worshippers return to Charleston's Emanuel AME The Rev. Norvel Goff delivers a sermon during the service. Hide Caption 5 of 11 Photos: Worshippers return to Charleston's Emanuel AME Parishioners pray and comfort each other during the service. Hide Caption 6 of 11 Photos: Worshippers return to Charleston's Emanuel AME Parishioners join together in song. Hide Caption 7 of 11 Photos: Worshippers return to Charleston's Emanuel AME South Carolina Gov. Nikki Haley embraces U.S. Sen Tim Scott. Hide Caption 8 of 11 Photos: Worshippers return to Charleston's Emanuel AME Parishioners Shakur Francis, left, and Karen Watson-Fleming sing as they attend the service. Hide Caption 9 of 11 Photos: Worshippers return to Charleston's Emanuel AME North Charleston Mayor R. Keith Summey, left, and Charleston Mayor Joseph P. Riley Jr. attend the service. Hide Caption 10 of 11 Photos: Worshippers return to Charleston's Emanuel AME People listen to the Sunday service from outside the church. Hide Caption 11 of 11

On Sunday at the church, at a service to memorialize the nine who were killed, a chair sat empty with a black cloak draped over it. It's where the Rev. Clementa Pinckney would have been sitting, had the pastor and eight other worshippers not been shot down as they studied the Bible.

But beyond the pain, hymns and gratitude also filled the church.

"The doors of the church are open," the Rev. Norvel Goff declared. "No evildoer, no demon in hell or on Earth can close the doors of God's church."

Racist manifesto

While the call to remove the flag grows, investigators are examining a website registered to Roof. It features a 2,000-word racist manifesto that details the writer's philosophy of white superiority.

"I chose Charleston because it is most historic city in my state, and at one time had the highest ratio of blacks to Whites in the country," the author wrote.

The website, called the Last Rhodesian, doesn't have Roof's name anywhere on it, but he is shown in many of the photos. Some photos show him burning and spitting on an American flag or holding a Confederate flag

On Friday, Roof appeared at a bond hearing. Families of the victims addressed him and did something many didn't expect: they said they forgave him.

JUST WATCHED Family of slain Charleston pastor: It's all about love Replay More Videos ... MUST WATCH Family of slain Charleston pastor: It's all about love 03:40

"I'm reminded of some news media persons that wondered why the nine families all spoke of forgiveness and didn't have malice in their heart," Goff said during the Sunday church service. "It's that the nine families got it."

And if the gunman was trying to start a race war, he failed.

"Lots of folks expected us to do something strange and break out in a riot," Goff said.

"Well, they just don't know us," he said, as the congregation stood and cheered.

'Take it down, take it down'

Protesters spanning all ages and races filled the front grounds of the state Capitol over the weekend, calling for the Confederate flag to be removed.

"Take it down, take it down," the crowd chanted.

JUST WATCHED Confederate flag: A symbol of hate or history? Replay More Videos ... MUST WATCH Confederate flag: A symbol of hate or history? 08:11

The dilemma of what to do with the Confederate battle flag -- a symbol of racism to many and of Southern heritage for others -- has flustered lawmakers for years.

As part of a compromise in 2000, lawmakers agreed to remove the Confederate flag from the top of the Capitol dome and place it across the street while also adding a monument to African-Americans. But the legislation mandated that only a supermajority of the legislature could change that setup going forward.

Republican state Rep. Doug Brannon has already committed to introducing a bill to remove the flag when the Legislature convenes in January.

"Personally, I have believed for years it needed to be in a museum," Brannon told CNN's Poppy Harlow. "I apologize to the people of South Carolina. I've been in the House for five years. I should have introduced this bill five years ago."

Asked why he didn't file a bill before, Brannon replied: "I didn't do my job."

Obama slated to deliver eulogy

The White House said Monday that President Barack Obama will travel to Charleston on Friday, and he will deliver the eulogy at the funeral for Pinckney.

Vice President Joe Biden will also attend the funeral, the White House said.