Since the shootings that killed nine people in Charleston, S.C., symbols of the Confederacy have been disappearing throughout the American South.

But the nation's southernmost city is restoring its memorials — to both sides of the Civil War.

That includes a white pavilion in Key West, which has stood in Bayview Park since the United Daughters of the Confederacy put it up in 1924. Many longtime residents, like Tom Theisen, never even noticed who had built it.

"I probably had seen it before, but it never crossed my mind until all the flag stuff," Theisen says.

As part of the installation of a new memorial dedicated to veterans of the Vietnam War, the city is restoring the Confederate memorial. It's also restoring another monument, 140 yards away. That one is dedicated to soldiers from two New York regiments who were stationed in Key West during the Civil War and died there of diseases like yellow fever.

Credit Nancy Klingener / WLRN / WLRN The monument to Union soldiers who died while stationed in Key West was built in 1930. It is 140 yards from the Confederate pavilion.

Clayton Lopez, the only black member of the seven-person Key West City Commission, says he does not object to the city's restoration of the Confederate memorial.

"I'm glad that we're actually doing it," Lopez says. "We have to preserve it. Why would we hide what actually happened? Good or bad, it's what happened."

While Key West makes a big deal out of its status as the nation's southernmost point, it doesn't feel like part of Dixie. Many of the original settlers were from the Bahamas. Later in the 19th century, thousands came from Cuba. Like many seaports, it has always drawn people from all over the world, including northern U.S. states, Europe and the South.

The Southern influence was at its height when the Confederate memorial was built. It was the height of the Jim Crow era. Key West had an active chapter of the Ku Klux Klan.

"But we've got an answer for that this year," Lopez says.

Another part of the city's Bayview Park project is a new memorial, dedicated to African-American soldiers who signed on to the Union Army while in Key West.

The Union Army was recruiting in Key West because even though Florida was the third state to secede, in 1861. Key West stayed in Union hands throughout the war.

A quick-thinking Army captain marched his troops across the island from their barracks to occupy Fort Zachary Taylor, which was still under construction when Florida seceded.

One prominent Key Wester, Stephen Mallory, became secretary of the Confederate Navy and a few wealthy slave owners left town, says Tom Hambright, historian with the Monroe County Public Library in Key West. But he said only about seven people actually joined up with the Confederate forces and local sentiment was basically neutral.

Credit Nancy Klingener / WLRN / WLRN The obelisk at Clinton Place near Key West Harbor, built to honor Union forces, is surrounded by a fence built by a Confederate veteran.

"A majority of the settlers at that time were probably from the Bahamas, who had settled the slavery question 20 years before," Hambright said.

Reminders of Key West's conflicting loyalties and accommodating memorials are even closer on a different part of the island. Downtown near the harbor, the Key West Navy Club put up an obelisk in 1866. It's dedicated to the Union soldiers and sailors who died in Key West during the war.

Surrounding that obelisk is a low cast-iron fence. A plaque on the fence names its builder: J.V. Harris — Confederate veteran.