COLUMBIA, S.C. — Tommy Daras had planned on a small ceremony Wednesday to remove the Confederate flag flying from the edge of the parking lot at his South Carolina restaurant. Then he encountered a hitch.

Two members of the Sons of Confederate Veterans showed up at the flagpole and advised Daras the group owns a small tract of land where the flag flies, deeded to them in 2005 by a former restaurant operator who kept literature supporting slavery in his barbecue joints.

Warned they could have him arrested for trespassing, Daras is now rethinking his options in his bid to replace the Confederate flag with an American flag outside his restaurant.

"I wanted to compromise. I tried to work with them," Daras said of months of negotiations. "But I realized today they are racists and they are bringing down my business with them."

Daras said he had talked to the group about putting up a different Confederate flag than the current Stars and Bars, which has been embraced by racial hate groups. He said the flag flying so close to his Edisto River Creamery and Kitchen is bad for his business.

The Sons of Confederate Veterans said they fly the flag and maintain a monument to an 1865 Civil War skirmish on the tiny piece of land to honor their ancestors who fought for the South. They bristle at any suggestions of being racist, noting they can't control what others choose to use the Confederate battle flag for elsewhere.

If Daras brings down the flag, the group plans to bring him to court, said member Carl Platts.

He stood by the flagpole for nearly an hour Wednesday until Daras gave up. In the event of legal action, he warned, "We'll take the whole damn place" on a video broadcast live by The Times and Democrat of Orangeburg.

About a dozen police officers stood by, but there was no violence.

Daras bought the restaurant building and opened his eatery in May 2015 at a major intersection about 50 miles south of Columbia, the state capital.

He knew the flag was there and thought nothing of it.

Then a month later, authorities say, nine African-Americans at a Charleston church were gunned down by a white man who put pictures of himself with the Confederate battle flag and writings where he said he thought blacks were inferior to whites on his website. South Carolina lawmakers pulled down the Confederate flag in front of the Statehouse within weeks, but Daras said the Sons of Confederate Veterans put a larger flag on its pole.

"That's when it all started. I have black groups boycotting me. I have white groups angry I'm thinking about bringing the flag down," Daras said.

Daras then studied his deed. It does not mention the sliver of land at all. He bought title insurance and paid for an inspection, so he thinks legally the land is his.

The Sons of Confederate Veterans disagree.

They also have a deed showing the restaurant's previous owner, barbecue baron Maurice Bessinger, gave 0.003 acres — or about 130 square feet of land — to the group in 2005, according to Orangeburg County's online records. That's about the size of an average master bathroom in a U.S. home, according to the National Association of Home Builders.

The organization also has paid taxes on the land, according to online records.

Bessinger once had nearly a dozen restaurants serving barbecued meat with a mustard-based yellow sauce. He began flying the Confederate flag outside of the restaurants in 2000 after the South Carolina Legislature removed the rebel banner that was on top of the Statehouse dome and put one up on a pole in front of the Capitol.

Bessinger died in 2014, and his family quickly removed the Confederate flags and pro-slavery and white supremacist literature in his restaurants. But the Confederate flag still flies in Orangeburg because of the land deal.

"Until you prove this flag belongs to you, we don't want you to mess with it," Sons of Confederate Veterans member Buzz Braxton told Daras outside his restaurant Wednesday.