Story highlights Fewer than 50 of the Gullah-Geechee people remain on Georgia's coastal Sapelo Island

After property taxes were increased by as much as 600%, many fear they will have to sell

The community "is a part of history. It will be a shame not to preserve" it, a resident says

"We have to follow the law, and assess at fair market value," the county attorney says

It's a culture struggling to survive. Fewer than 50 people -- all descendants of slaves -- fear they may soon be taxed out of the property their families have owned since the days of slavery.

They are the Gullah-Geechee people of Sapelo Island off Georgia's coast, near Savannah. This small, simple community is finding itself embroiled in a feud with local officials over a sudden, huge increase in property assessments that are raising property taxes as much as 600% for some.

Many say the increase could force them to sell their ancestral properties.

"That's part of the American history. That's part of what built this country," said Charles Hall, 79, a retired U.S. Air Force colonel who was born under a midwife's care in the same home he lives in today.

"Sapelo being the only intact Gullah-Geechee community in the country that's left, that is a part of history. It will be a shame not to preserve" it, he told CNN.

JUST WATCHED The future of the Gullah Geechee Replay More Videos ... MUST WATCH The future of the Gullah Geechee 04:01

JUST WATCHED Slaves' descendents nurture their roots Replay More Videos ... MUST WATCH Slaves' descendents nurture their roots 02:48

McIntosh County's decision to reappraise homes on the island sparked the problem.

County Attorney Adam Poppell told CNN that the Gullah-Geechee culture is invaluable, but the properties had been historically undervalued due to errors in previous property appraisals.

"We have to follow the law, and assess at fair market value," he told CNN.

To fix the problem, he said, "the state has to create a special exemption for cultural communities."

Sapelo Island, about the size of Manhattan, is a short 20-minute boat ride from Georgia's coast. But in some ways, it seems much farther.

The bumpy, unpaved dirt roads are a constant reminder that this is an island with few services. There are no police officers, fire rescue personnel, doctors or hospitals. There is no school or post office. People drive their garbage to a single garbage compactor. There are no grocery stores. The gas station is open only on Tuesdays and Thursdays.

Ninety-seven percent of the island is owned by the state of Georgia. Residents live on a small section known as Hog Hammock.

A ferry makes three round trips each day, with the last departure from the mainland at 5:30 p.m.

Residents can't miss that ferry if they want to work or go to school on the mainland. They complain this limits their employment opportunities and prohibits their children from participating in after-school activities.

Many have fled the island over the years because opportunity just doesn't exist there.

Cornelia Bailey has been one of the loudest defenders of the island where she was born and raised.

She's the ninth generation of her family to live on the island, whose slave roots are traced back to Angola. She said the taxes on her one acre property have gone from $600 a year to about $2,300.

"All these years of getting nothing, then all of a sudden, they want to lay this tax on your back and still not give you nothing," she said.

"For the last three years, we've been paying $128 a year for garbage collection. I don't even have my green garbage can. Where's my can?"

She added, with a hint of anger in her voice, "You can call 911, but nobody gonna squeal up to your front door, so forget it."

Homeowners are hiring lawyers now to have their displeasure heard in state and federal court.

Reed Colfax -- a partner at Relman, Dane & Colfax, one of the leading housing discrimination litigation firms in the country -- is heading full speed into court to have the tax bills struck down for at least half the residents of the island.

"The solution is that we freeze the tax assessments, we get the services to this island, so the people can live here," he said. "Families can move back in, have children here, have jobs on the mainland, or even develop their own economy here on the island."

Tax Assessors Board Chairman James Larkin suggests the Sapelo residents brought this issue on themselves, as some began to sell their property to developers and non-islanders who built bigger, upscale vacation homes, causing valuations to increase, and along with them their property taxes.

"If they hadn't started selling their property, there wouldn't be a problem," he told CNN.

But Reginald Hall isn't buying that argument. He and his family own three properties on more than seven acres of property on the island.

The assessed "fair market value" of their property went from $176,075 in 2011 to $910,333 in 2012. That brought on increase of more than 500% in property taxes. He is refusing to pay the taxes and he refuses to sell his family land, which he says is worth over $3 million.

"Once you leave, you are separated from more family members ... which is a real interruption in the generational teachings on this island of the culture," he said.

"We leave, and we're gone. Can't come back, because if we try to come back after we sell, you can't afford to buy," he told CNN.

Cornelia Bailey said her land may be worth about $384,000, but in reality it is priceless.

"I told one guy it was priceless, and he said everything has a price, and I said, you don't know me, this is priceless. You don't have enough money to buy it, so forget it," she said.

"We have a legacy that most people would die to have. We're fighting to keep it even for the unborn."