A veterans organization in a New York town is fighting for its right to fly a patriotic flag after the city council refused to let the group display the flag, calling it a symbol of the Tea Party with right-wing connotations.

The United Veterans Memorial & Patriotic Association replaced a tattered American flag at the New Rochelle Armory, flying the Gadsden flag underneath it. The bright yellow “Don’t Tread On Me” flag has been used by the Navy and Marine Corps since 1775.

When a city council member complained to the city manager that he found the flag offensive, the city manager initially decided to let the flag fly anyway. But the city council overruled him, and the flag was taken down. On Wednesday, the council voted 5 to 2 to keep the veterans from putting the flag back up.

The council objected to the flag because they said Peter Parente, the president of the veterans group, is a member of the Tea Party and the group wants to use the flag to make a political point.

But Parente said at the council meeting no one in his organization is a Tea Party member.

“I’m a proud Republican,” he told the council.

Once council member compared the flag to the rainbow flag used to symbolize gay pride, according to local news website Talk of the Sound. Another has compared it variously to the Nazi flag, a Mickey Mouse flag and graffiti.

The veterans group has retained the Thomas More Law Center to regain the right to fly the flag. They have not yet filed a lawsuit.

“Using the City Council’s reasoning, they would remove the Stars and Stripes from flagpoles because both Democrat and Republican Parties, as well as most political candidates, use the Stars and Stripes in their campaign messaging,” said Richard Thompson, president of the Thomas More Law Center.