MANHATTAN, New York (WABC) -- New York Governor Andrew Cuomo said he won't punish grassroots community groups that have endorsed his primary opponent, Cynthia Nixon.He also denied threatening the groups last week.Bill Lipton, the state director of the Working Families Party, recently said he was at a meeting where Cuomo supposedly said, "If unions or anyone give money to any of these groups, they can lose my number."Cuomo said Thursday that this is a dispute about who unions want to endorse."Who the unions support or don't support, that's up to the unions," he said. "Nobody's going to tell them what to do."Many unions have pulled their memberships of the Working Families Party after the Cynthia Nixon endorsement.Groups including Make the Road NY and the Working Families Party hosted an anti-Cuomo rally Thursday in lower Manhattan. They say they're worried because so much of their funding comes from the state and union membership."For us to be attacked by a Democratic governor whose instinct is to destroy us is concerning in many ways," Javier Valdez said.City Councilman Carlos Menchaca echoed the sentiment."Any threats, we're not going to take," he said. "And we're going to say 'Basta! No more!'"Appearing on the Late Show with Stephen Colbert, Nixon was asked why she's dropped "Nixon" from her campaign literature and just uses Cynthia."If I was given a choice, I'd rather be a good Nixon than a bad Cuomo," Nixon said. "We're a blue state, a proudly Democratic state, but we've got a governor in there who governs like a Republican."Cuomo scoffed at the accusation he's not liberal enough, citing his push for same-sex marriage, more gun control and a $15 hourly minimum wage."We got into this business because my father taught me that the Democratic Party at its best was a vehicle for social justice and social change, and that's what I've been doing since day one," Cuomo said.Those at the rally disagreed."Every single thing that (Cuomo) is claiming was organized over years by the same community groups that he's trying to off and starve now," said Bertha Lewis, of the Black Institute.When asked should Cuomo be worried about Nixon, Lewis said: "Yes, he should."----------