Bill de Blasio is the first New York mayor for 21 years to boycott the St Patrick's Day parade over its ban on gay participants – but is he doing enough?

For the last 10 weeks, Bill de Blasio, the fledgling mayor of New York, has been painting a fresh face on this endlessly changing city. Under the banner "a tale of two cities", he has pledged to overcome the growing gulf between rich and poor and re-establish New York as a global hub of progressive politics.

But in the last few days he has been embroiled in a tale of two cities of a different order. Not rich versus poor, but tolerant and modern versus bigoted and antiquated.

The focal point is the St Patrick's Day parade, the oldest Irish tradition in America, that has been held every year since 1762, more than a decade before the declaration of independence. On 17 March, 200,000 marchers, many in city uniform, will strut up Fifth Avenue from 44th Street to 79th Street in front of a million-strong crowd in celebration of all things Irish. Well, not all things Irish. Not gay or lesbian Irish. In 1991 a gay group that gained an invitation to march was showered with abuse from spectators, prompting organisers to institute a ban the following year. Since 1993, when the federal courts sanctioned the ban, the parade's organisers have blocked the attendance of gay individuals or groups who openly display their sexual identity.

A similar prohibition has existed in the St Patrick's Day parade in Boston since 1995, when the US supreme court ruled it was the organisers' first amendment right to dictate who they allowed to march.

This perennial sore, which has provoked protests every year for more than two decades, has now erupted into the public glare, partly as a result of the stance taken by De Blasio, who has broken with tradition and vowed to boycott the proceedings.

The move is in tune with the mayor's actions in his first two months in office, in which he has attempted to kick the city, sometimes squealing, in a liberal direction.

He has waged a very public fight with the governor of New York state, Andrew Cuomo, over raising taxes on wealthy New Yorkers; pushed his plan for universal pre-kindergarten education; put a stop to the controversial policing tactic of stop-and-frisk; and appointed a slew of progressive activists to top city hall jobs.

With all that under way, De Blasio could hardly stand by and watch impassively as the St Patrick's Day parade went ahead, anti-gay ban stubbornly in place. As a result, on 17 March the parade will go ahead without the mayor of New York in attendance for the first time in more than 20 years. De Blasio will earn himself the distinction of being the first mayor since David Dinkins in 1993 to boycott the event.

Last weekend De Blasio underscored his decision by turning up at a counter-event called the St Pat's For All parade in Queens. "This parade is what New York City is all about," he said sparingly, without alluding directly to the spat with the official parade.

For seasoned observers of New York, such as Tom Finkel, editor-in-chief of Village Voice, the surprising element of De Blasio's stand is how long it has been in coming. "He clearly feels the climate is ripe for this – his predecessors [Michael Bloomberg and Rudy Giuliani] didn't judge it expedient to cross this line in the past."

Finkel believes the fact the ban still exists in 2014 underlines the multifariousness of New York, or as Walt Whitman famously said about himself, that it "contains multitudes". "When New Yorkers look outward we are tempted to see the world as a very progressive place, but if you look a little closer – even inside the city – you find it's not so simple."

And yet a wind of change is blowing forcefully across America. Seventeen states, including New York, have incorporated gay marriages, and even the most conservative states such as Arizona and Kansas have held back from enacting overtly discriminatory anti-gay legislation, for fear of damaging the local economy.

Which leaves the New York parade looking all the more retrograde and anomalous, bizarrely so for a city that lays claim to being the progressive capital of America. So what has the parade committee to say about all this?

The organisers did not respond to a request for comment from the Observer. It is perhaps a sign of the times that a prominent supporter would defend the ban only on the basis of anonymity. The individual, who works for one of the parade's big sponsors, said that the story was far more nuanced than LGBT campaigners had suggested. "This is a parade that celebrates the Irish Catholic community in America. We want to be tolerant and accepting," he said.

So why wasn't the parade tolerant and accepting?

"The parade committee has been guarded about keeping politics out of the parade. It is not anti-homosexual, it merely wants to prevent people carrying signs that affirm homosexuality."

The sponsor went on to suggest that gay and lesbian groups were actively avoiding applying to march because that suited their political purposes. He recommended they set up a group in honour of Fr Mychal Judge, chaplain to the New York City fire department who died in 9/11 and who was revealed after his death to have been a non-practising gay man. "They could march under his name and avoid words like 'pride' or 'homosexuality', and that might be fine," he said.

A similar approach has been taken in Boston this year where parade organisers have been in groundbreaking, but so far fruitless, discussions with gay rights group MassEquality. The sticking point was the insistence by the parade committee that marchers not wear anything that signalled their sexual orientation.

"We made it clear that we would only march if LGBT people are able to march openly and honestly," said MassEquality's director Kara Coredini. To which the head of the parade committee, Philip Wuschke, replied: "We gave them what we figured was reasonable. They wanted it all."

Emmaia Gelman, whose ancestors came from Co Cavan in Ireland, runs the blog of the New York-based LGBT group Irish Queers. She explained why she hadn't applied to march: "Why would I want to? I don't want to march with guys who hate me."

Her only objective, she said, was to put an end to the homophobia that the parade enshrined. In that regard, she and her fellow campaigners were disappointed that in their view De Blasio had not gone far enough. The mayor might be boycotting the event himself, but, ignoring the demands of protesters, he has made clear he will allow officers of the NYPD and fire department, who make up a large proportion of the marchers, to attend if they wish.

That has given Bill Bratton, the media-savvy new police commissioner of New York, space to announce that he will attend. "My sister is gay," Bratton said, a remark that failed to impress LGBT campaigners.

"City officials, whose salaries are paid for by the people of New York, absolutely do not have the right to march in a homophobic parade. That's a hard message for De Blasio to give to police officers about their favourite parade, but it's still the right thing to do," Gelman said.

So the 2014 St Patrick's Day parade promises to be another lively affair, and not just because of the copious amounts of alcohol that will flow throughout the city. For De Blasio, the dispute threatens to become a persistent headache that could run throughout his term in office, dragging on him as he struggles to revive New York's reputation as the world's greatest liberal city.