The leader of Europe's most powerful far-right party held a rally in London on Sunday, despite calls to ban him from the UK and strong opposition from anti-racism activists.

About 100 supporters of the Hungarian Jobbik party, which has been accused of stirring up hatred of Jews and Roma, gathered at Speakers' Corner in Hyde Park to hear Gábor Vona speak.

The venue of the rally was changed after Jobbik supporters were confronted in central London by more than 100 members of Unite Against Fascism and other groups. The home secretary, Theresa May, resisted calls last week to ban Vona from entering the country to hold the rally, which comes before elections in Hungary and on the eve of Holocaust Memorial Day in Britain.

Demonstrators on both sides began arriving at Holborn – Jobbik's expected meeting place – at around lunchtime. They quickly blocked exits from the London Underground station, forcing most of the Jobbik supporters to stay inside.

After the Hungarian contingent retreated back down to the train platforms, most of the anti-fascist activists left, apparently believing the rally to have been cancelled. However, it went ahead with relatively little opposition further west.

Sara Liptai, a 60-year-old Hungarian who lives in the UK, protested against the rally. She said there was danger to Britain in giving a platform to the leader of a party that has previously called for a register of Jewish people who pose a "national security risk".

She said: "They gain legitimacy by being able to speak freely here in a properly democratic setup. That is quite dangerous, for them to believe that it is OK."

Another Hungarian, Eva Keleman, who is in her 50s, said she wanted to protest against the "gathering force and power" of Jobbik, and other extreme-right movements. She said: "We are protesting against all sorts of discrimination, racism and any rightwing movements."

Jobbik supporters said they were not authorised to speak to the press, but one – who did not give his name – said: "There is either democracy, or there isn't." Vona spoke to his supporters for about 20 minutes.

The London assembly member and former Labour MP Andrew Dismore was vocal in his opposition to Vona's arrival in the UK. He wrote to the home secretary demanding that Vona be banned from holding the event.

Dismore, who as a Labour MP founded the annual memorial day, has called Jobbik "the most powerful outwardly fascist political party in Europe".

Speaking outside Holborn station, he said: "I think it's very important to send the message that we won't have hatred spread on our streets of London by the Hungarian Jobbik party.

"We have laws here against the incitement of race hatred. Jobbik party is a racist party which targets Roma, it wants to segregate them into ghettos, and it targets Jewish people.

"This is so reminiscent in Hungary of what happened in Nazi Germany in the 1930s and it should not be allowed to continue in the UK."

Dismore said the home secretary had made a grave mistake by not acting on a Hope Not Hate petition with more 14,000 signatures that called for Vona to be excluded from the UK.

Unite Against Fascism's joint national secretary Sabby Dhalu said: "Jobbik's fascist hatred has no place in a modern society here, or in the rest of Europe. Wherever fascists have a presence, racist, antisemitic and Islamophobic attacks increase."

The Metropolitan police said no arrests were made. A spokesman said: "We assisted the group to assemble for the purpose of lawful protest. The demonstration went ahead at an alternative location in order to prevent a possible breach of the peace."