Protesters with clashing ideals faced off in front of City Hall on Saturday afternoon, with what began as a peaceful anti-racism demonstration escalating to violence.

The rally started with dozens of anti-racism demonstrators standing in Toronto's Nathan Phillips Square, holding signs featuring phrases like, "Fear is the prison, love is the door," as part of a planned protest about sensitivity towards Muslims.

"In the last about month and a half, there has been a really alarming rise of alt-right groups, like the Soldiers of Odin and the Jewish Defense League, and others having a really visible presence at Toronto City Hall," protester Sarah Ali, from the Organizing Committee Against Islamophobia, told CBC Toronto after the rally.

Ali called it "frightening," noting a rise in attacks on Muslim communities in recent years.

"We are not going to tolerate Islamophobia, we're not going to tolerate white supremacy, and we're not going to tolerate fascism on the streets of Toronto," she said.

Roughly 40 police on scene

Around a dozen counter-protesters at the scene were clad in clothing emblazoned with Soldiers of Odin, the international anti-immigrant street patrol group that began in Finland, and carried signs reading "Justin Trudeau must go."

At one point, some people began chanting, "Nazi scum off our streets."

Fighting began among those gathered and one of the Soldiers of Odin protestors was punched in the face.

Around 40 police were on scene and, at one point, used their bicycles to separate the groups.

At least five people at the scene were apprehended by police but it's not clear how many were arrested or released.

Toronto police have not confirmed how many people may have been arrested or charged despite repeated requests for comment from CBC Toronto.