The Swiss are to decide this weekend whether to ban minarets on mosques, in what is in effect the first direct vote in a European country on Islam and the practices of Muslims.

The controversial referendum on Sunday, accompanied by a prohibition campaign denounced as racist and in violation of human rights, is the latest tussle in Europe over the limits of multiculturalism and immigrant lifestyles.

Pushed by anti-immigrant rightwing populists, it has triggered months of debate in a country that uses direct democracy for single-issue politics. The referendum has turned into much more than a vote on architecture and urban planning.

"The minaret has got nothing to do with religion. It's a symbol of political power, a prelude to the introduction of sharia law," argued Ulrich Schlüer, of the rightwing Swiss People's party, an architect of the campaign.

Two years ago the SPP became the strongest party in Switzerland, with an anti-immigrant election campaign that featured posters of three white sheep kicking a black sheep off a red and white Swiss flag. UN experts and human rights activists condemned the campaign as overtly racist.

This time the SPP has plastered the country with posters showing the same flag as a base for several black minarets, portrayed as missiles, alongside a woman clad in a black burqa. Church leaders, the Jewish community and Muslim leaders have all opposed the campaign. The foreign minister, Micheline Calmy-Rey, warned that a vote in favour risked turning Switzerland into "a target for Islamic terrorism". The city of Basel and other towns have proscribed the incendiary posters.

Amnesty International said: "Freedom of religious belief is a basic human right and changing the Swiss constitution to ban the construction of minarets would clearly breach the rights of the country's Muslims."

UN human rights experts have said the proposed ban violates freedom of religion and liberty. The Swiss justice minister, Eveline Widmer-Schlumpf, has agreed, declaring that it would breach anti-discrimination laws and rights to free religious observance, raising the question of why the campaign has been allowed.

The rightwing anti-immigrant lobby has led the campaign, but it has been joined by some secularist leftists and liberals fiercely critical of Islam. Julia Onken, a prominent feminist and psychologist and bestselling author of self-help books, last week called on women to vote for the ban. "Mosques are male houses, minarets are male power symbols," she declared. "The building of minarets is also a visible signal of the state's acceptance of the oppression of women."

There are about 400,000 Muslims in Switzerland, 5% of the population. Most are immigrants from the Balkan regions of Kosovo, Albania and Bosnia and are not practising Muslims.

Of about 150 mosques or Islamic prayer houses in Switzerland, only four have minarets.

"And we don't want any more," said Schlüer. "There's no sense in banning them once you've got hundreds. What's the point in waiting for that?"

Prohibitionists and prominent SPP officials argue that the four minarets are the thin end of the wedge, to be followed by sharia law, honour killings, oppression of women and stoning.

Opinion polls indicate the anti-minaret movement will lose on Sunday, but a sizeable minority – at least one-third of voters – could support the call for a ban, and their ranks appear to be growing as the referendum nears. People are to vote on the statement: "The construction of minarets is forbidden." This would be added to article 72 of the Swiss constitution if endorsed.

The government, the political mainstream and the powerful business lobbies in one of the world's richest societies all oppose the ban for reasons of tolerance, trade prospects and fears of Muslim radicalisation and a backlash. The business lobby is worried about the potential impact on Swiss exports to the Middle East, pointing to the experience of Denmark in the wake of the Muhammad cartoons controversy four years ago.

The arguments over the place of Islamic architecture in western Europe are not confined to Switzerland, with big mosque-building projects currently the focus of opposition in places as diverse as Copenhagen and Marseille. Italy, Austria, Germany, and the Netherlands have all been locked in disputes over mosques in recent years as some of the biggest Islamic prayer houses and cultural centres in Europe spring up.