FABRICE COFFRINI / AFP / Getty Images An asylum seeker at the entrance of a military bunker in the remote Alpine village of Realp, central Switzerland, on Aug. 9, 2013

The official website of Bremgarten, a picturesque Swiss town of 6,430 residents located about 20 km (12 miles) from Zurich, bids visitors “a hearty welcome” and boasts of the “wide range of sports and leisure facilities” available in the community. But for two dozen new arrivals — and 130 more expected in the future — the town’s hospitality and its recreational options remain untested.

That’s because local authorities have announced last week that asylum seekers housed by the federal government in Bremgarten’s army barracks won’t be allowed to move around town freely or use the local swimming pool and other sports facilities. “For security reasons, we decided not to allow access to these areas,” the town’s mayor, Raymond Tellenbach, explained in a TV interview. He did not specify what sanctions, if any, would be imposed on those breaking the rule.

And Mario Gattiker, head of the Federal Office for Migration, told the Tages Anzeiger newspaper last week that the restrictions are meant to “prevent a situation where 50 asylum seekers all want to use a football pitch or the pool at once, which could lead to friction and resentment” from the locals.

Bremgarten’s decision has quickly drawn criticism from human-rights groups and incited accusations of racism, since the refugee center’s residents come from Africa and Asia. The announcement coincided with the news that an exclusive Zurich boutique refused to sell American media mogul Oprah Winfrey an expensive handbag last month, ostensibly because she is black — the store’s owner has denied that the incident was racist, calling it a “misunderstanding caused by the language barrier.” (Even though Switzerland’s tourist body apologized to Winfrey, Swiss animal-rights group Vier Pfoten, or Four Paws, criticized her for wanting to buy a $35,000 crocodile bag, saying she should have been aware of cruel methods used to obtain the skins of exotic animals.)

(MORE: As Europe Reels, Switzerland Builds New Barriers Against Immigrants)

Also recently, local media reported that the country’s recruitment agencies are refusing to find jobs for French workers because Swiss employers consider them to be “lazy and arrogant.” Although the three events are unrelated, they deepen a growing international perception of Switzerland as a country where racial profiling and discrimination have become increasingly rampant.

But Swiss Justice Minister Simonetta Sommaruga, whose department oversees migration, told reporters soon after the Bremgarten imbroglio that asylum seekers would not be prevented from using public facilities. “The basic rights, including the freedom of movement, are guaranteed for everybody,” she added.

That may be, but the issue of asylum and plans to open more refugee centers — mostly in small towns and villages — remain contentious. (The federal government has the right to use the premises it owns across the country to house asylum seekers, without asking the permission of the local authorities.) Roman Staub, the mayor of Menzingen, a town of 4,300 inhabitants that will house between 100 and 200 asylum seekers in 2015, echoed the unease expressed by his Bremgarten counterpart, telling a Swiss TV station on Aug. 6 that asylum seekers in his community would not be allowed to enter “sensitive areas” like schoolyards to prevent contact with children. Bremgarten is hardly the first town to come down hard on the refugees. Alex Sutter, co-director of humanrights.ch, a Swiss human-rights group, says that the practice of barring asylum seekers from certain public spaces is common, “but, until now, nobody has noticed or cared about it.”

There is no specific reason why the Bremgarten ban made the news when others didn’t, but the country has experienced a recent spike in the asylum requests, mostly due to the 2011 Arab Spring uprisings; nearly 50,000 people have sought asylum in 2012, the highest number since the refugee influx during the 1999 Balkans war. And in June of this year, the Swiss voted in favor of a stricter asylum law that includes measures such as the exclusion of ex-soldiers who apply for asylum as military defectors — a measure interpreted as a bid to block many Eritreans fleeing forced enlistment in their own country (Eritreans comprised a considerable proportion of Switzerland’s asylum seekers last year).

(MORE: Why Rich Switzerland Is Livid About Rich-Executive Payouts)

For all the talk about xenophobia, however, Switzerland has had a generous asylum policy, in line with its humanitarian traditions, which include hosting the U.N. refugee agency as well as the International Red Cross. And while 50,000 refugees may not seem like a high number, Switzerland is a small country of only 8 million people. According to government figures, the nation has one asylum seeker for 332 inhabitants, a higher rate than the European average of one asylum seeker for every 625 residents.

If the issue continues to stir controversy, it is due in large part to the rhetoric of the far-right Swiss People’s Party (SVP), which has long insisted that the majority of asylum seekers are not “real” refugees escaping persecution in their home countries, but rather regular immigrants who want to go to Switzerland for economic reasons. These “false” refugees, SVP says on its website, live off generous welfare benefits while turning to life of crime and drug dealing.

At least some of these claims are based on facts. Police records show that asylum seekers are responsible for many of the cases of theft, burglaries, sexual assaults and other infractions committed in their host communities. These statistics have fueled SVP’s antiasylum stance, which has trickled down to the public, sparking fears about safety in places that house refugee centers. (Switzerland still has one of the world’s lowest crime rates.) In recent years, the party has used Switzerland’s system of direct democracy, which allows citizens to draft new laws and force nationwide referendums on legislation such as the ban on building new mosque minarets and the mandated deportation of foreigners who commit violent crimes in Switzerland (both initiatives were affirmed by voters).

Still, human-rights activist Sutter denies that the Swiss are racist or hostile — certainly not more so than other countries dealing with the influx of refugees. “The ban forbidding asylum seekers access to certain areas will be lifted as soon as a court declares that it violates the constitutional right to move freely,” he predicts. In the meantime, though, the residents of Bremgarten’s asylum center may well have to stay out of the town’s public pool — or they could find themselves swimming against the tide.

MORE: Switzerland: Are Its Days as a Tax Haven for Foreigners Over?