Since news of assaults on women in Cologne and Sweden surfaced, it feels like open season on refugees and migrants. We seem to be witnessing a Europe-wide spasm of self-doubt, a cold shower of realism after the warm righteousness of hospitality.

In Germany, New Year’s Eve celebrations turned nasty as incidents of sexual assault, many by Arab or north African migrants, were reported; by this week police investigating events that night in Cologne had received more than 200 complaints of sexual offences. And in Sweden, attacks on women were undeclared by authorities allegedly because the perpetrators were immigrant men, reportedly of Afghan origin.

The pendulum is abruptly swinging back towards closed borders and punitive measures against those who have already been let in. Suddenly it seems acceptable for even liberals to ask whether women’s rights and immigration are compatible.

Let us pause here for a moment and consider what that means: immigrants of a vaguely Middle Eastern or Asian character are being catalogued as threatening to women’s rights, liberal values, and the social cohesion of Europe.

The problem lies in the tenor of the conversation that preceded news of these assaults. It was either naively optimistic – Europe cast as the liberal superpower embracing needy refugees in a bear hug of humanitarianism; or apocalyptic – swarms of migrants, traumatised, damaged and of inferior cultural stock would place an intolerable burden on the state and then literally assault their hosts.

The pendulum is swinging back towards closed borders and punitive measures against those who have already been let in

The first does not acknowledge the profound cultural and economic implications of admitting a large number of people from very different societies, while the latter refuses to accept the moral duty to offer refuge or the fact that not all migrants are bad people.

The balance is somewhere in between. Is it Europe’s moral duty to take in asylum seekers and refugees? Yes. Do most migrants go around in big groups groping and raping women? No. Among them will there be some who are economically deprived, culturally incompatible or downright criminal? Absolutely. It is hard to believe that among the thousands of Syrians who have entered Europe, not a single member of Bashar al-Assad’s vigilantes slipped through.

The focus instead should broadly be on two things: in the short term, tackling the logistics of the influx by, for instance, enhancing policing and punishing those who transgress; and in the long term, developing clear integration policies with political backing and funding. This can all be done while acknowledging that people might have legitimate concerns about criminal behaviour or even just general cultural dilution, where the character of their town or neighbourhood changes dramatically in a short time.

‘The problem lies in the tenor of the conversation that preceded news of these assaults. It was either naively optimistic or apocalyptic.’ Photograph: Frederic Seguin/Rex/Shutterstock

The discussion should also be about preventing such attacks in future, and identifying the conditions in which they are more likely to take place.

Where migrants sit on the economic spectrum and how socialised they are into the mainstream is relevant. Large groups of unemployed, disenfranchised young men are probably going to be a problem in most cultures. Some religious or cultural attitudes should also be recognised as problematic, without attributing aggressive behaviour simply to religion or culture. The ugliness comes in when all migrants are lazily rejected as a threat to society, as if faceless brown people were always going to be bad news.

Striking the right balance means understanding without justifying, and keeping in check strong populist impulses. It means not lumping together every incident across Europe as indicative of the original sin of immigration. Second or third generation crime in Rotherham is not related to that of newly arrived asylum seekers in Germany or long-term migrants in Sweden. Discussing these crimes only through the lens of identity collapses all its different features into one bogeyman of an assailant, improbably both a sexually assaulting drunken lout and a Muslim salafi. This is hysteria.

Expecting the victims of crime or the general populace to embrace the nuances of long-term policy is a lot to ask. But instead of answering whether we care more about women or immigrants, we should reject that question altogether. If everything is reduced to identity, xenophobia and racism quickly creep in.

Liberal values have to be able to withstand difficult times. A certain narcissism is challenged by those migrants, especially refugees, who fail to play the part of grateful recipients of western help – but the answer isn’t to insist a mistake has been made. There has been no mistake. The test is to acknowledge that liberal values and tough questions are not incompatible, and to stick to the commitment to helping the poor, war-ravaged and dislocated.

• This article was amended on 14 January 2015 to clarify details of the offences reported to police in Germany. An earlier version referred to “up to 40 incidents of sexual assault”.