Talking to David Cameron about the refugee crisis is a bit like discussing human rights with your bank manager. He understands why you’re upset, he really does – his heart bled when he saw those awful pictures. But you’ve already made a generous donation to a refugee charity, so why not leave it to the politicians to get on with creating stable governments in Syria and Libya?

The Prime Minister is of course one of those politicians, even if he sometimes looks like a prosperous company chairman. His election victory in May suggests quite a lot of voters regard him as competent, at the very least, and it’s unlikely that the consequences of the war in Syria were high on their agenda. Until last week, that indifference allowed Cameron’s Government to remain remarkably unscathed by its attitude toward desperate refugees, which could be characterised as a classic exercise in Nimbyism.

The UK Government has given more than £900m towards maintaining refugee camps in Lebanon, Jordan and neighbouring countries, making it the second-largest aid donor, behind the US, in the world. But conditions in those camps are so dreadful, with families sweltering in summer and freezing in winter, that expecting people to subsist in tents in the Middle East isn’t any kind of solution. Against that background the British response – accepting fewer than 5,000 Syrian refugees since the conflict began four years ago – is pitiful, especially at a time when Germany is preparing to receive 800,000 asylum applications.

The only thing that can be said about the British Government’s stance is that it’s consistent, although consistent mean-spiritedness is hardly something to boast about. Last year, Cameron refused to participate in a UN programme to resettle 30,000 people fleeing Syria; instead, the Government set up a vulnerable migrants relocation scheme, designed to help the elderly, the disabled and victims of sexual violence and torture. Plenty of Syrians fall into these categories, but by this June just over 200 individuals had been admitted to the UK under the scheme. The Government has refused to take part in two EU programmes to relocate refugees in Europe, one involving 40,000 people and the other 20,000, arguing that providing legal routes would be a “pull factor”.

After getting away with a policy of naked self-interest for so long, this summer’s escalating crisis has left Cameron floundering. He looked uncomfortable when he had to deal with questions about the grim photographs of a boy’s body on a Turkish beach, tartly reminding journalists that he’s a father himself. That kind of response usually spells trouble for politicians, who are supposed to put policy before personal considerations. Two years ago, Downing Street had to deny that Samantha Cameron, who had recently visited a refugee camp in Lebanon, was influencing government policy on Syria.

It’s unusual to see the Prime Minister so exposed, teetering on the edge of outright irritation as he fends off claims that the UK isn’t doing enough. It’s also a lesson to populist politicians: of all the subjects that Cameron might have been worried about, tough attitudes to refugees must be the one he least expected to trip him up. Only last month it was business as usual, with the Prime Minister describing people trying to get to the UK as a “swarm”, and talking about not allowing them to “break into our country”. He sounded like an angry householder who might, if pushed, dig out that old shotgun he’s been keeping in the under-stairs cupboard.

Now even The Sun is urging the Government to let in more refugees, demonstrating that Cameron has misjudged the public mood. In a tight spot, he fell back on more of the same: he’s announced another £100m to support refugee camps on Syria’s borders, a policy that has already been shown not to work. He’s bowed to pressure and said that the UK will accept “thousands more” Syrian refugees from those camps, but he won’t put a figure on it. He’s also put himself at odds with key European leaders, whose support he will need when he goes into difficult negotiations over the terms of the UK’s membership of the EU.

It’s worth recalling that Cameron has spent years talking about “British” values. It’s a weaselly use of language, designed to appeal to Tory and Ukip voters who want to think of themselves as decent human beings without giving too much away to foreigners; “British” values are for people who think universal human rights are about silly things such as accusing UK soldiers of war crimes and giving council houses to terrorists. Now it’s threatening to rebound, with “British” values emerging as a distinct and lesser species of compassion. What it means, it seems, is paying aid agencies to ensure that people fleeing Assad’s barrel bombs and the black-clothed fanatics of the so-called Islamic State mostly stay away from our country.

What’s even more extraordinary, bearing in mind the indelible stain placed on Tony Blair’s reputation by the Iraq war, is that Cameron hasn’t been called to account for his massive misjudgement over Libya. Once again, a British prime minister sanctioned military intervention without planning for the aftermath; it has created perfect conditions for people-smugglers, putting thousands of Syrian, Iraqi and African refugees at risk of drowning in the Mediterranean.

Cameron has always looked like a fair-weather politician: confident, smooth-talking and persuasive as long as things are going well. He’s staked a claim to values in the vaguest possible terms, but now they’re looking decidedly hollow. Few would have predicted it, but the refugee crisis is fast becoming the biggest test of his premiership – and his character.

Whether he has the guts to disown his harsh rhetoric and show genuine leadership is another matter.