Prime Minister David Cameron flew off to Friday's summit of EU leaders in Latvia with the tail wind of his election victory and the head wind of Thursday's immigration figures leaving the whole question of whether Britain will ultimately vote to stay in or quit the European Union in 2017 pretty much where it was before the election.

Opinion polls aren't very helpful -- not that they're viewed right now with much affection after the 2015 general election debacle. For what they are worth, they've been inconclusive about how the referendum will turn out in recent years.

A YouGov poll in February cheered the In camp putting the Ins on 45 percent with the Outs on 35 percent. But looking at YouGov's polling over the last five years on the subject -- see chart here -- sober appraisal would suggest that it could really go either way.

But here are some reasons why, as things stand today, the Outs have reason for cautious optimism.

First, we have to contend with the same confirmation bias in favour of liberal-progressive causes as confounded the general election polls. The anti-EU case has been led by UKIP and eurosceptic members of the Conservative Party. These are viewed with disdain by polite society, giving a disincentive to Outers to be honest about their real preferences.

Second, there's the turn-out-the-vote factor. Referendums, like all elections, are often determined by each side's ability to get out their voters. There are far more people in Britain who actively and passionately oppose our membership of the EU than there are who passionately support it.

People may tell pollsters they are broadly in favour of remaining in the EU. But if enough of those people are only expressing a reluctant, unenthusiastic support, they may skew the referendum result in favour of leaving by not having had the energy on polling day to get out and cast their vote.

Third, Cameron is definitely not going to be able to get an end to the free movement of people, and he's not even wasting his time asking for it. His efforts to stop benefit tourism may or may not be successful -- if they're not, it really is curtains for the UK's EU membership -- but uncontrolled migration across the EU will still be a reality come the referendum.

Anyone who might be holding off from expressing definite support for leaving so as to give Cameron the chance of addressing their concerns on this score will soon understand that it really is a zero sum game. If we're in the EU, we can't control the numbers coming in from Europe.

The In side will counter that inertia is always a good bet when people are faced with the prospect of significant change. It's a decent argument, and it corresponds with our intuitions about human behaviour. When the polls open in 2017, there will certainly be some who will vote to stay in because they are fearful of the consequences of change.

That is why the In side is so heavily focused on scare tactics. They want as many people as possible to be fearful for their jobs and their futures, hoping that that will be enough to hold us in.

It may be. But building one's entire strategy on scaremongering is a dangerous game in its own right. So many scare tactics -- 3 million jobs at stake if we quit, for example -- have already been used that it is difficult to see how the In side can ramp things up any further.

When they've already played their best cards, it will be hard for them to sustain the momentum in the two year run-up to Referendum 2017.

On balance then, we think that as things stand the energy of the Out camp will trump the scaremongering of the In camp. When David Cameron returns empty-handed from Latvia, the first decisive moves towards Brexit may have already begun.