It is no secret that Ed Miliband has a problem. While the latest polls suggest that his party remains on course to return to power, they also reveal that one of its major obstacles to achieving a wider breakthrough is Miliband himself. Voters are just not warming to Ed.

This dilemma has been summarised by the pollster Peter Kellner, who noted that while a majority of voters support some of Miliband's policies – from renationalising the railways to curbing the power of landlords – the Labour leader is failing to convert enthusiasm for his ideas into loyalty to him.

We can build on this picture by taking a look at the latest British Election Study. At first glance, the story that it tells is a familiar and bleak one for the left: the percentage of Labour's own voters who openly say they dislike Miliband (26%) is six points higher than the equivalent among Conservatives (20%). But even more worrying for Labour is that its leader is also failing to inspire his most loyal voters. Of those who turned out for Labour in both 2010 and 2014, just 61% like Miliband. The equivalent figure among loyal Conservatives is 73%.

Graph Photograph: Graph

That a leader of the opposition is struggling to rally wholesale support among his own base should already ring alarm bells; but Miliband is also failing to impress Labour voters on a specific set of social and cultural issues that are central to the minds of many voters, and which have already fuelled the rise of Ukip in Labour as well as Conservative areas. On the key issues for Ukip – immigration, Europe and the economy – Miliband really does have a problem.

Over half of Labour voters who prioritise immigration feel negatively or indifferent toward Miliband. Compare this to David Cameron; while he has continually failed to achieve his own goal of reducing net migration, only one in three Conservatives who prioritise this issue feel negatively or indifferent towards him.

It is a similar story on Europe, an issue Ukip is merging with immigration. Over two-thirds of Labour voters who think that European integration has gone "too far" readily say they dislike Miliband. Sure, there will always be fewer voters on the left than on the right who are animated by these issues. But it should be remembered that these issues that overlap with concerns over national identity also look set to guide British politics over the coming years. Immigration, for example, is pushing ahead of the economy as the number one issue in Britain. So whether seen in terms of Labour voters or more broadly, it simply does not look good for Miliband.

In fact, even on the economy Miliband is struggling among his own base. The good news is that most Labour voters who prioritise the economy – some 62% – like Miliband. But given all the talk about responsible capitalism, the cost of living and tackling things such as energy prices, this still feels far from convincing. It lags well behind the 73% of Conservatives who prioritise the economy and who like Cameron. And it masks a sizeable 38% of Labour voters who prioritise the economy while disliking or feeling indifferent toward Miliband. Put another way, more than one in three economy-minded Labour voters do not see a reason to like Ed Miliband. For Cameron the ratio is nearer one in four.

Only on social welfare issues – such as pensions, housing, and spending on social services – does Miliband have a very slight advantage over Cameron in terms of how much he is liked by his own supporters. Yet even among Labour voters only 7% prioritise social welfare issues, far less than the percentage of Labour voters who prioritise immigration or the economy.

All of this points to the conclusion that on issues at the centre of British politics, the Labour leader is struggling to inspire a convincing majority of his own followers.

There is also some evidence to suggest that Miliband might also be contributing to Labour's vulnerability to Ukip; 62% of Labour voters who switched to Ukip in 2014 elections dislike or feel indifferent toward Miliband, compared with a far lower 45% of Conservative defectors who feel the same way about Cameron. While more Conservatives are switching to Ukip, this suggests that in relative terms Cameron might actually have an easier time than Miliband in winning some of them back. Around 80% of Conservative voters who are already planning to return to the fold in 2015 like Cameron, but only 61% of the Labour voters say they are thinking of coming back like Miliband.

The business consultant Peter Drucker once said that effective leadership is not about being liked or personal attributes; it is about results. That might be so in business but in political science we know from years of research that party leaders are profoundly important to voters' decisions about how to vote. For Labour to achieve its desired result and make a convincing breakthrough, Miliband must ultimately overcome his own weakness.