It is often said that even the least prepossessing backbenchers arrive in Westminster thinking they could one day be prime minister. I have met a lot of MPs, and unless the human capacity for self-delusion is greater than I’d grasped, I find this hard to believe. In Keir Starmer’s case, however, the old cliche probably understates his ambition. If he didn’t believe he would one day move into No 10, I doubt he would be there at all.

It’s a hope widely shared by many Labour supporters. In a poll of party members last week, almost half named Starmer as their candidate of choice to replace Jeremy Corbyn, should the current leader stand down. I frequently hear his name cited by the faltering party faithful as they search for reasons not to abandon all hope. In fact, the only person I’ve come across who has nothing to say about his leadership prospects is Starmer himself. When invited to confirm that he would, whenever the vacancy next arises, throw his hat into the ring, he assumes a solemn expression and tells me rather stiffly: “There isn’t a vacancy.”

Such discretion is of course no more than political convention, and everything else he says when we meet sounds like the talk of a politician with big plans. The 54-year-old was only elected in 2015, and stood down as a junior shadow minister last summer to back Owen Smith’s leadership bid. But having been director of public prosecutions (DPP) from 2008 until 2013, and before that a senior QC, he is in no sense a novice.

Starmer was in the Commons on Wednesday when Westminster was attacked, and remained locked in the chamber for five hours. “I spoke to my wife and my family on the phone, but it hit me hardest when I got home, and there was my family, and I thought about the families of the victims who weren’t coming home.” He describes the news of casualties filtering through to the chamber, and the terrible silence that fell when PC Keith Palmer’s death was announced. Having worked closely with the intelligence and security services while DPP, Starmer knows more than most MPs about their operations, and sees nothing in this week’s attack to necessitate a radical extension of their powers. “If we profoundly change the way we go about our lives, then to some extent the terrorists have achieved what they want to achieve.” He strikes the sort of level, statesmanlike tone that would sound rather incongruous if coming from the backbenches, and he has no regrets about leaving them last summer.

Following Corybn’s re-election last year, Starmer rejoined Labour’s front bench team as shadow Brexit secretary, and claims the decision was pretty easy. “Brexit is so important, it would have been neglect of duty to simply sit it out,” and once Corbyn had won again, “the right thing to do is to get behind the leader, and do the best job you can, and that’s all I’m trying to do.”

But while this logic might have looked irrefutable on paper, in practice it has led him – as one friend and admirer put it to me – to “a very bad place”. A passionately pro-EU campaigner, whose north London constituency of Holborn and St Pancras voted heavily to remain, Starmer has spent recent months in the unedifying role of handmaid to Article 50. If many voters couldn’t tell which side Labour took in the referendum campaign, the party’s decision to impose a three-line whip supporting the Tories’ Brexit bill was unlikely to clarify the confusion. Anyone looking to Labour for robust opposition on behalf of the 48% will have been bitterly disappointed. And on top of supporting legislation he doesn’t believe in, Starmer has to affect loyalty to a leader we all know he didn’t want. A degree of pragmatic expedience is expected from anyone serious in top-flight politics, but to survive such a calamitously compromising chapter has begun to look like it might be a tall order, and I’m curious to see if he will still come across as the leader-in-waiting hitherto assumed.

We meet in his Portcullis House office, which is so brutally tidy that at first glance I assume he either hasn’t finished moving in, or is halfway through packing to move out. He looks faintly embarrassed as he explains that, no, it’s always like this. He laughs the military-grade order off as his antidote to the chaos at home – he and his wife have two young children – but it looks like more than that to me. His manner is affable and informal, with none of the typical politician’s guard, and feels authentic. He’s looser and less wooden than he can sometimes comes across on television, and quick to laugh. But the clutter-free desk suggests a severity of discipline I doubt I’d have guessed had we met somewhere else, and looks like of evidence of ruthless purpose.

Starmer justifies his Brexit bill strategy to me with the elaborate patience of someone trying to pretend this isn’t the thousandth time he has had to. “Everyone says: ‘What about the 48%?’ Well, I’m one of the 48%, and my family is part of the 48%, and many of my friends and colleagues are part of the 48%. But we had a referendum, and we had a result. You can’t logically then say, ‘I accept the result, but I won’t let the prime minister start the process.’” He accepts Labour might have unwittingly given voters the impression that its position on Europe is exactly the same as the Tory party’s. “Because all they know is that we’ve been fighting for process, so it’s been very frustrating.” The party is diametrically opposed to the government’s attitude to Europe, he says, but “it’s been very difficult to get that message across. But as we move now from the process, which we’ve been stuck in for the last few months, on to substance, the Labour party needs to be very loud and very clear in articulating the version of the future we are fighting for. Now we’re in a fight over what version of the future we want.”

Is this the fight Corbyn was talking about when he declared – only after all the legislation had been passed – that it “starts now”? Starmer winces. “I’m not going to comment on what Jeremy said about that.” He allows a dry, mirthless laugh. “I think that’s a question for Jeremy rather than me. That’s not what I think. The fight for me started before the referendum, and it was a fight during the referendum, and I did whatever I could to fight that fight, and I’m going to go on and fight for a version of our future relationship with the EU that reflects the values I believe in. If this goes the way I hope it will go, and will fight for it to go, there will be in due course a new treaty that is the EU-UK treaty. Something that, while it’s not formal membership, reflects our belief that we do things together with our EU partners; we don’t sever and walk away from them.”

Labour lost the vote last June, he goes on, not just because it failed to make a winning case for remaining, but also to offer any meaningful vision of a future.

“You don’t win elections by telling people what you’re against. We’re very good at listing things we don’t much like about what the Tories are doing. But you win elections by telling them what you’re for, what you’re going to change, what’s going to be better. And we’re not in that place. And that’s where we’ve got to go to. Labour only wins when it offers that big vision that actually means things to people, where they think, ‘My life, my family, my community, and my country can get better, and will be better if these steps are put in place.’ And Labour can do that, but it does it rarely.”

I ask for an example of what that vision might consist of. “An economy that works for everybody,” he replies. My heart sinks. That’s it? Seriously? “I don’t mean that just as a slogan.” But that’s exactly what it is, I say, and he laughs. Then, like a rabbit from a hat, from vacuous banality he suddenly conjures up a glimpse of his vision of a Labour party that people might actually want to vote for.

While shadow immigration minister, he says, he travelled all over the country talking to people who run businesses, from sole traders to corporate CEOs, and asked them all the same question: what is the single biggest inhibitor to your success in the next three to five years? He says he got the same answer every time: skills. “And when you get that level of consensus, you realise there is a real political failure that needs to be addressed, and so one of the headlines of an ambitious, bold project for the future is an absolute skills agenda.”

Labour’s vision for public service, he goes on, can’t simply be to spend more money. “It’s also partly about thinking how to deliver public services differently. I was really struck, as a frontline public servant, by how siloed everything is. So criminal justice operates in a different sphere to education and to health, and yet if you want to track those most likely to end up in prison, you can start with kids in primary school – because if they’re beginning to fail there, and if they are then excluded from secondary school, they are more likely to go into a life where they’ll end up in criminal justice. So if you want a really effective criminal justice strategy, you don’t build bigger prisons, you invest money in young kids – and you accept that it’s going to take years to work through, but it’s a more effective strategy. So I think there’s a lot to be said about what a future-looking Labour programme could look like, about which most people would say: ‘I can see the sense in that and I want to be part of that.’ And it’s when those things come together that Labour actually wins elections. And we need to win elections.”

Keir Starmer speaking during the article 50 debate in the House of Commons in January. Photograph: AFP/Getty Images

At happier times in Labour’s history, I’m not sure this would pass for cutting-edge policy innovation, or distinguish the speaker as an obvious future leader. Currently, however, it ranks as one of the few attempts by a Labour MP I can recall to apply progressive values to a problem the public really care about, and find a solution worth voting for. On Monday he will deliver a big Chatham House speech on the party’s future direction, so I’m curious to know who he shares his thinking with, for if he’s serious about shaping a new path for Labour he must be having these debates with colleagues. Is it Corbyn and John McDonnell? “It’s across the party.” Who does he phone when he wants to talk through a new idea? “Oh, various colleagues around in the PLP.” Are these conversations taking place within the shadow cabinet? “Er, it has to be across the whole of the party.”

The identity of his collaborators is never disclosed, but from his cagey evasions one thing is perfectly clear: they do not include Labour’s leadership.

The dexterity with which Starmer navigates his way through all mentions of Corbyn is both frustrating and highly impressive. Loyalty and honesty are tricky to reconcile, but somehow he pulls it off. For example, he talks passionately about the imperative of winning elections. “That’s all I’m here for, that’s why I came into it. I didn’t come into it to be shadow anything. I came into it to change lives.” But when I ask if winning matters as much to the leadership, he offers: “Um, well, I said after [Labour’s recent byelection defeat in] Copeland, we can’t become casual about failure. We’ve got to get from where we are to where we need to be. And there needs to be that focus. So, you know, we’ve had some pretty frank exchanges.” Was he surprised to have to spell out the importance of winning? “I would hope that that is obvious to everybody in the Labour party.” How badly does he think the leadership want to win? “There needs to be a renewed urgency and focus on winning, and getting ourselves into a position to win.” Twice I ask if he could describe the leadership as a smart, slick, competent operation, and twice he avoids answering.

If Starmer can survive being Corbyn’s Brexit secretary – and I would be amazed if he doesn’t – it does feel like only a matter of time before he leads Labour. I come away feeling oddly hopeful. Some have questioned whether he has the necessary charisma for leadership – but with Theresa May in charge of the Tories, this doesn’t strike me as much of a worry.

He leaves me with an analysis of May’s political future that feels even more encouraging. “What I observe is her characteristic of being so uncomfortable with scrutiny and accountability, and I think that leads her to a position where she’s more and more isolated. It’s beginning to be her defining characteristic. And I don’t think it’s sustainable. The moment it begins to go wrong, once you’re isolated, you haven’t got colleagues and partners to call on. That’s when you’re in trouble.”