Not long before the new Labour leader was announced, Chuka Umunna bumped into a constituent in a Streatham supermarket car park. “I asked him who he was supporting, and he said: ‘Well, you’re not going to like it, but I’m...

Not long before the new Labour leader was announced, Chuka Umunna bumped into a constituent in a Streatham supermarket car park. “I asked him who he was supporting, and he said: ‘Well, you’re not going to like it, but I’m voting for X.’” (Though the former shadow business secretary declines to name the mystery contender, it seems clear that X equals Jeremy Corbyn).

“My constituent told me that he just wanted a fight; that he wanted a leader who would say what he thought in the media and the Commons about ‘these damned Tories.’ He accepted that would not get us elected, but he said the election was not for a long time. I reminded him that when we last chose that course, the extreme injustice of years of Tory government boiled over, and we had two riots in the 80s in my constituency.”

Umunna accepts that there were diverse triggers for the Brixton riots. “The primary cause was racism in the police, but [Lord] Scarman found that a huge contributory factor was poverty and inequality.” Is he really suggesting that a Labour party led by Corbyn could pave the way for violence and uprisings? Though the new leader’s name is never mentioned, it appears so.

“I really hope we won’t see a repeat of the social unrest we had during the last long period of Conservative government. [But] I wouldn’t dismiss it. The longer the Tories are in, the more social injustice we will see. If history teaches us anything, it’s that if you want a more just world, then at some point electability has to come into view.”

When we first meet, the latest chapter in Labour party history is still unwritten. Although the announcement of the new leader is then some days away, it is clear that Corbyn has already won. The impending landslide is inscribed not only on betting slips and in pollsters’ forecasts but on the face of Chuka Umunna, who withdrew his own bid for the leadership. At the end of a long and wearying contest, the candidate-who-never-was appears more exhausted than the actual contenders.

We speak again exactly 24 hours after Corbyn stormed to victory. Although neither of us knows it, this is to be Umunna’s last day in the shadow cabinet. While he had decided that he could not serve a leader who was not unequivocally committed to staying in the EU, nor did he wish to follow the example of colleagues who had staged instant departures.

“I’m not planning to serve in [Jeremy’s] shadow cabinet,” he told me when we first met. “But I didn’t want to resign immediately, because we’ve got parliamentary business for which I am responsible.” And so Umunna planned to remain in post to argue the opposition’s case on the trade union bill that came before the house 48 hours after Corbyn was elected.

“I thought that going off and leaving them in the lurch when they hadn’t even appointed anyone in my place would be wrong,” he told me on the eve of the debate. “I will have a discussion with him [Corbyn] about these things. I totally respect others who have resigned, but I have immediate business.” A few hours later, Umunna was gone. In a meeting described as “businesslike and collegiate”, his offer to stay was declined by the new leader. As Umunna was informed, his successor, Angela Eagle, was already in post.

Umunna had not expected any attempt to keep him. Corbyn’s team had not previously “offered anything specific, but they wanted to know my position,” and his mind was already made up. “I don’t intend to carry on in shadow cabinet,” he told me. Nonetheless, the speed with which Corbyn pre-empted a politician as skilful as Umunna reveals an unexpectedly ruthless streak.

If Umunna was startled at the speed of his departure, he was “not wholly surprised” by Corbyn’s vast mandate. “It became clear soon after the YouGov poll [which predicted the eventual result some weeks beforehand] that barring a miracle, he was going to win. I was expecting a resounding victory. Usually Labour supporters and members believe that you make your values real by getting into government, but that is not the desired outcome at present. That will change as a general election draws nearer, but the desire for now is the greatest protest against what is happening. I may not agree with that, but I understand it.”

In the short term, the departure of Chuka Umunna stands as an emblem of New Labour’s fall. A moderniser who struck many Blairites, and Tony Blair himself, as a future leader, he now faces a long spell on the backbenches. The role swap under which Labour’s mainstream voices have become the insurgents has not seemed an entirely easy process for Umunna.

The news, early in the contest, that he and Tristram Hunt, the former education spokesman, were setting up a group named Labour for the Common Good prompted media speculation on a “Resistance” movement to advance the modernisers’ agenda from within the ranks of the parliamentary party.

So angry was Lord Prescott that he accused Umunna of disloyalty. “I had a lively discussion with John – just put it like that … Somebody decided to leak [the group’s founding] to a newspaper and put a particular spin on it – which is part of the problem with the culture of the parliamentary party. It’s become quite factional. We need to get off that.”

Umunna, who finally left the shadow cabinet promising not to be “a thorn in the side” of the new leader cites the Tories’ “much more open approach” to incubating new thinking outside and within parliament as his model. “There used to be a rich tradition in the Labour party of different groups – Tribune and others. That had a negative side because it bred factionalism, but there was a positive side [too]. You actually had colleagues coming together and talking about policy.”

Having been portrayed (unfairly, he says) as veering from confrontation to emollience, Umunna favours qualified cooperation with Corbyn. “We have to accept the result … and try and make the thing work. I don’t think frankly the party would forgive us if we did anything but that. I think it’s very unlikely that [Corbyn] would junk many of the positions he’s campaigned on. But … while one may not serve in shadow cabinet, that does not mean one doesn’t support the leader and ensure the party gets elected again.”

Had his mentor, Tessa Jowell, been chosen for the mayoral nomination, Umunna was considering devoting much of his time to helping her. The unsuccessful campaign of Liz Kendall, whom he backed, and the defeat of Dame Tessa, signify the closing of routes to a politician for whom all avenues once seemed open.

Trained as a lawyer and billed as the British Obama, Umunna’s background was one of wealth on his mother’s side (his grandfather was Sir Helenus Milmo, a prosecutor at the Nuremberg Trials) and of poverty on his father’s. Bennett Umunna, a penniless migrant from Nigeria, subsequently became a successful businessman before dying in a car accident when his son was in his teens.

Umunna won his Streatham seat in 2010 and embarked on a parliamentary fast track. Briefly Ed Miliband’s PPS, he was made shadow business secretary in 2011 by the new leader he once referred to as his “mate.” Though he served loyally in the shadow cabinet, that warmth appeared to evaporate amid rumours that Umunna, like other senior colleagues, felt that they were not given sufficient leeway to air their plans.

Were his talents under-used? “I wouldn’t put it quite like that. Ed took responsibility for the defeat, and I don’t think that’s fair because it was a collective failure. All I would say is that there were difficult times.” Tellingly, he embarked on the general election campaign believing that Labour would fail. “I started the campaign thinking we would lose in probability and that there would be a renewal of the coalition. I didn’t expect a Conservative majority.”

The actual result, and Miliband’s immediate resignation, gave Umunna his chance to replace him as leader. Hailed as the favourite even before he had declared his candidacy, he withdrew early on, thus removing one obstacle in the Corbyn ascendancy. Does Umunna think that he could have stopped Corbyn and blame himself for failing to try?

“People have been very generous about what might have happened. But I don’t necessarily share the view that things might have turned out differently. So I don’t think that. I have never thought: Oh my goodness, I might have stopped this.”

Though his decision not to fight is now long past, it still seems curious that as shrewd a media operator as Umunna should have been blindsided by the press attention he cited as his reason for quitting the race. He admits now that he felt it incumbent on him to heed those urging him to stand. “I did feel a degree of duty to those urging me to put myself forward … I completely under-estimated the extent to which my girlfriend, my family, my loved ones would be [scrutinised].”

But he, I suggest, was the one who appeared to draw his girlfriend, the employment lawyer Alice Sullivan, into the public glare. “I’d appeared with her in public several times before. Until you’re actually in the eye of the storm it’s hard to know how you and people around you will think about it … I felt terrible about it. My family is incredibly important and precious. I’ve said that I could live without leading the Labour party, or [without] politics. For once I decided to put the rest of my life first.”

Untrue rumours about his sexuality or financial circumstances were, he says, merely part of the speculation to which he was subjected. “There was drugs! The other [story] doing the rounds was that I had a family member in Isis. Part of the relief of getting out [of the race] was the liberation from all that nonsense. I have nothing to hide. You don’t put yourself forward as party leader or a high profile member of shadow cabinet if you do have anything to hide.”

There was another trigger for his exit. “Part of the judgment was whether I was ready or whether I needed to spend more time thinking about how you renew social democracy. I’d always had doubts about whether I should put myself forward or whether I needed more time to think about this stuff. If this contest has exposed anything, it’s that we haven’t rebuilt the Labour project properly since 2007.”

Nor, in his view, did modernisers like him understand the trauma of the general election rout. “We underestimated the emotional upheaval precipitated by the result. Members who had been campaigning for Labour candidates were in deep grief. It was a massive shock to the system. Others were, and still are, very angry and bewildered.

“In order to make an argument for an alternative way forward, you have to build relationships. We’re not very good at doing that with people who are part of our party, never mind maintaining good relations with others.” Moderates, he says, relied on factual evidence and rational argument. “It wasn’t wrong, but we failed to engage with the emotion. We displayed a woeful lack of emotional intelligence in making our argument, and that includes myself.”

Too often, he says “it sounded like we were dismissing people who were critical of New Labour and saying they were mad … How on earth are you going to persuade people of a case if you don’t meet them where they are?”

The Blairite old guard, I suggest, don’t get that message. “Blair does.” Then why would he tell Corbyn sympathisers to get a heart transplant? As Umunna says, the former leader later offered a much more nuanced view. So he got it wrong first time? “People got it wrong.”

As for the question of how Labour modernisers could get it right, Umunna might be the first to admit that the ideas on offer seem amorphous compared to Corbyn’s clearcut prospectus. The future direction will “draw on New and Blue [Labour].” His party, Umunna believes, must emulate the Tories, who incubated ideas in a range of think tanks and in-house policy units before “grabbing the best of those new ideas.”

As he has previously said, he favours asking Arnie Graf, Ed Miliband’s sidelined community organiser, to mobilise those new members amenable to centrist policy. “The worst thing that could happen would be if the new joiners were a flash mob who came in and left after the result.” There seems little sign of that happening while Corbyn is still able to command the excitement that eludes Labour’s beleaguered moderates.

Where will the focus be for the radical thinking that Umunna’s own sub-group plans to undertake? “A lot of it will revolve round the conception of the state and what it can do in an era of globalisation. And secondly, how can we make supranational co-operation much more meaningful and effective.

“If we set ourselves up as an alternative [to Corbyn], I’m not sure that will win over the movement. We want to feed powerful ideas into the debate, and hopefully the leadership will run with them. We shouldn’t be seen as an opposition. I know this is going to be a difficult thing to pull off because inevitably the media will be looking to juxtapose one view against another, but there may be quite a lot of common ground. I agree with Jeremy Corbyn that we have to change the culture of the Labour party.”

But as Umunna warns, sooner or later Labour will have to confront the question of how it gets elected again. “I believe very strongly that there isn’t a choice to be made between your values and principles and being popular and plausible. The two are mutually dependent. That is why Keir Hardie and the party founders were clear that Labour was there to take government in order to deliver for the people. Now Jeremy has won, it will be good to have the debate about how Labour gets into office.”

How, with Corbyn at the helm, will it all end? “Whether we get back into office depends on the programme. I’m very clear that a Labour government is better than a Tory one.” It is conceivable, he believes, that Corbyn will lead the party into power in 2020. “Anything is possible as the last few months have shown. I wouldn’t want to make any predictions about the next general election.”

In the next few months, it will become clearer whether Labour is heading for victory or for the wilderness years that gave rise to civil unrest. Whichever scenario prevails, Chuka Umunna, the candidate who escaped the maelstrom, will be waiting for his second chance.