When Chuka Umunna announced he was running for the Labour leadership on 12 May last year, it looked as if the party might have found its saviour. It was only four days since his friend Ed Miliband had stood down, after Labour’s mauling in the general election, and the bookies installed Umunna as favourite. Liz Kendall was a little-known outsider, Andy Burnham and Yvette Cooper seemed stale, and Jeremy Corbyn wasn’t even a twinkle in the electorate’s eye.

Sure, there was nothing radical about Umunna’s politics, but that appeared to be just what the party needed. The early post-election consensus was that Miliband had steered the party too far to the left; the founding fathers of New Labour (Blair, Mandelson) had announced that their legacy had been betrayed. Umunna looked like a reassuring return to traditional, election-winning, Blairite values. He was mixed-race, youthful, ludicrously handsome, smart, pro-business, a lawyer – he’d even been a DJ, for God’s sake. The inevitable Obama comparisons were made. Chuka Umunna, yes he can.

Only he couldn’t. Three days later, Umunna announced he was withdrawing his candidacy. He had not been prepared for the level of press intrusion, he said, and his family was too important to him. Social media and the Westminster rumour mill went into overdrive. Everybody waited for the big reveal in that Sunday’s papers. What had Umunna done that he was so ashamed of – was it drugs, sex, serial murder?

But the weekend papers passed without a revelation, and the Umunna story drifted away as an unexpected political phenomenon emerged: the rise of the new left and Corbynism. Umunna disappeared from frontline politics. He continued his work as MP for Streatham in south London, where he had recently quadrupled his majority; he set up a cross-party parliamentary group on social integration. Always a private man, he retreated further into his privacy. Last October a small notice in the Times announced that Chuka Umunna was engaged to Alice Sullivan, a fellow lawyer with whom he had been photographed walking hand in hand to The Andrew Marr Show two days before he announced his candidacy.

But now Umunna is back, a leading player in Labour’s campaign to persuade us to stay in the EU. It’s less than a month until the referendum when we first meet, and Umunna is anxious: he thinks the vote could go either way. The problem, he says, is that traditional Labour supporters are not clear on the party’s position, and he fears a low turnout from the innies. We meet in the Commons, where Umunna is chairing the all-party parliamentary group on social integration. He is tall (6ft) and elegantly suited, with a long tie that slides deep into his lap. As a chair, he is warm, with an easy manner. The group hears disturbing evidence from Professor Ted Cantle, who carried out a review into community cohesion after the Oldham riot 15 years ago, and who says that lessons have still not been learned; communities across the country are living entirely separate or “parallel” lives.

The fact is, I didn’t really run, did I? I was in it for three days!

For Umunna, aged 37, social integration and the EU are two sides of the same coin. As a young boy growing up in Streatham, he tells me, there may have been more overt racism, but communities were not so isolated; the immigration debate has become “toxic” in Britain. “Labour finds this issue hard to deal with, because it worries about racism. I don’t think the answer is to pander to a Ukip agenda and say, well, let’s just chuck all these people out of your area. But equally, you’re ignoring how these people feel if you try and pretend they don’t feel their area is changing. Unless we reconnect and have a Labour answer to these fundamental issues, which all emerge from globalisation, we ain’t ever going to get back into power again.”

Umunna believes Labour has lost touch with many areas of society: voters in the old industrial heartlands, black and minority ethnic communities, “where we are shedding support at an alarming rate”, and the aspirational lower middle class. “These three groups have left our coalition, and this is just the start of it. The challenge for us is not to go, in a clinical way, ‘Oh, what appeals to these three groups?’ and we’ll just say it. You need an overarching story, an idea of where Britain can go. My worry is that after the European referendum, regardless of whether we remain or stay, there is going to be a huge concern and paranoia that Ukip will be grabbing support from the Labour party.”

It’s a bleak prediction. Is this a scary time for Labour? “Yes, it is.”

Sharing a platform with Ed Miliband in April 2015. Photograph: Carl Court/Getty Images

A day later, we meet at Umunna’s office, which looks over Westminster. It’s a beautiful morning, and Big Ben and the Commons are bathing in a golden sheen. He looks out of the window. “Every day I walk in here and see that view and think, bloody hell, am I really here? When you start thinking you have a right to be in this place, that’s when you lose it.”

He has been thinking more about identity and integration. As a boy, Umunna was always aware that his family stood out. His mother, Patricia, a lawyer, is white and half-Irish; his father, Bennett, was Nigerian. But that was not the main reason people would stare: she was 6ft 1in and her husband barely 5ft. “You can imagine us walking down the road, the four of us. We were unusual.” His sister, Chinwe, is 35 and a teacher in Denmark.

Bennett Umunna was penniless when he moved from Nigeria to London in 1964, but determined to make his way. Patricia was upper-middle class; her father, Sir Helenus Milmo QC, had been a prosecutor at the Nuremberg trials. The couple met at a party in the early 1970s, by which time Umunna’s father had built up a successful import-export business (largely mobile clinics and other portable buildings). The family enjoyed a good standard of living. Chuka went to private school, and Bennett drove a sports car.

Back in the 1980s, the police were suspicious of black men who drove sports cars. One day they followed him home, accused him of being a drug dealer and beat him to a pulp. “They got him out of the car, beat the living hell out of him, then took him to the police station. Every time he opened his mouth, he got another hook. We took the police to court and not one officer was disciplined.”

Was he aware of this at the time? “No. I was seven or eight, and my mum never told me until much later.”

Bennett, like his son, did not like to be pigeonholed. He was an entrepreneur who worshipped Harold Wilson, became a director of Crystal Palace football club, and split his time between England and Nigeria. He was made chief of his village, and stood unsuccessfully for political office in Nigeria on an anti-corruption ticket. Soon after, he was killed in a car crash there, aged 51. His friend Ron Noades, the former chairman of Crystal Palace, said he believed Bennett had been assassinated. Was his death investigated? Umunna looks away. “I’d rather not go into that, to be honest. Partly because I haven’t had the conversation with my mother.”

Was he close to his father? “Yes.” If he closes his eyes, what comes to mind? “A larger than life personality, even though he was pint-sized – an infectious personality and an utter determination to succeed. He always sought to turn any negative into a positive. Maybe that’s where I get my optimism from.”

At 13, he found himself the man of the house. His mother returned to university, to update her law degree. Chuka had not been a particularly conscientious boy, but now he focused. He worked on a local market stall at the weekend, and at Crystal Palace in the away box office on match days. While his friends were doing daft teenagey stuff, he was busy being responsible. “Losing a parent at such a young age changes the whole way you view the world. I had to grow up very quickly.” How? “Well, on a basic level, you see your mother is distressed and we’re all going through a difficult time, and you can’t indulge the kind of moments you have as a teenager, because you don’t want to create more stress.”

Did he lose his optimism? “At times, maybe.” He pauses. “Yes, I did,” he says eventually. “It wouldn’t be very human if you didn’t, would it? I thought I had been robbed. But I’m not one to dwell and get angry and fester. My father had hammered into me, you want to look after your family, you want a nice house, and you want to be able to enjoy yourself, and you have to work very hard for those things. Don’t think the world is going to come to you.”

I ask if his father’s death politicised him. No, he says, what politicised him most was his first visit to Nigeria, aged seven. “I saw the grinding destitution of poverty that you often saw on TV. And coming face to face with that, particularly in the village where my dad was from, I could not get my head round why it is that I’ve been born into this situation, in this country, and I get a nice meal every day, and there are people from the same village, who are part of my family, who don’t have that at all. As a kid, I thought: ‘That is wrong.’”

After studying for a law degree at Manchester University and an MA at Nottingham Trent University, Umunna started work as a lawyer. In a way, he says, this also politicised him. “When I qualified, I thought I wouldn’t be able to do employment law, because I had this black-and-white view: if you’re acting for the employers, you’ll be looking to screw over the employees. But under a Labour government, the whole employment law architecture was much fairer than under the Tories. This is why I get so furious when people say it makes no difference whether there is a Labour or Tory government. That’s from people who, frankly, don’t feel the effects.”

In his spare time, he DJ’d, specialising in house and garage, with a residency in a south London bar (at school he had played cello and was a chorister at Southwark Cathedral; his voice can be heard on the theme tune to Mr Bean). At one point he considered jacking in the law to become a full-time DJ. Was he making money? “No, that was part of the problem. And I wasn’t sure I fancied working long nights when I was into my 40s and 50s.”

Umunna won a seat as an MP at the first time of asking, in 2010, and quickly progressed to shadow business secretary in 2011. Back then, he worked with the left-of-centre pressure group Compass, and in 2010 called for the scrapping of Trident (something he has since changed his position on). He was a politician with promisingly wide appeal, as happy talking to younger voters about music as he was addressing the business community about reform in the City. He was regarded as highly competent, a slick operator.

Too slick for some: the Commons wasn’t used to a man who moisturised. In 2013, the Mail On Sunday reported that Umunna belonged to a “shady” City men’s club where bottles of brandy went for £4,000 a pop, that he hung out with celebrities, and that he would happily pay £1,200 for a suit.

Today he is wearing a natty pinstripe. Was it £1,200? “No, that’s bollocks.” What would he spend on a suit? “Certainly not £1,200. I’ve had so much bullshit about this. We got a call in the lead-in to the general election asking if I was wearing a Patek Philippe watch, which is, like, £15,000. I was wearing a £120 watch from Gap. Where would I get 15 grand for a watch?” As for the “shady” club, Umunna says he did attend a business function at its restaurant, but is not a member. “They took that story down because it was rubbish.” He threatened the Mail on Sunday with legal action: why? “Because it was seeking to paint a picture of me that wasn’t true, and I thought, this is going to follow me around for ever. Look, I like my music, and I don’t have a conventional background for a politician, but I’m pretty conventional in many senses.”

Ed Miliband did not get all this personal stuff during his leadership campaign. I got it in the first week

There were other stories that did not show Umunna in a good light. That same year, the Daily Mail published an old post he had made in 2006 on an elite online network called ASmallWorld. Under the pseudonym Harrison (his middle name), Umunna wrote: “Is it just me or is there a serious lack of cool places to go in central London at the weekends? Most of the West End haunts seem to be full of trash and C-list wannabes.”

He grimaces when I remind him of it. “Gosh, that was so embarrassing. It was a closed, private exchange, relating to where to go out clubbing, basically. And in a rather stupid fashion I was drawing a distinction between somewhere people want to go and have a good time, and somewhere people want to get lairy and end up in a fight. Look, I’m not one of those people who had some grand plan from the age of 15 to become prime minister and ran my life accordingly. We all say stupid and foolish things. I was lucky, because I didn’t get on to social media until my 20s, whereas the foolishness of today’s kids will follow them around for ever. You’ll end up with more career politicians if you go for safe people who have never had a life and never made a faux pas.”

But it wasn’t the last embarrassing story. A week later, it was revealed that the Wikipedia entry comparing Umunna with Obama had been created in 2008 (when he stood for selection) using a computer at Umunna’s law firm. Today, he says the comparison is reductive and daft (“Obama is a two-term president!”), but insists that he was not personally responsible for the entry. “I suspect somebody in the campaign or a colleague did that. Lots of people did make that comparison, but it’s just because you’re a black politician. In the end, you want to be defined by who you are.”

He’s got to leave for a meeting. On the way out of his office, I look at the pictures on his mantelpiece – mum, dad, sister, maternal grandfather, and a woman holding a baby on her knee. Who’s she, I ask. “That’s Alice with her nephew,” he says. “It’s funny, she always says to me, you are so different from the way you’re portrayed in the papers.”

‘Looking at the other people who were in line to do it, I thought I could do a good job.’ Photograph: Shamil Tanna/The Guardian

It’s little more than three weeks until the referendum and we’re on a train to Manchester. Another day, another look. This time, it’s casual Chuka: skinny jeans with micro turn-ups, blue suede shoes, pristine white shirt, jacket. You couldn’t get a more extreme contrast with Jeremy Corbyn.

A few months before Umunna announced he was standing, he was asked if he would like to lead the party. He laughed, and said no, he was still a baby in political terms. So what changed? Well, first of all, he says, the vacancy came up. “Then a lot of people asking me to stand – people in the street, not just in my constituency. Secondly, I thought I had something different to offer. And thirdly, looking at the other people who were in line to do it, I thought I could do a good job.”

What made him different? “Three things.” Umunna likes his lists of three. “One, I was quite young. Obviously my background was different. And third, I was not tribal. I had been involved with Compass, but I also had an appreciation of the modernising wing of the party. I thought my candidacy could perhaps fuse the two.”

So what went wrong?

“Well, the fact is, I didn’t really run, did I? I was in it for three days!” He laughs.

Take me through those three days to help me understand, I say.

He starts two days earlier, with the Sunday he appeared on The Andrew Marr Show. “I stupidly went with Alice. And by the way, I had appeared in public with her before. We’d been to a number of public events together – we’d walked down the red carpet at the Baftas.” Why was it such a mistake? “All the cameras were there. I didn’t foresee there would be so much interest in her.” How did she react? “It was quite a shock. Two to three hours after that, photographers ended up at her parents’ house.” Sullivan’s father told the Telegraph: “I haven’t even met him. This is very early on.”

On the Marr show, Umunna appeared alongside Peter Mandelson to talk about the election defeat. It couldn’t help but look as if Mandelson was blessing the heir apparent. Umunna argued that Labour was wrong to run a financial deficit before the financial crisis and, in an echo of Mandelson’s 1998 declaration that “we are intensely relaxed about people getting filthy rich”, said that nobody is “too rich to be in our party”.

Was that not a terrible misreading of the mood? Umunna delivers a long answer, telling me that labels such as Blairite and Brownite are irrelevant, before admitting that, yes, it was a mistake. “I would have done things differently. I don’t think you want to be defined by what has gone before, because people are in the future business and they’re looking at what comes next. I don’t want to centre it around an individual, whether it’s Peter or whoever, but the fact is New Labour was appropriate for its time and now we need something different.”

How would he do it differently? “We should have started with our values rather than just with ‘What do we need to do to win?’ And what flowed from that was a framing of the debate throughout that summer, which was either you’re about Labour values or you’re about power, which is nonsensical, because the two are not mutually exclusive. Actually, they are mutually dependent.” (After he dropped out of the race, Umunna backed Liz Kendall, the candidate farthest to the right.)

On the Monday after the Marr show, more hacks arrived at Sullivan’s parents’ home – and Umunna still hadn’t announced he was standing. That came on the Tuesday, in the form of a shambolic video shot on the streets of Swindon in the middle of a gale: Umunna, normally so self-possessed, appeared nervous and as if he was about to be blown out of shot any second. “It was a bit of a rushed job, to be honest.” You looked out of your depth, I say. “OK,” he says. “I wouldn’t say I felt out of my depth, but it reinforced my sense that if you don’t have a campaign ready to go, that is a very big problem.”

By Wednesday, the press attention had turned to his family. “My mother was coming back from work, it was dark, 10.30pm, and there’s a reporter waiting for her, and at that point I thought, this isn’t fair. Then they rang my sister up in Denmark.”

What I don’t understand, I say, is how you couldn’t have been prepared for this – after all, you had a ringside seat at the general election as the press eviscerated Ed Miliband’s family. Umunna says he was naive: he didn’t expect it would happen so quickly. “Ed did not get all that personal stuff during his leadership campaign. I got it in the first week. Actually, one of the first people to drop me a text and say, ‘I totally understand what you’ve gone through’, was Ed.”

When Umunna withdrew his candidacy, there was speculation galore, the most persistent being that he is gay and wants to hide it. Is it true? “No, it’s bollocks. The most ludicrous one was that I had a family member in Nigeria who is a member of Boko Haram. And I said, seriously?”

Was he surprised by the rumours about his sexuality? “No, I wasn’t, because I don’t run my social life or private life around Westminster. I had one relationship in politics [he dated Labour MP Luciana Berger], but my closest friends are not in politics.” And he is not, as has also been speculated, bisexual? “No. I’m boring, man. Honestly, I’m pretty conventional.”

As far back as he can remember, people have tried to pigeonhole him, just as they did his father. When he was young, everybody assumed he lived on an estate. When he went into politics, his fellow MPs assumed he was out clubbing every night. And yes, he is urban and slick and a bit cool, but at the same time he is conservative and Christian and serious-minded and a bit uncool.

We get off the train, and I tell Umunna that my favourite rumour is that he is currently in a relationship with his friend, the pop star Tinie Tempah. He bursts out laughing. “Hahahaha! Hehehehe! Are you for real?”

As we reach the station exit, a man runs up to us and introduces himself as Michael Sharman, a businessman from Peckham in south London.

“Hi, Chuka, how are you?”

“I’m all right, man. How are you?”

“Please stand again, save the party. See it through this time.”

Why does Sharman believe Umunna would be a strong leader? “He has the personality, the media persona and also the experience, and the party needs to broaden its appeal. At the moment it seems far too narrow. We are talking to ourselves and we’re never going to win elections that way. And if anyone thinks we lurch to the left, that that’s what the country’s looking for, they’re fatally mistaken.”

A car driven by a party activist is waiting for us. Umunna giggles as he gets into the back. I ask what’s so funny. “Did they really say that about me and Tinie?” He shakes his head. “I wouldn’t give a shit if I was gay. It’s just that I’m not.”

There were other rumours about why he stood down. First, that he had accepted £2,500 from a company that helped non-doms. “Ah, that one! The company denies this, by the way. They put out a statement at the time.” Another theory was that he quit because he didn’t think Labour could win in 2020 and he wanted to keep his powder dry. Again, he shakes his head. “Well, I do believe we can win in 2020.” Did he at the time? “Yes. We have to change, but absolutely.”

Umunna’s father, Bennett. Photograph: Jim Bennett

In Moss Side, Umunna meets a first-generation African-Caribbean immigrant who wants to leave the EU because it is too costly. He says we are haemorrhaging money that could be used to pay for the operation his wife needs.

“We spend around £9-10bn paying into the EU,” Umunna says. “What do you think the government spends in total?”

“To be honest, I don’t know,” the man replies.

“OK. £772 bn, so the money we spend paying into the EU is less than 2% of what the government spends every year. So here in the north-west you’ve got 350,000 jobs linked to our membership of the EU. And do you go and buy a pint of milk at the newsagent? Well, 28% of the stuff in your local newsagent will have come from the EU in some way, and you can buy all of that without a tariff. If we come out, there are going to be tariffs. Have I made you think, man? Do I see a conversion?”

“Yes, I was thinking of the money going out.”

“Well, think of the lower prices, the jobs, Britain standing on the world stage, all the advantages.”

“Maybe you have converted me,” the man says. Maybe he just wants to get back home.

We head off to the university, where Umunna faces a tougher crowd. The students who challenge him are on the left, and disappointed that Labour is not demanding radical change from the EU or arguing for Brexit. He continues to plough his lonely, resolutely centrist furrow – human rights, employment rights, you can’t simply blame the banks for Greece’s austerity programme.

“Yes, I get that,” one student says. “But what gets me is the neo-liberalist structuring of the EU and that there is no challenge to it. How do you make the EU progressive?”

“Define new-liberalism,” Umunna demands.

“Deregulation within the economic system, austerity,” the student fires back.

“Privatisation,” shouts another.

“Profit-making from the NHS,” calls a man at the back.

Despite Umunna’s eloquence, he has not won them over. Too many of his answers start with, “I have some sympathy with what you say, but...”; he comes across as just another politician.

But he seems energised when we get out. “The guy who was railing against profit-making, I used to be that leftwing kid. I should have said, well, actually there are a lot of people who believe in markets who are not neo-liberals. The Keynesian economic model accepts markets, accepts profit, and nobody would say Keynes is a neo-liberal. That would have been a better answer.”

With fiancee Alice Sullivan, ‘the person who makes me more happy than anything in the world’. Photograph: Courtesy of Chuka Umunna

We meet later that evening on the train back to London. Both of us are exhausted, and we get a beer from the buffet car. We talk about how deeply split Labour is and the battles ahead. Where did Umunna rate in the leaked league table of Corbyn loyalists? “Oh, I was down as hostile.” Is that fair? “No, I don’t think of myself as hostile.”

In fact, he says, his relationship with Corbyn is positively cordial these days. “One of the reasons I was so pleased about his speech [in support of remaining in the EU] was because his position had changed. That’s a big, courageous thing to do. He’s actually a really nice guy.”

He would also like to set the record straight on the manner in which he left the shadow cabinet. “John McDonnell made several comments giving the impression that I had simply walked out. But Jeremy and I agreed it would be a good idea for me to stand down because we didn’t want a running commentary on my position and his – and at that point his position was different on the EU.”

Would he serve in a shadow cabinet now? “Well, I think it’s highly, highly unlikely that I will ever be asked.” Why? “Because, not so much Jeremy, but I think people around him can be quite divisive.”

A woman passes us, and asks if she can have a word with Chuka. Her name is Jeanette Kwakye and she is a former sprinter who ran for Britain in the 2008 Olympics. We all end up chatting away, and Umunna, who has never met Kwakye before, says she has to be in the article.

OK, I say, ask him a question. She wants to know why he stood down.

“I think the press were out of order,” she says, after hearing him out. Was she expecting a huge expose? “Yes! I thought, ‘Oh, God, what’s going to come out?’” She stops. “It made me feel, maybe now is not the right time.” What does she mean? “I felt this country’s not ready for somebody like Chuka. He’s too… dynamic.” But even he says he’s pretty dull, I say; it’s just that he’s mixed-race, knows how to make a mix tape and wears a decent trouser. “It’s enough,” Kwakye says. “We’re not ready. That’s where we are.”

I ask Umunna if he would still like to lead Labour. “I’d certainly like to play a big role in it.” Look me in the eye, I say. “We’ve got a leader and he’s got a big mandate and he deserves a fair crack of the whip. But if a vacancy came up, I certainly wouldn’t rule it out.” And if there was a vacancy before 2020? “Yes,” he says instantly. Why would he cope better this time? “I’d be ready. I wasn’t prepared to run a campaign for the Labour leadership – I was preparing to run the business department for Ed Miliband.”

The problem is that those three days could well have cost him the confidence of the country. People might just think you’re too flaky, I say. “Look, I ended up meeting the person who makes me more happy than anything in the world,” he says, “and I was not prepared to sacrifice that on the altar of politics. It’s as simple as that. I know there will always be speculation as to why I left the field. And if people don’t want to believe the story I’ve given, that’s fine.” Now there is fire in his eyes. “But if putting your family first, and putting the person you love first, is a problem for the British public, so be it. I can live with that.”