“Who is this? I don’t recognise a lot of these people. That’s Rachel Kushner. That’s Per Pedersen. Ismail Kadare. There’s a good quiz going on here.”

It is the end of my conversation with Neel Mukherjee. In the past hour, the 47-year-old has ranged effortlessly from his extraordinary new novel A State of Freedom to his experience of being the favourite to win 2014’s Man Booker Prize with The Lives of Others. He has pondered his Indian birthplace and his adopted homes (England and the United States). The joys of writing by hand have been weighed against the perils: “I have a panic attack at 3 o’clock in the morning. The house is going to burn down. I’m going to lose my only copy. Then there is a flurry of scanning.”

Finally, as we wind down, Mukherjee indulges in a game of spot-the-author. One wall of our meeting room at his London publisher is festooned with moody black-and-white photographs of writers. But who is who?

I identify William Faulkner and the British hypnotist-turned-self-help-guru, Paul McKenna. Mukherjee is more prolific. “Tom McCarthy. Jo Nesbo. That’s Helen Fielding.”

When he gets stuck, he admits cheerfully: “I am more interested in the ones I can’t identify.”

It is tempting to adapt this sentiment to describe Mukherjee’s concerns as a writer – his fascination with the unknown, his empathy for the underdog. Set in modern-day India, A State of Freedom narrates the lives of a broad cast. The most vivid stories follow the most heroically vulnerable: young girls dragged from the families and sent to work; others attacked by predatory men; a wife and mother with terminal cancer; a cook whose unceasing work for wealthy employers supports her ambitious nephew. “I wanted to say something about the world. I wanted to say something about inequality. It interests me a lot. The Indian stripe of it is particularly egregious.”

Mukherjee’s experience of India is present too, reflected in characters like the scientist-father in the opening section, and more obviously the younger writer in part two, who return to the country from lives abroad. “Do I find myself a tourist in the country of my birth? Yes and no,” Mukherjee says with typical open-mindedness.

Born in Kolkata, he left India at the age of 22 to study in England. He was an undergraduate at the University of Oxford and completed a PhD at Cambridge. At the time of his departure he was “fully-formed. Every wiring was in place. I can go back [home] and after a while slip into it.”

But he adds, each return involves a period of adjustment, and not just to polish up his “rusty Hindi” (his mother tongue is Bengali). “Every year, my learning curve is very steep. The country has moved on but [for me] it has remained frozen at the point when I went back last time. You only notice change in something when you are outside it. Being away from India lets me see this process of change in a very tangible way.”

The Mukherjee I meet has also just returned – this time to London from America, where he spends a semester teaching creative writing at Princeton University. His evident enjoyment is undercut with scepticism. “I don’t believe in teaching creative writing. I don’t think writing can be taught. They can be taught to be careful readers.”

Mukherjee’s misgivings about the academic discipline date to the year he was a creative writing student on a prestigious course at the University of East Anglia. Its alumni include Ian McEwan, Kazuo Ishiguro, Mohsin Hamid.

“It helped me in the negative way. You develop a filter with people giving comments on your work. You learn what not to take into your work.”

While he made friends with some similarly successful novelists (Paul Murray, author of the glorious Skippy Dies), Mukherjee speaks with unapologetic derision about his teachers, including former poet laureate Andrew Motion: “The most insincere man I have ever met.”

Similar ambivalence accompanies his memories of the Man Booker Prize. His last novel, The Lives of Others, was the favourite to win in 2014 until Richard Flanagan’s The Narrow Road to the Deep North.

"I am grateful for the attention. Things have changed for me,” Mukherjee says of the exposure the prize brought his work. Another upside was the friendships he formed with fellow shortlisters such as Flanagan, Karen Joy Fowler, Howard Jacobson and Joshua Ferris.

Mukherjee is distinctly less enthusiastic when describing the hoopla of the competition’s finale. “I feel they should be like the Pulitzer. They should cut out all the drama in the lead-up. That is very costly as a writer. I do not know a single writer who enjoys it. But they don’t have the power to say, ‘We are not going to go to the dinner, we are not going to dress up and be your performing monkey.’ I feel that whole thing playing out in the public eye is really terrible.”

If there is any justice, A State of Freedom will also propel its creator into Man Booker contention. Clearly more experimental than The Lives of Others, it employs that most modish of modern forms – the interlinked short story.

“There are people who have written very cohesive books with fractured narratives. David Mitchell comes to mind. Cloud Atlas, I think, is so wonderful in what he does with structure and tectonics. You will see everything join up in very unexpected ways. I wanted to write in a very non-David Mitchell kind of way. I thought a properly realistic novel would mean that things don’t join up. One ramification of the word ‘freedom’ is chaos. Things don’t cohere and they spin apart.”

Another ramification of “freedom” is to imply a conversation (“If that is not too hubristic”) with VS Naipaul’s “great masterpiece” In a Free State. Mukherjee’s debt to Naipaul is a question of both form and content: the state of India; India as a state.

“It is a wonderfully formally audacious book. He has three novellas bookended by a prologue and epilogue, and not a single of those narratives join in any kind of obvious way, and yet it is a novel. I found myself asking, Why is it a novel? It returns very interesting answers.”

Also divided into five sections, A State of Freedom eschews linear narrative in favour of startling echoes, recurring motifs and characters who appear in different guises or at different stages in their lives. Milly, a housemaid in section two, recalls her childhood in section four; this same chapter offers a tantalising, and possibly illusory glimpse of the scientist-father in the opening story. Perhaps the most unsettling bond unites the first and last tales: the wealthy young Indian-American; the poor, migrant labourer building a skyscraper. This haunting was deliberate and is loaded with political overtones.

“I was thinking carefully about what a ghost story does. A ghost exists always because something unhappy in the past has not been settled. I thought the perhaps the ghost story could be opened up to think about painful histories and unsettled history.”

It is hard to miss the reference to India, a nation whose forward momentum cannot break the chains of its tumultuous and unstable past, before, during and after the British Empire.

“I meant the title not just to echo Naipaul, but also (Jawaharlal) Nehru’s tryst with destiny speech at independence in 1947.”

The resulting historical and social turbulence is brilliantly evoked in Mukherjee’s final section, which teeters like its vertiginous narrator on the brink of destruction. He explains his intentions by citing the response of his friend and fellow novelist, Anjali Joseph, whose recent novel The Living is an inspiration.

“She felt that instead of tying everything up, A State of Freedom ends with fraying and chaos. That is a realistically Indian novel I feel. The whole Indian state that held together, miraculously, for the last 70 years since independence is fracturing.”

Mukherjee is not suggesting that India will disintegrate as Yugoslavia did at the end of the last century. “The state machinery of India is too strong, too militarised and too militant to let that sort of thing happen.”

Instead, he depicts a nation characterised by disorder, instability and contradiction. “It was the economist Joan Robinson who said about India, ‘Everything you say about it, the opposite is also exactly true’,” Mukherjee says with a laugh. “On a good day, which are very few and far between, you can see India as the most pluralistic country you can think of. There are so many disparate of people, languages, cultures, beliefs – all held together. I think it is an enormous success in some ways. On other days, I feel it is a country that is so marred by divisions, caste, religion and wealth. Gender, too, I think is one of India’s dirtiest, dirtiest things.”

And as an unflinching portrait of one man’s inhuman treatment of a bear suggests, India’s treatment of animals is another. “India’s attitudes to animals are just so terrible and then you have the cow fetishising that is going on now.”

More than once during our conversation Mukherjee suggests that A State of Freedom might be the last book he writes about India. He accepts that this change of direction is not without risk.

“In the United States, if you are not white Caucasian, you forever have to keep writing the immigrant novel. When try to break out of it, you are punished.”

His hero in this regard is his colleague and friend, Jhumpa Lahiri. “She hasn’t written an immigrant novel for a while now. She started her career writing about Bengali immigrants which made her enormously successful and she wrote beautifully about them. She has decided she is no longer going to do that. She is going to do her own thing. She is going to write in Italian from now on. I don’t know whether she would agree, but this is her rebellion against the strait jacket she has been put into.”

Mukherjee has been rebelling in a different way over the past 12 months. Since Brexit, he has refused to read the news. “I watched the entire Leveson enquiry live at my computer. I would have done that for (the sacking and testimony to the US senate of FBI director James) Comey as well, but I didn’t. I thought I am just so diminished by what has happened over the last year that I have to protect myself.”

I suspect that when Mukherjee does lift his blackout, he won’t feel much more cheerful about the state of the planet. As he says towards the end, “I think my interaction with the world is a little bit skinless. But perhaps most people are like that. Who doesn’t get angry or eaten up by the world? The world is a terrible place. It just destroys us. It crushes us ultimately.”

This bleakness is not just a matter of fact, but central to Mukherjee’s character. Not that he doesn’t try to fight his pessimism. He recalls the first meeting he had with his editors about A State of Freedom. “One of them said, ‘Neel, this book is very bleak’. I said, ‘Well, yes, but it’s less bleak than my previous books.’ I was mindful of putting in some lives that have a happier trajectory."

Just before we finish, I ask how Mukherjee hopes to find his existential silver lining. “AS Byatt once said this to me: the more mature writers find it in themselves to give happiness to their characters. It is a mark of maturity. Bleak and tragic books are for the young. The older you get the more likely you are to relax, forgive your characters and find happier destinies for them.”