At the age of 72 Howard Jacobson is clinging on to his natural pessimism in the face of persistent provocation. It may be difficult, but Jacobson is in for the long haul and will not be cheered up much. A "happy domestic situation" with his third wife, Jenny de Yong, coupled with growing acclaim for his work, including a Man Booker prize four years ago for The Finkler Question and another novel newly shortlisted, are doing their best to derail him. The most the writer will allow is that fresh critical esteem has "taken the edge off my anger".

With the post-apocalyptic J, his 13th novel, named one of the six books on the Man Booker shortlist last week, Jacobson says he is grateful that a work he regards as his most confident yet has been recognised, and for the sales boost that will follow. But he points out its subject is determinedly bleak not for literary effect, but because he really believes in its dark message. It is the duty of novelists to take a gloomy perspective, he argues. "I have never met an intelligent optimist. That is not to say I think pessimism makes you intelligent, but I have always felt like an Old Testament Jeremiah or Cassandra from ancient Greece. I want to run down the streets warning people."

J tells the story of a nightmare future world, hobbled by the suppressed memory of a mass pogrom, only ever referred to as "the thing that happened, if it happened".

Jacobson's inclusion on the prize list this summer, putting him in the rare position of a potential two-time winner, has been applauded even by those reviewers who have quibbled in the past about his clever kind of word-wrangling. Last month the poet John Burnside rated J as worthy of being "seen as the dystopian British novel of its times" because it "so artfully mirrors the main features of our current 'lifestyle', from the endless production of formulaic pop culture and the glorification of infantile consumption, to the avoidance of difficulty and a systemic contempt for privacy".

While The Finkler Question was remorselessly funny, the humour in J is buried in remorse for humanity. The mood is never light. "I held myself back from that. There are other emotions that readers want to experience. The story must be allowed to resound with desolateness," he says.

At the same time Jacobson generally distrusts blithe distinctions between comic literature and serious novels. "I would go so far as to say if there isn't a deep vein of comedy in it, it isn't a good book. A joke is an act of intellectual comprehension, so I feel I have always been serious. It is just that for this book I have invented another sort of universe."

Those who tell Jacobson to calm down and stop exaggerating the parlous state of the world get short shrift: "It is a nonsense to me when people come along and tell me not to be pessimistic; or that culture has always been going to the bad. Well, yes, it has, and it is an author's job to point it out."

He finds it hard to talk about the meaning of J at this early stage. "I can say it is about paying the price for winning an argument. It is about the aftermath of something terrible, but something that is still happening and that will get worse." Although J may appear to respond to a recent resurgence of antisemitism, Jacobson had the original idea four years ago, before the current crisis in Gaza. He imagined a sinister scene on a cliff. "A man is standing up there, full of anxiety, with the sense he is being observed and is not safe."

The new book is about the annihilation of any group, any "other", Jacobson says. "The Jews happen to be the group that I know about, so it is informed by antisemitism, but the point is that if you get rid of 'the other' you then have an absence; an absence of irony, an absence of disputatiousness. No argument should ever win that completely."

Jacobson extends this rule even to Holocaust deniers. He remembers a sense of flatness, of anti-climax, when the theories of the historian David Irving were squashed in court in 2000. "I didn't care about him, but when one side defeats the other totally, there is a terrible price to pay."

This is why he says J could not be an overtly funny book. "It was not possible for me to have written this in my normal ironic, argumentative voice because these are the very qualities that are not available in that world I am describing." Jacobson cannot get over, he says, the fact there is antisemitism in Europe again, a mere "blink" after the Holocaust. Yet J is not a book about contemporary politics. It addresses a bigger, timeless problem and so falls in line with the literary tradition of Aldous Huxley's Brave New World and George Orwell. "I never really liked Nineteen Eighty Four, in fact," says Jacobson. "I admire Orwell enormously, but he is a much better essayist. J is a novel. A story about what it is like for people after a terrible event. And it is a love story, because I feel a novel is inevitably a love story."

He did not set out to write a futuristic genre novel, but concentrated instead on the texture of the lives he described. It was fun though, he concedes, working out how people would talk and how they might differ from now. "In a way all my books are apocalyptic. I am a catastrophist, perhaps like all Jews."

The hero of J, Kevern Cohen, "shares neuroses and anxieties" with Jacobson but is not quite the proxy, the author feels, that the lecturer Sefton Goldberg was in Coming From Behind, the campus novel that made his name in 1983.

The absence of Jacobson's trademark satirical tone is not the only startling omission in J. The word Jew is also never used and the author smiles when he recalls reassuring his 91-year-old mother ("still in good nick") that he had written a story at last that was not about being Jewish. In fact the subject is obliquely present throughout, not least because all the characters have adopted Jewish surnames. It is a dry reference to the way emigres once dropped their Jewish names to lose their identity in a new land. "It is a kind of grand revenge," says Jacobson.

The nature of the "thing that happened" is also never made plain in J, but Jacobson intends it as a cautionary scare. "People say it could never happen again. But it did happen and even on this island Jews were once kicked out. We have to have people to hate, to create a menace.

"It is not pleasant, but that is how we are." His novel offers no answer, it simply counsels against any "final solutions".

So what will this serious, pitch-dark book do for the freshly-minted literary term "Jacobsonian"? It stood for prose that was quick-footed and full of word play: the style his novels Kalooki Nights and Zoo Time made familiar.

Well, the author says he has not changed. "Sometimes you have to stop yourself being funny. It interferes with emotions. The real danger of the comic gift is that you don't have to try hard, you just have a facility. I will always know how to end a chapter with a joke, but sometimes I should not."

The process of writing J was not cathartic, however, he says. "J is not going to help me when I am worrying about things. Nothing is out of my system sadly, although I hope it might be cathartic for readers."

After decades categorised as an "under-rated" British humorist, Jacobson is now established as a man of letters, something he always wanted. All the same, he is still driven by a fear of obscurity that he feels masks a fear of death. "This book is shot through with that fear, the fear that drives all writers."

But prize nominations help. "I would like to say I am insouciant; that I know prizes are for schooldays, but after the relief of winning for the first time I must admit that, if there is a list, then I think 'Why shouldn't I be on it?' I want people to read me. Ideally, I would like everyone in the world to read and love my novels. In fact, I can't believe that everyone in the world doesn't love them. What is there not to love?"