On a cold afternoon this winter I sat before a glass wall at the Museum of Jewish Heritage in New York City, fielding questions about Jewish dystopian literature. Outside was New York Harbour and the audience seemed distracted by the passing boats. My fellow panellist was Joshua Cohen, author of Witz, a novel about the last Jew on earth. My novel The Flame Alphabet concerns a poisonous language spoken by children and is set in a world of failed science where Jewish mysticism might offer the only clue to the language toxicity. Cohen and I were asked, with some impatience, why the future in our novels was so dour. Why write about the future at all when the present was, you know, so interesting? Doesn’t the real trump the unreal? And maybe most importantly: what was this attraction to dark visions of the last days, a burgeoning literary genre that might as well be called “end times porn”?

For years I’ve been asked to justify mercilessly sad endings, stories lacking in redemption and narrative visions that strip characters of their humanity through gruelling moral tests. Finally it is difficult to argue – no matter how true it feels – that pain and sorrow, in the literary sense, equal pleasure. Sometimes rhapsodic pleasure. But you know what they say about one man’s pleasure: you start to feel like a fetishist luring customers on the street to come inside and sample the delights of your whip.

The audience that day wasn’t satisfied by the observation that the phrase “happy novel” might be an oxymoron. What a lonely, underpopulated bookshelf would result that could house only happy novels. Neither did it help to intone that “the positive has already been given”. When you quote Kafka, even to a middle-aged Jewish crowd, people cry foul. Kafka was welcome to excuse his own bleakness, but we were not permitted to borrow his alibi.

Yet, in American fiction at least, the end times has graduated into de rigueur subject matter. Increasingly novelists cut their teeth on it and it’s starting to look like a rite of passage. Long a preoccupation of science fiction and horror writing, the apocalypse, as it looms closer, has become more intriguing to writers of literary fiction, more necessary to address. The last days no longer seem like a harmless fantasy. If this is a new development, it is worth considering why the end of the world is poised to join the suburbs and bad marriages as a distinctly American literary fascination.

After 11 September 2001, American novelists, asleep in their lairs, took heat for not responding soon enough to the attacks. In this case, “soon enough” meant within 24 hours. The question was pressing. Why must novelists take so long to process history? Or, worse, why must the novelist neglect history altogether, obsessed instead with the politics and sorrows within homes and suburban neighbourhoods that, measured against the rest of the world, are among the safest?

While other professional communicators had adapted to the faster times, the technological immaturity of the novelist was glaring, to some. What on earth was a novelist for, if not to rush to the microphone when his or her poor nation was battered and confused, to explain to everyone what the whole thing meant? Shouldn’t all of that training in graduate creative writing programmes have prepared these people for something like this?

Novelists are comfortable with the idea that their essential purpose should never be questioned but almost no one else is. In this case, concerns about the sluggish novelist’s absence from the public discourse betrayed a woeful, if predictable, failure of imagination. For a little while, the conspicuous silence of the novelist was in the news and eulogies were sung, again, for an art form that, over the past decades, had been left for dead several times over. But this was not the time for novelists to be defending themselves. As the 9/11 commentary and op-eds piled up, it became clear that many contemporary American novelists, in their work, maintain an asexual relationship to current affairs, even with game-changing events. Gone are the days of Mailer and Vidal pirouetting on television.

Among writers, many of whom were halfway out to sea on projects for novels that suddenly looked fit for the fish tank, a unique kind of despair set in. It wasn’t just that novelists supposedly had nothing to say about the events of 9/11. This didn’t, in the end, seem to trouble the writers I heard from. They weren’t political analysts and, in any case, opinion wasn’t the preferred rhetorical mode for writers besotted by narrative. The public field was already crowded with experts and it wasn’t hard to find articulate arguments from every perspective. Of far more concern to writers, if not shared very publicly, was that their narrative spectacles – when measured in terms of drama – could not possibly compete with the events of that day (never mind the atrocities that have occurred on other soil). Did all fiction suddenly amount to a kind of escapism, cowardly flight from what mattered most?

Many book projects were abandoned, and not just out of grief for the toll wreaked that day. In literary fiction, there was a new kid on the block and he was called “bad shit happening in your own backyard”. If this kind of thing could happen and wasn’t just fodder for science-fiction fantasies, by rights it must be a candidate for literary realism, which meant that the genre had some stomach-stretching to do. American realism would need to grow beyond the suburban bedroom, absorbing what, the previous day, had been pure fantasy. If truth used to be merely stranger than fiction, now it had also gorged itself on part of what made fiction special: its use of nightmarish material to haunt us. Reality seen supersize was more disturbing and commanded the nation’s attention in ways that made the imagination, and its artificial fruits, seem newly frivolous.

What is odd about this literary despair – the sense that one’s work is boring compared to a generation-defining surprise attack on the homeland – is that American novelists had not been especially celebrated for their ability to imagine plausible calamity . . . or, really, any kind of calamity (witness the serial indifference in critical circles to science-fiction writers, whose doomsday scenarios, worked out sometimes with fanatical logic, could populate a large encyclopaedia).

Nor were novelists frequently measured and assessed for their ability to conceive of grand, complex political dynamics, with the exception of Don DeLillo and Philip Roth and a few other writers. It was only now that calamity had struck that novelists were expected somehow to match it, to wield it, to make us feel the way the strongest events of history could make us feel.

To call the novel irrelevant because it couldn’t top 9/11 – that seemed strange, a botched diagnosis. But it did not prevent a shame from settling over writers who favoured domestic literary subject matter that could very well be deemed minor. One could find American fiction as a whole being dismissed by various parties (ie, entire nations) with this same accusation.

It just wasn’t about anything, or its concerns were too microscopic, indulgent, removed from the larger human drama that we all supposedly share.

The drama of the kitchen, the prototypical setting for American realist fiction, ignores the storm overseas. Marital fiction – fiction about how the people who supposedly love us the best misunderstand and wound us the most – is symptomatic of a deep lassitude enabled by decadence and comfort, which had been achieved, if only we could forget it, by a series of violent acts committed abroad in the name of our country’s foreign policy.

There is little question that US aggression by a bullying military that regards its home turf, which it must protect, to be roughly equivalent to the entire planet has allowed novelists, lest they be mistaken for pundits or political activ­ists, to focus on dramas closer at hand. Even fat, rich countries where no one ever dies have their grief and American fiction has a fine antenna for sadness in the home. Yet it may be that the relentless charting of this domestic territory, mastered decades ago by John Cheever and kept in fighting shape by Jonathan Franzen, came to seem especially redundant to novelists who were rethinking their enterprise.

A decade later, if American novelists have become less visibly interested in tackling the political complexities of our times, most notably how a superpower’s assertion of dominion exacts changes on our creative imaginations, that other call to arms, which asks for an escalation in our narrative spectacles, seems to have been taken more seriously.

Cormac McCarthy’s The Road (2006) is a good example. Among other things, the book is rigorously free of politics. There are no nations,

no governments. All of that is moot, because in the dusty aftermath of the apocalypse, wouldn’t it be? McCarthy, like most of his compatriots, has nothing to say. But he has a great deal to show, and the immense popularity of The Road may have something – aside from the excellence in its telling – to do with its perceived realism, its high degree of plausibility. This is an interesting achievement for a book in which almost nothing, outside of the characters’ reactions, is explained. How the world got this way we don’t and can’t know. McCarthy doesn’t bother going into any of the details, because what matters most to him is not the logistics of our demise but the fact of it. To read this book is to encounter something that seems inevitable, nearly indisputable.

A larger, unanswerable question arises. Do Americans read this book differently – say, as a great realist novel – because our nation’s ridi­culous lucky streak (a luck sustained through tremendous violence to others) was broken ten years ago and we got to sample, however briefly, feelings of deep vulnerability?

Nothing of the 9/11 attacks even remotely suggested an apocalypse but they certainly helped expose the troubling fiction of our immortality. Which might mean that fictions of our end times are now, through bad luck or comeuppance, however you wish to view it, among the truest and most realistic stories that we can tell.

Ben Marcus is an associate professor of creative writing at Columbia University. His next novel, “The Flame Alphabet”, will be published by Granta Books in June