The scene: a radio station in Bremen, Germany. The year: 1998. I'm doing what authors are often doing when they aren't writing a book: plugging one that they've already written. This one is called The House of Sleep. On the whole it's a rather melancholy, not to say twisted love story but there are a couple of scenes in it of the sort that critics refer to as "comic set-pieces". In one of them, two people are talking at cross-purposes about the burial of a dead body: one of them thinks they're talking about a human corpse, the other about a cat's. In another, the omission of one misplaced footnote reduces a scholarly article to nonsense.

I know these scenes are comic because I've read them aloud to audiences, and the audiences have laughed. It's always nice, that – when something you've written connects strongly with people. But I'm not connecting well with this radio interviewer. It's all going very badly, full of hesitant silences and dead airtime. So I decide to start riffing about comedy. I did an MA dissertation on theories of comedy at university so I'm on solid ground here. A whistle-stop tour of 18th-century theories of laughter, followed by a bit of Bergson and a dollop of Freud. My interviewer, I can tell, is impressed. Impressed, but also a little confused. From the other side of the microphone, he flashes what we comical writers like to call a quizzical look. "This is all very interesting, Mr Coe," he says. "But one thing about what you have just said puzzles me." I lean forward, confident that I can field pretty much any question on this subject. "If you are so interested in what makes things funny …" (a finely judged, possibly lethal pause) "why don't you write funny books?"

To this day I still don't know whether he just wanted to humiliate me on air or whether he had genuinely read the whole of The House of Sleep without laughing once. (Quite possible, I'm sure.) But what I do remember, as my mouth flapped open and shut like a landlocked fish, was having a sort of out-of-body experience. I suddenly saw myself, as if from a detached, ironic height, as a fictional character: as a character, that is, in a comic novel. A naif at loose on the continent, straight from the pages of Malcolm Bradbury, as prey to the whims of a random, indifferent universe as a luckless Michael Frayn protagonist.

I learned that day that the Germans have – or my interviewer, at least, had – very decided views on what constitutes a comic novel, and mine didn't fit the bill. But it has occurred to me since then that for most British writers and critics, the comic novel is an elusive thing to define, even though it's meant to be something at which we excel. The comic tradition, according to VS Pritchett in his 1969 Clark lectures, is "a dominant tradition of the English novel. In comic irony our novelists have been pre-eminent. It is their most militant and most graceful gift. It has moderated or refined their didactic habit and drawn them closer to nature." Agreed: but that still doesn't solve the conundrum. What is a comic novel? Is it simply a novel that makes you laugh? Is it a novel that takes a generally benign view of human nature and has a happy ending? James Wood, in an eloquent essay on Pritchett's own comedy, inches us a bit closer to the answer by pointing out that in the writing of those lectures, Pritchett "was really defining himself against the dominant English comedy of his day – that of Waugh and the early Powell, in which characters are clicked like draughts across metropolitan boards; a comedy of apparent heartlessness, in which the novelist is always a knowing adjective ahead of his characters." Wood sounds disapproving, and sure enough goes on to pick apart a passage from Evelyn Waugh's Decline and Fall, which he calls "the crudest comedy … clumsy … undergraduate" and so on.

Wood, I fancy, is not a fan of humour in fiction, or at least not the sort of humour that makes (some of) us laugh out loud. He says as much in his book The Irresponsible Self: Laughter and the Novel, which begins by listing a number of categories of comic novels, the final one being defined in the most damning terms of all: these are what he calls "comic novels" (the quotation marks are his own, the apparent punctuation equivalent of a clothes peg on the nose), namely "novels which correspond to the man who comes up to you and says 'Have you heard the one about …?', novels obviously very busy at the business of being comic". He names no names, but I'm hazarding a guess (perhaps unfairly) that the work of writers such as David Lodge, David Nobbs, Howard Jacobson and Kingsley Amis might fall under this heading, in his view. The defining pictorial identity of such a novel would be Paul Sample's cartoon cover illustrations for the Pan editions of Tom Sharpe in the 70s and 80s: half Giles, half Beryl Cook, full of purple-faced men in tweeds and buxom women popping grotesquely out of their corsets.

If that is indeed what we mean by the modern English "comic novel", we can say that just as sexual intercourse began in 1963, so this phenomenon arose in 1954, with the publication of Amis's Lucky Jim. This might be the place to look, then, if we are going to define the essential characteristics of the genre. To start with, we have a protagonist (male) whose morality, intellect or value system brings him into conflict with the world around him. This conflict expresses itself in a series of episodes that create frustration and embarrassment, but nothing worse: in this instance our hero, Jim Dixon, has to attend a farcically awful artistic weekend organised by his boss, accidentally burns a hole in his host's bedclothes with a cigarette and, in the book's celebrated climax, goes on to make a mess of an important lecture by turning up drunk. The cumulative consequence of these episodes is that Dixon loses his job, but in a swift deus ex machina he gets another one anyway, and scoops up the prize of an attractive girlfriend into the bargain: thereby bringing to a triumphant conclusion the novel's treatment of its main theme, as flagged up in its opening pages: "the awful business of getting on with women".

Albert Finney in the film adaptation of Henry Fielding's Tom Jones. Photograph: REX/Moviestore Collection

Summarised like that, Lucky Jim is not so very different in its outline from Joseph Andrews or Tom Jones, written by Henry Fielding some 200 years earlier. Nowadays, I think, both Joseph and Tom remain far more attractive characters than Jim Dixon, many of whose attitudes look like mere peevishness. What these books do have in common, however, is their judgmental quality. Amis was a huge admirer of Fielding, paying explicit homage to him with a scene by his graveside in Lisbon at the end of I Like It Here (1958). And although Amis's third-person narrator in Lucky Jim is not as intrusive as Fielding's, or as keen to spell out his opinions, he consistently directs us to see things from Dixon's point of view, and leaves us in no doubt that from this perspective most of the surrounding characters are to be considered, for one reason or another, ridiculous and worthy of mockery. This would seem to be a typical instance of what Wood, in his introduction to The Irresponsible Self, calls the "theatrical tradition" in comic writing, and he traces the moment at which most English novelists grew out of it to the work of one writer: Jane Austen. He sees Austen as a crucially transitional figure because in her books "there are the minor characters, who seem to belong to the theatre, and who are theatrically mocked and 'corrected' by the author in her old 18th-century satiric mode; and there are the great heroines … heroic because they exercise their consciousness, who seem to belong to the newer world of the novel and not of the theatre, and who are not mocked but gradually comprehended and finally forgiven".

This is an attractive theory, but I think it underestimates the amount of groundwork that had been done by the 18th-century novelists in general and Fielding in particular. If we are to pinpoint a moment at which comic writing moved from the two-dimensionally theatrical to the novelistically nuanced, the key figure is surely Fielding himself, who began life as a highly successful playwright before turning to the novel in the 1740s. In his definitive book on 18th-century comic literature, The Amiable Humorist, Stuart Tave argued that "the 17th-century concept of humour as an aberration demanding satiric attack was essentially reversed … by the creation of amiable humorists, among whom Parson Adams [from Joseph Andrews] and my Uncle Toby [from Tristram Shandy] were the most important". Because he is drawn in such broad comic outlines, in other words, we tend to forget that Fielding's Parson Adams – a figure who is both ridiculous and lovable – was himself a revolutionary creation who may not be equal to, but certainly paved the way for, the more psychologically complex characters we now have a tendency to associate with non-comic fiction.

Gwyneth Paltrow and Toni Collette in Emma. Photograph: Allstar

Rather than setting up a binary opposition between the theatre and the novel, Pritchett proposed a more intriguing division of English comic writing into "masculine" and "feminine". His "masculine" tradition, unsurprisingly, is the one that leads straight to Amis, via Fielding, Scott, Austen, George Eliot, Waugh, Ivy Compton-Burnett and Anthony Powell, among others. This tradition, according to Pritchett, is "sanguine, sociable, positive, morally tough, believes in good sense and suspects sensibility. These novelists have paid their dues to society or a moral order."

Against this lineage he posits an alternative "feminine" tradition, which is initiated, rather confusingly, by a man – Laurence Sterne – and encompasses the likes of Peacock, Dickens, Firbank, Woolf, Joyce and Beckett. "Sterne dissolved the sense of order," Pritchett wrote. "He saw that … we have a mind that does exactly as it pleases, moves back and forth in time. The 'I' is not a fixture: it dissolves every minute; its movements are as uncertain as the transparent jellyfish as it washes back and forth in the current. The Sterne or feminine strain in our comedy … follows consciousness from sentence to sentence, image to image … [It] is receptive to sensation and believes in the mingling of meanings and in the oblique." And whether or not we agree that this alternative tradition should be referred to as "feminine", how modern it sounds, suddenly, how much more appropriate to the instantaneity, the fractured realities, the fluid identity politics of 2013 than its "masculine" counterpart. All at once those crude cartoon covers, and the sexual and moral certainties they embalm, feel even more out of step with the times.

However, the legacy of Lucky Jim continues to survive into the 21st century, just about. Novels featuring academics who get themselves into scrapes are still being written. Although the principal inspiration for Zadie Smith's On Beauty (2005) is clearly EM Forster, it climaxes with a lecture given by the protagonist, Howard Belsey, which seems to be a knowing replay of Jim Dixon's woeful performance. In Michael Frayn's Skios (2012), the delivery of a lecture at a mysterious privately funded institute on a Greek island is the comic denouement which the narrative endlessly promises, and endlessly defers. Frayn's novel, in fact, can be read as a kind of apotheosis or ne plus ultra of the genre. As the plot whirrs its way forward like a brilliantly designed Heath Robinson machine, powered by serial cases of mistaken identity and linguistic confusion, we feel that never has any novelist been more determined to see his characters "clicked like draughts" across boards. The final coup de grace comes with the throwing of a random spanner into the works: a moment when all is reduced to chaos, as Frayn writes, by "a completely unconnected and irrelevant event … a velleity that comes out of nowhere and has no imaginable significance or place in any self-respecting causal chain". But this irruption of apparent unpredictability does not make the novel more naturalistic: on the contrary, it simply serves to make us see the puppeteer's strings more clearly, and to increase our admiration for the deftness with which his fingers are manipulating them. Skios, above all, is a triumphant affirmation of the comic novel's wilful artificiality.

Ian Carmichael in Lucky Jim. Photograph: Ronald Grant Archive

In days gone by we might have expected something similar from David Lodge: a sense of high-spirited absurdism inscribed within the twists and turns of a well-patterned plot; but his most recent "comic novel", Deaf Sentence (2009), is a more complicated beast. It starts off with all the familiar tropes: a retired academic (Desmond Bates) haplessly buffeted by events outside his control; a potentially absurd intrigue with a female postgraduate; a succession of virtuoso comic set-pieces at various awful social gatherings. But the tone of the book soon begins to darken. Sharing Frayn's love of the pitfalls of the English language, Lodge plays insistently on the similarity between "deaf" and "death", and before long it's the latter that begins to dominate. As the aurally impaired narrator sets off on a British Council tour of Poland, we wait for another comically climactic lecture in the tradition of Lucky Jim, On Beauty and Skios. But we never even see him step up to the lectern. What we get, instead, is a haunting account of his visit to Auschwitz, followed soon afterwards by the news of his father's death. It's as if Lodge himself, weighed down by melancholy, is announcing the death of the "comic novel"– or at least the impossibility of writing it any more, in the face of human tragedy. The narrator says as much earlier in the book, when, after a short passage in which he offers some wry one-liners about life's petty irritations in the Kingsley Amis mode, he admits: "Somehow it is easier to focus one's anger and despair on these comparatively trivial offences to reason and decency than on the larger threats to civilisation like Islamic terrorism, Israel/Palestine, Iraq, Aids, the energy crisis and global warming, which seem to be beyond anyone's ability to control. I don't think I have ever felt so pessimistic about the future of the human race, even at the height of the cold war, as I do now, because there are so many possible ways civilisation could come to a catastrophic end, and quite soon."

And here, perhaps, we can see the obvious reason why the "comic novel" might currently seem to be on life support. As Pritchett observed, "our comic tradition in the novel … has one serious defect: the lack of tragic irony. It is extraordinary when one thinks of the influence of Don Quixote on the English novelists how they have all – with gentlemanly or middle-class optimism, natural I suppose to an expansive culture which was dramatising its satisfactions – avoided the tragic conclusion." Is that all Jim Dixon was really doing, then? Were his potshots at the minor hypocrisies of postwar society just a form of complacency: just a way, essentially, of "dramatising its satisfactions"? And now that we know, post-9/11, post-2008, how extremely fragile those satisfactions are, is it time for novelists to jettison comedy altogether, like Desmond Bates in Deaf Sentence, and finally get serious?

It sounds convincing. Indeed, the rational case for the stifling of laughter until our global problems have been soberly addressed and sorted out seems unanswerable. Jim Dixon may have had the cold war to worry about, but in other respects he seems to have been living in a fool's paradise, a bubble of provincial ignorance which today's novelists are simply not permitted to share. Where are the laughs in massacre, famine and climate change, exactly? What's so funny about the Middle East, North Korea and Afghanistan? Who's going to chuckle when they pick up the London Review of Books and find John Lanchester arguing, effectively as always, that the banking habits of the British people pose a greater threat to their own security than terrorism?

Another recent article in the LRB, for that matter, argued that Britain's much-vaunted tradition of political satire was itself an obstruction to real social change, since it diverted everyone's contrarian impulses into harmless laughter. I was the author of that piece, which sprang from a growing disillusionment with the role played by laughter in the national political discourse. But satire and comedy are two quite different beasts – although many people insist on using the terms interchangeably – and no amount of rationalism or essay-writing can undermine my allegiance to comedy. Whenever I start to look at the world from the novelist's point of view, the first thing I notice is that it contains grimness and absurdity in equal measure; and whenever I look at it from the reader's point of view, I long to find that same sense of comic ruefulness reflected back at me. You get a healthy dose of it, certainly, from the work of Marina Lewycka and Catherine O'Flynn. Even Nicola Barker's warped imaginings summon up gales of uneasy laughter, and AL Kennedy is at her funniest when she's at her most dour. Will Self's Umbrella made me laugh, on occasion, because he's so clearly channelling Joyce and, as Flann O'Brien (who else?) wisely remarked, "humour, the handmaid of sorrow and fear, creeps out endlessly in all Joyce's works".

Hugh Laurie and Stephen Fry in the TV adaptation of PG Wodehouse's Jeeves and Wooster. Photograph: Rex Features

All of this leads us inevitably to PG Wodehouse, the elephant in my comic room, about whom I've been silent for too long. We must admit that there is not a grain of satire or moral seriousness in his novels and of course he proved himself, during the war, to be possessed of an incredible political naivety. But while it should have been obvious to me that these very qualities are the key to his greatness, for a long time they made me feel stupidly snobbish about Wodehouse and reluctant even to read him. Some years ago I was lucky enough to be awarded a prize in his name, and with it came a complete set of the Everyman edition of his works. It was only then that I realised the pure, unpolluted humour of which he was possessed was the greatest possible gift he could have offered to the world: the same thing, I suppose, that Italo Calvino had in mind when he extolled the virtues of "thoughtful lightness", or "comedy that has lost its bodily weight". More and more I feel that, just as all art aspires to the condition of music, all humour should really aspire to the condition of Wodehouse.

As a postscript, I'll add that receiving this particular prize usually entails being presented with a pig, but it didn't happen in my case, because the year was 2001 and the foot-and-mouth crisis meant there was a nationwide ban on the transportation of livestock. That strikes me as pretty funny, as well, in a way: so funny that I might even put it in a novel some time. But I doubt that radio presenter in Bremen will crack much of a smile when he reads it. Some people just never seem to get the joke.

• Jonathan Coe's Expo 58 is published by Vintage.