David Foster Wallace was a maximalist. His masterpiece, Infinite Jest, is a 1,000-page, polyphonic epic about addiction and obsession in millennial America. His journalism and essays, about television and tennis, sea cruises and grammar, always swelled far beyond their allotted word counts (cut for publication, he restored many of them to their full length when they were collected in book form). In a letter sent to a friend from a porn convention in Las Vegas, Wallace exclaimed that, “writing about real-life stuff is next to impossible, simply because there’s so much!” It might seem surprising that a writer like this could or should want to function within the confines of the short story, yet besides Infinite Jest it is arguably his three story collections that represent the most important part of his work.

That said, many of Wallace’s short stories aren’t all that short, and often test the limits of traditional conceptions of story. As he told Larry McCaffery in 1993: “I have a problem sometimes with concision, communicating only what needs to be said in a brisk efficient way that doesn’t call attention to itself.” In fact, Wallace’s later works would rewire this statement: in order to say what needed to be said, he found his writing had no option but to call attention to itself. To experience a Wallace story is often also to experience someone making an agonised attempt to write a story. This was nothing new, of course: the postmodernists of the 1960s were committed to metafiction, the literary technique of self-consciousness that puts the lie to realism, making the audience constantly aware that what they are reading is an artificial construct.

This approach appealed to the young Wallace, who once remarked that Donald Barthelme’s short story The Balloon was the first work of fiction to “ring my cherries”, and who subsequently found a deep affinity with the work of Thomas Pynchon. Yet by the time of his first collection, 1989’s Girl With Curious Hair, and despite the significant debts individual stories owe to postmodern writers (John Billy is a tribute to Omensetter’s Luck by William Gass, while the political epic-in-miniature Lyndon takes its lead from Robert Coover’s A Public Burning), Wallace’s relationship with postmodernism had grown more complicated. He believed that a movement that had taken shape to unmask the hypocrisies of mass culture had come to lend them an insidious power: once advertising became knowing and ironic, the postmodernist game was up. Wallace began attempting to move beyond irony towards a new sincerity, although he struggled with how to achieve this.

‘Reflexive knots’ ... Wallace photographed circa 1996. Photograph: Gary Hannabarger/Corbis

The novella that ends the collection, Westward the Course of Empire Takes Its Way, is a tortuously long assault on postmodernism that paradoxically satirises the strategies of metafiction by employing an encyclopaedic array of metafictional strategies – skilfully enough that it could easily be taken for a piece of metafiction itself. It is illustrative of the struggle Wallace had throughout his career with the shape and content of his fiction, that after several years of considering the story to be by far the most important thing he had written, he then disowned it: “In Westward I got trapped one time just trying to expose the illusions of metafiction the same way metafiction had tried to expose the illusions of the pseudo-unmediated realist fiction that came before it. It was a horror show. The stuff’s a permanent migraine”.

It is possible to see Wallace as an artist who grew less certain of what he was doing the longer he did it, but at the same time becoming increasingly certain that this uncertainty was where he should focus his energy. It is this decision, and the scrupulousness with which Wallace pursued it, that can make areas of his work so tricky to engage with. His second collection, for example, Brief Interviews With Hideous Men (1999), is a brilliant book that is very difficult to enjoy. Reviewing it, the novelist Lawrence Norfolk noted the “brutal trades” one story makes “between the integrity of its rhetorical position and any pretension to aesthetic pleasure”, which is a useful way of considering much of his work from this point on: it tends to have very specific reasons for existing in the forms it does, but those forms can be rebarbative. They include extremely long and knotty sentences, monologues by obsessive bores, and a perverse love of the extreme ugliness of certain types of specialised language – marketing terminology, programmer’s English, therapy-speak, and so on. They are ouroboros-like stories that consume themselves at the same time as we consume them.

One of the best of this latter group is Octet, an exhilarating but enervating story that Norfolk found “maddening”, although he remained insightful enough to note that the story’s multiplying “reflexive knots … are not ironic reflexive gestures meant to distance the writer from the imminent implosion of his own artefact. They are Wallace’s own, sincere misgivings.” The story, a series of what Wallace calls “short belletristic pieces” that present a situation then ask the reader a question about it, collapses into itself halfway through, then proceeds to analyse the reasons for that collapse in a section that begins, “You are, unfortunately, a fiction writer”. By its author’s own admission, the story becomes increasingly “dense and inbent” as successive arguments and positions are explored, expanded on, and digressed from. There is a moment in Infinite Jest when the weed addict Ken Erdedy dwells on “the paralytic stasis that results from the obsessive analysis of all possible implications of both getting up from the couch and not getting up from the couch”. In Octet, Wallace is working himself to just such a standstill.

But when Wallace’s stories devour themselves, it is not a hip trick. The misgivings catalogued and explored in Octet are, as Norfolk noted, sincere. Zadie Smith has written that “how you feel about Octet will make or break you as a reader of Wallace, because what he’s really asking is for you to have faith in something he cannot possibly ever finally determine in language … his sincerity, his apparent desperation to ‘connect’ with his reader in a genuine way”. This idea of human connection, and whether it is even possible, is present right through his work, from the early story Little Expressionless Animals onwards, but its most urgent interrogation comes in his final and darkest collection, Oblivion (2004).

In an interview he gave in 1996, referring to Infinite Jest, Wallace talked about addressing “a real American type of sadness”. It is unsurprising to note that a writer who struggled with depression from his teens, who committed suicide at the age of 46, and whose stories nearly all contain at least a passing reference to depression and/or suicide, should have returned to sadness as a theme, but it is in Oblivion that it is captured most starkly. Critics tend to focus on the stories Good Old Neon (told from the viewpoint of a suicide) and The Suffering Channel (about a man who can defecate great works of art, and, indirectly, 9/11), and both count among Wallace’s best fiction, but there is another story in the collection, The Soul Is Not a Smithy, that is at once typical of his themes and extraordinary in the way it addresses them.

The editor and critic Sven Birkets, writing about his experience of first reading this story, describes “a density that was, at every step, forbidding – those sentences, the micro-obsessiveness of the narrating voice, the slow unfolding of suggestive implication that Henry James, title-holder in this category, would have applauded”. The story operates on several different levels: as an adult, the narrator reflects on the day in 1960 when he and three classmates were apparently held hostage by an unhinged supply teacher; in fact the narrator was unaware of being a hostage, because he was deeply involved in his habitual pastime of authoring a mental comic strip, the images of which appeared in the reticulate mesh of the classroom windows. Alongside this nested story – which we are told in great detail, and which is extremely funny, violent and sad – are interspersed memories of the narrator’s childhood home, and glimpses of his life since, including a detailed, digressive investigation of a dream sequence from The Exorcist.

Then, following the description of a recurring nightmare about his father’s office, the narrator presents an extraordinary portrait of this man, an insurance actuary who “for almost 30 years of 51 weeks a year … sat all day at a metal desk in a silent, fluorescent room, reading forms and making calculations and filling out further forms on the results of those calculations, breaking only to answer his telephone or meet with other actuaries in other bright, quiet rooms”. “I did not know,” the narrator says “that in mild weather he took his lunch down in the elevator and ate it sitting on a backless stone bench that faced a small square of grass with two trees and an abstract public sculpture, and that on many mornings he steered by these 30 minutes outdoors the way mariners out of sight of land use stars”. Then, devastatingly, he remarks that his father “died of a coronary when I was 16, and I can acknowledge, despite the obvious shock and loss, that his passing was less hard to bear than much of what I learned about his life when he was gone”. He means the everyday sorrow of it, the smallness of its pleasures against the vastness of its mundanity, and the fear that this is his birthright. Earlier, describing his father’s daily ritual on arriving home, he describes how “this routine … cast shadows deep down in parts of me I could not access on my own”. This account, in tandem with numerous other glancing references to disappointments and misfortunes scattered throughout the story, is the capstone to a profoundly sorrowful work of fiction.

Flipping the entropy of stories like Octet, or Adult World parts I and II – where the story begins coherently only to become more and more unconventional until it atomises – here, unexpectedly, it is mimesis that comes to dominate a narrative that for much of its length has been fragmented and surreal. For once, Wallace slips his bonds and writes through to the end without the story dissolving or blowing up in his face, and it feels as though he caught himself by surprise in doing so (which is perhaps why he described the story as “a very strange piece”). This story realises the ambition Wallace described to Larry McCaffery back in 1993, when he was willing himself to become the writer he most wanted to be: “Really good work probably comes out of a willingness to disclose yourself, open yourself up in spiritual and emotional ways that risk making you really feel something. To be willing to sort of die in order to move the reader, somehow. Even now I’m scared about how sappy this’ll look in print, saying this. And the effort actually to do it, not just talk about it, requires a kind of courage I don’t seem to have yet.”

Wallace’s fiction contains enormous cruelty: rape, animal torture, child abuse, the severe and perhaps fatal burning of a baby. Relationships are fractured, parasitic, and often the cause for psychic pain and disturbance; sex is furtive or coercive. It can be difficult to take, even when the knots and involutions of it aren’t making it difficult to read on a purely formal basis. But it is also a deeply moral body of work. Its difficulties, and many of its cruelties, exist for specific reasons. Whether Wallace’s fraught projects are successes or failures is up to the individual, but these are judgments that all serious readers should want to make for themselves.