In Martin Amis's England, a curious documentary broadcast earlier this year on BBC4 – essentially a not-particularly-dynamic interview padded out by lots of nostalgia-inducing stock footage of cricket matches, punks and toffs – the author found himself trying to sum up what it meant to him to be English. Amis's comments were characteristically revealing. Germany, Amis remarked, had made immense efforts to come to terms with its wartime past; France, by contrast, had not, bolstering its resistance myth to conceal the collaborationist reality. But either way, he continued, "it takes all my powers of imagination and empathy to imagine myself in a French skin or a German skin for that reason. How tremendously diminished I would be."

But it is foreign skins – predominantly German, one Polish, with other nationalities, including French, merely glimpsed – into which Amis attempts to insert himself in his 14th novel. The Zone of Interest follows Lionel Asbo, subtitled "State of England"; but what could a work of fiction set in Auschwitz – here named Kat Zet, as in Amis's 1991 novel about the Holocaust and its perpetrators, Time's Arrow – be reasonably trying to establish the state of? We imagine, perhaps, that its subjects will include violence and psychopathy, racial hatred, systematised killing, suffering on an enormous scale and survival on a far smaller one. And this proves to be true, although they are joined by other, ostensibly more mundane themes: sexual rivalry and failure, the frequently shabby power struggles of hierarchical organisations, the sheer amount of manpower and materiel it took to attempt a genocide.

And yet Amis's project comes with a serious caveat: in his Afterword, he describes his previous failed attempts to get anywhere with the Holocaust; despite reading yards of books and amassing plenty of knowledge, "I gained nothing at all in penetration." It was only when he came across a piece of writing by Primo Levi – a small-print addendum to a companion volume to If This Is a Man – in which Levi argued that we can and should place both the thoughts and the actions of the Nazis beyond comprehension, to mark them as "non-human" or even "counter-human", that Amis began to feel artistically liberated. As he told Ron Rosenbaum, author of Explaining Hitler, in a 2012 interview for the Smithsonian magazine, "as soon as the pressure to understand" left him, he was able to write.

What, then, has Amis done with Levi's "sacred duty not to understand"? He has turned it, of course, into a brutish comedy, an occult tale of jealousy and revenge, a farce of thwarted will and missed cues. He has thrown in a gently painful love story of sorts and even more quietly underpinned the novel's manic top notes with the agonising testimony of Szmul, a concentration camp inmate forced to become one of its key workers. Szmul's story, framed in the language of fairytale or parable and yet brimming with grimly suggestive detail ("nearly all our work is done among the dead, with the heavy scissors, the pliers and mallets, the buckets of petrol refuse, the ladles, the grinders"), sits alongside a comedy of debauched manners, amid the tea dances, piano recitals and boozy dinners enjoyed by those marooned in the muddy Polish fields to execute the Final Solution. The focus in The Zone of Interest is the Nazi officers themselves, and their increasing difficulty in fulfilling the demands of the Chancellery.

It is 1942, and they are beginning to find themselves hamstrung by their own strictures and actions; German women are not supposed to smoke, and yet tobacco fug comes in handy to disguise the stench of decaying bodies. A sergeant is killed in the line of duty but when his wife returns to Berlin to complete the paperwork, she is arrested on the grounds that she has Romany blood and sent back to the camp as a prisoner, thereby complicating matters for officers who would like to sleep with her. Those other, less distinguished prisoners, who arrive in consignments of hundreds in trains for which they are forced to pay their own one-way fare, are becoming harder to manage in the face of the evidence. As the war continues, thinks Camp Commandant Paul Doll, they begin to realise that "Nobody Has Ever Come Back"; "we have lost," he reflects ruefully, "the 'element of surprise'".

The blustering Doll, drunken, puffed up, powered by ambition and grievance, endlessly self-pitying and self-justifying and yet utterly convinced of his greatness, is one of the novel's three narrators and a terrific comic creation. He is, essentially, off his head, and only becomes more so as matters worsen: he is the zealous middle-manager spouting jargon and euphemism, in love with his own creation myth – "in Dachau, where I launched my meteoric rise through the custodial hierarchy" – sexually incontinent and incompetent, loathed by his wife. The camp's struggle to process its victims ("How to make them burn, naked bodies, how to make them catch?") he regards as a personal torment, his struggle to keep head office at bay as just the latest in a series of bureaucratic headaches.

More sane – though it is all relative – is Angelus "Golo" Thomsen, who delivers his take on concentration camp life from a rather privileged perspective; he is the blue-eyed, strong-jawed nephew of Hitler's private secretary, Martin Bormann, and consequently more or less untouchable, even when he falls in love with Doll's wife, Hannah. Employed in a shadowy liaison capacity that involves smoothing the wheels for the creation of IG Farben's synthetic rubber factory, much of his time is spent contemplating what to do with his body, with "the extensile penis, classically compact in repose (with pronounced prepuce), the thighs as solid as hewn masts, the kneecaps square, the calves Michelangelan", and getting drunk with his friend Boris, a pugnacious member of the Waffen-SS champing at the bit to go east for the fight. Until he does, he and Golo, as young men are wont to do, spend their time swapping anecdotes about their bosses and pondering what it all means. Here they are, mulling over whether the Nazis' hatred of the Jews has gone too far:

I said, 'Would you agree that we couldn't treat them any worse?'

'Oh, come on. We don't eat them.'

For a moment I thought about this. 'Yes, but they wouldn't mind being eaten. Unless we ate them alive.'

'No, what we do is make them eat each other. They mind that … Golo, who in Germany didn't think the Jews needed taking down a peg? But this is fucking ridiculous, this is.'

It is the hackneyed complaint of clear-eyed corporate underlings everywhere – "this is fucking ridiculous, this is" – and with it Amis captures a world of frustration, jockeying for position and organisational strain. Here, the drive towards racial purity, the destruction of an entire people, the reconstitution of Europe's population, is seen as a boss-pleasing initiative gone wrong, a project badly handled. Even when Golo is caught in conversation with his uncle, towards the novel's end, the talk is all of party machinations, of the latest antics of "the Cripple" (Goebbels), "the Transvestite" (Göring), "the Quack" (Himmler). Elsewhere, the continuation of mass murder, sterilisation and abortion is figured as a self-perpetuating madness with quasi-supernatural overtones, in which "the appetite of death is truly Aztec. Saturnian."

The question is this: has Amis created anything more than this bravura black comedy? Are his monsters more than cartoon grotesques? The answer is a definite yes, if what we mean is that he has created a fictional artifice that allows us to see the outline of that which is beyond words. The silence, as so often, is heard in the gaps between the laughter, in the realisation that we've been laughing at all. Both Doll and Golo's narratives, filled with striking incidents and opaque mysteries, bounce us happily and zestfully along. But it is Szmul, far less entertaining a narrator, who calls us to examine the inner life of a man forced to become an accessory after the fact in the murder of his own people. "I feel we are dealing with propositions and alternatives that have never been discussed before, have never needed to be discussed before,'" he tells us. "I feel that if you knew every day, every hour, every minute of human history, you would find no exemplum, no model, no precedent." How can these three stories possibly coexist in the same sphere, in the same zone of interest? How can we bear to read them, and what are the implications of our doing so, and of our laughter? If there is an answer, we cannot give voice to it. Szmul again: "I am choking, I am drowning. This pencil and these scraps of paper aren't enough. I need colours, sounds – oils and orchestras. I need something more than words."

• To order The Zone of Interest for £14.79 with free UK p&p call Guardian book service on 0330 333 6846 or go to guardianbookshop.co.uk.