Sometime in 2006 or 2007, Karl Ove Knausgaard, an acclaimed novelist in his native Norway, discovered that he was sick of fiction. "It was a crisis," he wrote. "Just the thought of a fabricated character in a fabricated plot made me nauseous." The only genres of writing he still found valuable were diaries and essays, "the types of literature that just consisted of a voice, the voice of your own personality, a life, a face, a gaze you could meet". His solution was to embark on a massive project, a first-person narrative about his life that eventually ran to six volumes and 3,600 pages. He wrote at a feverish pace, publishing three books in 2009, two more the following year and a 1,000-page finale in 2011.

The reaction in Norway was unprecedented. By 2012, the books had sold 458,000 copies – a staggering figure in a country of 5 million people. Knausgaard's unflinching descriptions of his marriages, his father's alcoholism, his second wife's bipolar disorder and his conflicted feelings about fatherhood were profoundly shocking to the Lutheran sensibilities of a country that is less comfortable with public confessions than the Oprah-soaked anglophone world. Commentators obsessed over the new literary sensation. Was Knausgaard brave or shameless? Was it ethical to reveal so many other people's private lives along with his own? He received hate mail, death threats. His uncle started legal action. One Swede took the extreme critical step of setting fire to the K section of a Malmö bookstore, telling police that he did it because Knausgaard was "the worst author in the world". And what about the title – Min Kamp or My Struggle, a reference to Hitler's notorious autobiography? The final book (yet to appear in English) contains a 400-page essay on the Nazis and ends with a discussion of the anti-immigrant mass murderer Anders Breivik. What face was this man revealing? What kind of gaze did he want his readers to meet?

So far only the first three volumes of My Struggle have been translated, so it's hard for a non-Norwegian speaker to give a definitive answer to this question, but Knausgaard's reputation is already soaring in the English-speaking world, not least among writers. "I just finished the second volume of Karl Ove while high above the clouds in a plane," emailed one acclaimed novelist. "It was really one of the most sublime reading experiences of my adult life – almost equal to the kind I remember from childhood." "His arguments remind me why I got into this game in the first place," added another. I understood. I'd badgered Knausgaard's London editor for early copies, dispatched them in a few long sittings. But why? The commentary on My Struggle tends to focus, as I have in these paragraphs, on the phenomenon of the book's publication, rather than the writing itself. It is peculiarly difficult to get a grip on what makes the book so compelling, because much of it appears painfully banal. Knausgaard can spend pages describing cooking dinner, or his daughter's abstracted expression as he changes her nappy. He remembers teenage arguments with his father, making out with girls at parties. It is the stuff of an ordinary life, told without much embellishment, without much that declares itself (at least on first reading) as literary style.

Literature is usually considered to be something to do with heightened intensity. We call some kinds of language "poetic". We find some kinds of narrative "gripping" or "page-turning". So what are we to do with this?

I found a hairdresser's I hadn't seen when I passed by the first time, in a passage beside the big department store. I just had to take a seat. No wash, my hair was moistened with water from a spray. The hairdresser, an immigrant, a Kurd I guessed, asked how I wanted it, I said short, indicated with my thumb and first finger how short I had in mind, he asked what I did, I said I was a student, he asked where I came from, I said Norway, he asked if I was here on holiday, I said yes, and that was it.

Knausgaard is capable of writing sharp, essayistic sentences:

Memory is pragmatic, it is sly and artful, but not in any hostile or malicious way; on the contrary, it does everything it can to keep its host satisfied. Something pushes a memory into the great void of oblivion, something distorts it beyond recognition, something misunderstands it totally, something, and this something is as good as nothing, recalls it with sharpness, clarity and accuracy. That which is remembered accurately is never given to you to determine.

Oslo, where Knausgaard was born. Photograph: Jeff Gilbert/Alamy

He can also write lyrical description, the kind of prose it is easy to tag as "literature". But often he chooses to write as he does about his trip to the Stockholm hairdresser, scrupulously, with mild pedantry, about something mundane. However, these passages are often deceptive. When his haircut is finished, he looks down at the locks of hair on the floor around the barber's chair and makes an odd observation:

They were almost completely black. That was strange because when I looked in the mirror I had fair hair. It had always been like that. Even though I knew my hair was dark, I couldn't see it. I saw fair hair, as it had been in my boyhood and teens. Even in photos I saw fair hair.

So, Knausgaard thinks of himself as a blond. What of it? It is an incident that doesn't have any particular fallout. He never sees the Kurdish hairdresser again. The haircut is satisfactory. There is no revelation about his – what to call it? Hair dyschromia? – beyond the author's curiosity at the trick his memory and perception have played on him, and a slight note of unease at the gap between reality and his image of himself.

This is not boring in the way bad narrative is boring; it is boring in the way life is boring, and somehow, almost perversely, that is a surprising thing to see on the page. My Struggle (a slippery, self-ironising title) is composed of small incidents, some described at great length – 50 pages at a children's party, more about a teenage plan to hide some cans of beer one New Year's Eve. There are sections about more traumatic or intimate events – the harrowing job of cleaning up after his father's death, a drunken episode of self-cutting after a sexual rejection at a young writers' residential course – but Knausgaard appears to have shaped his narrative according to the "sly and artful" dictates of his memory. One has the sense that many significant things have been omitted, while seemingly insignificant things are being given undue or unlikely weight. In the first two volumes the narrative hops about between times and places, incorporating digressions about art and writing and the nature of remembering. The third is a more conventionally linear childhood memoir. What there isn't is a plot. The various events are allowed to take their own shape, without being forced into a conventional mould.

We are used to memoirs that read like bad novels, following certain familiar arcs. Survivors describe the obstacles they have surmounted to become the healthy, happy people they are today. The rich and famous take us with them on their rise to the top. Whistleblowers and the protagonists of scandals tell their forbidden secrets. In My Struggle, Knausgaard attempts something more disconcerting. He tries, as he puts it, to show his face. The incident of the dark hair speaks, like much in the book, to his search for a kind of self-knowledge that isn't in thrall to hackneyed narrative conventions or, crucially, to the near-universal desire to appear a certain way to others, to be admirable or heroic or simply likable. "The life I led was not my own," he writes. "I tried to make it mine, this was my struggle, because of course I wanted it, but I failed, the longing for something else undermined all my efforts."

The governing emotion of My Struggle is shame. When Knausgaard drunkenly cuts his face at the writers' residency, he tells us he had "never experienced such shame before". As a child he is ashamed of a floral bathing cap his parents make him wear to the pool. As a 24-year-old he is ashamed of his intense ambition, which he fears makes him ludicrous. As an aspiring writer, he is ashamed that he takes a well-known novelist's sarcasm for a compliment. Doing a TV interview, he decides his weakness for approbation is a personality flaw. "I was a whore," he writes. "This was the only suitable term."

This shame is bound up with his father, one of the great suburban monsters of literature. Knausgaard isn't beaten or sexually abused, but grows up in an atmosphere of grinding rectitude and pervasive bitterness. The older Knausgaard terrorises his two sons, his thwarted hopes finding expression as sarcasm (Karl Ove has a speech impediment, which is cruelly mocked) or eruptions of anger occasioned by the most minor infringements of discipline. When he loses a sock at swimming practice, Knausgaard is so afraid that he weeps. It is too simplistic to say that My Struggle is written to overcome a sense of worthlessness, but that is surely one of its driving impulses. It is also an act of self-shaming, a public display of wounds that has opened the author to accusations of narcissism and self‑aggrandisement.

Knausgaard's struggle with shame certainly feels genuine, though not altogether straightforward. If you are truly ashamed of something, do you share it? Doesn't sharing your shameful secret imply you take some pleasure in exposure, the flasher's thrill? But even if Knausgaard is serving his self‑revelation in a sauce of kink, it takes on more than therapeutic importance when one considers the way shame functions as a sort of barrier to authenticity. We do something in private – cheat or steal or tell a lie. If our moral failing is exposed, we feel shame, not because we have transgressed, but because we have been observed in our transgression. Shame is essentially public, and the near-universal desire to avoid it is an effective form of social regulation. So, if your artistic project is to look at yourself without shame (also, presumably, without vanity) in order to see yourself authentically, you must pay a social price. We deeply value the good opinion of others – at the limit it is all that keeps us safe from attack – and even if a writer's desire to overcome shame is sincere, it would be surprising, perhaps even pathological, were he to be able to overcome it fully and without pain. Seen like this, My Struggle becomes an almost political exercise in transparency, a response to the "always on" culture of cams and surveillance that has destroyed our old expectations of privacy.

Bergen, where Knausgaard went to univeristy. Photograph: Graham Turner for the Guardian

Many critics treat My Struggle straightforwardly as memoir, praising it as some kind of unusually neutral transposition from life into art. In Norway it was published as a novel, a small provocation that the English-language publishers have dispensed with. Knausgaard is no James Frey, cashing in on the public's desire for sincere self-revelation by faking events and people that don't exist. But he is, inevitably, an unreliable narrator. How could he not be? We live a life of many dinners, many haircuts, many nappy changes. You can't narrate them all. You pick and choose. You (in the unlovely vernacular of our time) curate. There are also personal and legal pressures. Before publication, the manuscript of My Struggle was shown to people who appear as characters, and some asked to be taken out. Calling this book a novel wasn't (or wasn't just) some kind of legal fiction, but a warning to the unwary, to those who don't stop to consider that this is art. Knausgaard explores his relationships with his brother and father in depth, his mother less so. In reality, there is at least one other close relative who does not appear in the text and does not wish to be associated with the book or its author. Knausgaard's paternal uncle asked to have his name changed, and has attacked the project in the Norwegian press. Once one has started tugging at these threads, the idea that this is a neutral recollection begins to unravel.

Remembering the traumatic day his father tried to teach him to swim, Knausgaard specifies that he was wearing "the blue Las Palmas T-shirt I had been given by my grandparents". As they return to the car, his father carries a cooler bag, a life jacket and a wet towel. Knausgaard specifies the order in which they were placed in the boot. The author appears to have perfect recall, though of course he has already told us he is aware of the sly, artful tricks of memory. So are these details and the thousands like them invented? Are they perhaps collaged together or reconstructed? Part of Knausgaard's struggle is with truth and memory, a battle to reconstruct the past performed by someone using all the tools of the novel. In a memoir, one would call the missing relative a problem, a sin of omission, and ask, what is reality? Am I being fooled? In a novel, we apply different standards. We ask, what is this feeling of reality? It is visible to us as an aesthetic, an effect of artistic technique.

In an earlier passage, Knausgaard looks at paintings, trying to chase down a feeling he has in front of some but not others: certain works by Vermeer or the aged Rembrandt's self-portrait in the National Gallery, but not Rubens or Whistler or Michelangelo.

I don't know what it was about these pictures that made such a great impression on me. However it was striking that they were all painted before the 1900s, within the artistic paradigm that always retained some reference to visible reality. Thus, there was always a certain objectivity to them, by which I mean a distance between reality and the portrayal of reality, and it was doubtless in this interlying space where it 'happened', where it appeared, whatever it was I saw, when the world seemed to step forward from the world.

As a description of aesthetic realism, of how it seeks to function, this is very fine. The experience of reading My Struggle is that of the world seeming to step forward from the world. It is not the world mirrored or photocopied; its relationship to reality is less direct, less innocent. The book is the record of someone trying and failing (failing better, as Beckett has it) to make an accurate representation of himself; the gap between the world and that representation, between the world and itself, is the space where all sorts of questions about truth and personal identity arise.

There is a Nordic tradition of such writing, writing that seeks, at considerable length, to make the world step out from the world. Knausgaard himself nods to the Norwegian Modernist poet Olav Hauge, who lived his whole life in the same small coastal village and left a 4,000-page diary that he kept from the age of 15 until his death 70 years later. There is also the Finn Kalle Päätalo, whose 26-volume Iijoki series is considered the longest published autobiographical narrative in the world. Though it deals only with the author's life from birth to the publication of his first novel, a period largely spent in forestry and construction work in northern Finland, it is apparently very popular, earning Päätalo the nickname King of the Reprints. Knausgaard's most relentless public detractor is the novelist Ole Robert Sunde, whose own multi-volume autobiography (in progress since 1984) has received significantly less attention than his rival's, leading him to give bitter interviews in which he declares that he is the better writer, and is only less popular because his prose is technically demanding and he does not stoop to "lifting the duvet".

Knausgaard's second novel, written just before his crisis of fictional confidence, is A Time for Everything, an extravagant and compelling history of angels and their interactions with humans. Knausgaard's angels are wounded, tragic and, finally, debased, gradually degenerating as they fall away from God. The book retells several Old Testament stories, reimagining those that take place before the flood in a Norwegian landscape of fjords and forests. If a reader comes to it, as I did, after My Struggle, its exotic subject matter seems like a skin over a familiar body. Those two brothers, Cain and Abel. Loving, jealous, competing. Are they familiar? What about that wrathful God? There is a sense that the writer is using material from his own life, not as biographical details (though there is some of that too) but in the form of emotional dynamics, clusters of feelings and experiences. You find yourself thinking, yes, this is how brothers are. This is what it is to have an angry father. The two books, one that seems so flagrantly novelistic, the other so rigorously autobiographical, are actually far closer than they appear. The concern for personal authenticity is also part of a quest for purity. The sensibility attracted to writing about angels is clearly the same as that of the self-disgusted man who needs to tell the truth about his life.

When I look at a beautiful painting I have tears in my eyes, but not when I look at my children. That does not mean I do not love them, because I do, with all my heart, it simply means that the meaning they produce is not sufficient to fulfil a whole life. Not mine at any rate.

For writers who are also parents, Knausgaard wins points for facing up to the undeniable conflict between domestic life and work. His love for his children is portrayed with great tenderness, but he yearns to have more time to himself, to live "another kind of life". He strains at his bonds, eyeing up women, outraged at the thought that an attractive young teacher sees him as a sexless, unthreatening dad. At times, he finds his lot intolerable:

Everyday life, with its duties and routines, was something I endured, not a thing I enjoyed, nor something that was meaningful or made me happy. This had nothing to do with a lack of desire to wash floors or change nappies but rather with something more fundamental: the life around me was not meaningful. I always longed to be away from it, and always had done.

At times he can be very funny, describing himself striding about the city, a "19th-century man" seething with anger, a baby in a carrier strapped to his chest. He feels an affinity with the baroque, a period before life lost touch with "the essence". As Knausgaard drops off his kids at the state-run daycare, he grumbles about the emasculating effects of Scandinavian social democracy, to the sound of teeny-tiny violins from those of us who are not eligible for a year's parental leave at 100% of our salaries. His sociologist friend Geir, who seems to be his chief intellectual foil, writes a book about a boxing gym. "There the values that the welfare state had otherwise subverted, such as masculinity, honour, violence and pain, were upheld." Geir produces essays on such topics as Karl Jaspers' existential notion of the "limit Situation", and introduces Knausgaard to "the great classical antiliberal cultures, in a line extending from Nietzsche and Jünger to Mishima and Cioran. There nothing was for sale, nothing could be measured in terms of monetary value …"

These are all fine writers and I am more than sympathetic to the idea that if we are to survive the age of neoliberal globalisation we need to revive non-economic conceptions of value, but when someone lines up this particular canon, all I see is a truckload of honourable violence and pain heading in my direction, driven by a man with very shiny buttons on his uniform. Knausgaard's unapologetic Romanticism, his love of Hölderlin, his wish for an art in which "the flames of truth and beauty burn" heighten his sense of spiritual impoverishment, but also leave him receptive to the kind of neoreactionary ideas peddled by his friend. If the narrative of the decline of the angels is that of the spiritual gradually exhausting itself in the everyday, My Struggle seems animated by the desire to reawaken it, to infuse higher meaning into his own, ordinary life. The Hitlerian provocation of the title is thus more than a flippant joke, since it seems (at least from these volumes) that the author wishes to drink from the same wells of Nordic tradition as his predecessor. How far Knausgaard wishes to go with such flirtations will, I suppose, become clear as further volumes are translated. As a rootless cosmopolitan who has reason to be suspicious of appeals to authenticity, this is where I part company with his struggle, though not with his work, which I will continue to read – and argue with.

Interviewed by the Paris Review, Knausgaard told a story about a famous poet who was asked by a journalist whether, escaping from a burning building, he would take "the Rembrandt or the cat". He chose the cat. "I would do that too," said Knausgaard, but it is far from clear whether that is true. In a radio interview with his ex-wife, Tonje Aursland, he admitted he had made a Faustian bargain. "I have actually sold my soul to the devil. That's the way it feels. Because in addition I get such a huge reward." Aursland agrees with this assessment. "He has made a very clear choice, that this book is more important than having … a good relationship with family or a working relationship with his ex." There is no doubt that My Struggle has been bought at a price, both for the author and his loved ones, but for the reader, there is the thrill of confronting a passionate writer who is prepared to take such risks, to try to show his face as it is, rather than as he wishes it were.