It’s not difficult to spot Karl Ove Knausgaard waiting at Ystad train station. Partly, it must be said, because there’s no one else present at the open-air platform on a brilliantly cloudless February afternoon in southernmost Sweden. Yet even if there had been a large crowd, Knausgaard is the sort of man who would stand out.

He’s 6ft 4in with a strikingly handsome face, a writerly beard and a thick mane of swept back silvery hair. But it’s not that he’s a commanding presence so much as a conspicuous absence. There’s an air of separateness about him, something noticeably aloof and withdrawn. He doesn’t look like the sort of person, in other words, who would readily divulge his innermost secrets, desires and insecurities.

Yet that is precisely what he did in writing a six-volume, 3,600-page novel-cum-autobiography, provocatively titled My Struggle – or in Norwegian Min Kamp. In Knausgaard’s native Norway, where the book was published between 2009 and 2011, it was an unprecedented phenomenon, selling half a million copies in a nation of 5 million people. It has subsequently been translated into 22 languages. In English, the fourth volume, Dancing in the Dark, is about to be published. And given the size of the undertaking, the widespread critical acclaim and cultural buzz the series has generated, it has strong claim to be the great literary event of the 21st century – so far.

Although the series is ostensibly fiction, it is also an unflinching memoir, from early childhood right up to the controversial reception of the book itself. Such an endeavour is hardly unique. Bookshops are full of unflinching memoirs, and even the literary novel is no stranger to the genre. Knausgaard’s portrait of his cold, authoritarian father, who drank himself to a squalidly premature death, may have scandalised the reading public of Norway, but it’s a tale of mild familial dysfunction by comparison with, say, Edward St Aubyn’s autobiographical Patrick Melrose series.

So what is it that has led fellow authors like Zadie Smith and Jonathan Lethem to rave about Knausgaard and hail him as literary pioneer? Why did the novelist Jeffrey Eugenides speak of the Norwegian breaking “the sound barrier of the autobiographical novel”?

Knausgaard in his writing studio: ‘I get so tired of my own voice.’ Photograph: Katherine Anne Rose/Observer

The answer lies not in Knausgaard’s depth of revelation so much as the intensity of focus he brings to the subject of his life. He seems to punch a hole in the wall between the writer and reader, breaking through to a form of micro-realism and emotional authenticity that makes other novels seem contrived, “made up”, irrelevant. As Smith put it: “You live his life with him. You don’t simply ‘identify’ with the character, effectively you ‘become’ them.”

Whether he’s writing about his adult alienation at a toddler’s birthday party or the memory of trying to get hold of alcohol as a teenager on New Year’s Eve, Knausgaard is prepared to go into extraordinary sensuous detail that can last 50 pages or more.

The most obvious antecedent is Proust, whom Knausgaard has said he “virtually imbibed” when A la recherche du temps perdu was finally translated into Norwegian in the 1980s. But the prose has little of Proust’s delicate refinement. At times it’s poised, beautiful, profound, and at others mundane and casually tolerant of cliches. People pick up and put down coffee cups a lot, close doors and light cigarettes. There’s a great deal of the sort of writing that is usually edited out before publication, and yet the overall effect is utterly hypnotic. As the critic James Wood put it, even when he was bored he remained interested.

It’s not to everyone’s taste. Writing in the Nation, the critic William Deresiewicz complained: “Instead of thinking about the character, I was thinking about the author, and the fact that they were the same individual only made it worse. Who cares? I kept wondering. Why is he telling me this? Who is he to think his life is worth this kind of treatment? I wasn’t just bored (even his fans are bored), I was angry about being bored. I felt my time was being wasted.”

But to see in Knausgaard’s epic bildungsroman the evil of banality is to miss the point. Cyril Connolly declared that the enemy of promise was the “pram in the hall”. Knausgaard, feeling trapped by young children, bravely confronts the enemy and then recruits it to the cause of what Connolly called “good art”.

His digressions, chronological shifts – time washes back and forth like an unpredictable tide – and forensic observations of the everyday are an astonishing effort to capture the vast mystery of consciousness through the techniques of a novelist. At any given time our selfhood is the sum of our memories and Knausgaard understands that this means the throwaway ore as well as the precious metal.

What, though, are memories? How accurate and reliable are they? Knausgaard makes several mentions of his poor memory during the course of the thousands of pages of precisely rendered recollections. And it’s not false modesty. This is less a work of formidable retention than concentrated invention.

To do something good you have to step out of society, almost out of humanity, if that's possible

“For me,” he says, “there has been no difference in remembering something and creating something. When I wrote my fictional novels they always had a starting point of something real. Those images that are not real are exactly the same strength and power of the real ones and the line between them is completely blurred. When I write something, I can’t remember in the end if this is a memory or if it’s not – I’m talking about fiction. So for me it’s the same thing. It was like I was writing a straight novel when I was writing this but the rule was it had to be true. Not true in an objective sense but the way I remember it. There’s a lot of false memory in the book but it’s there because it’s the way it is, it’s real.”

We’re all familiar with the idea that fiction can achieve a greater truth, but in the real world, regardless of differing perceptions, either something happens or it doesn’t, and where and when it happens are similarly objective facts. The actual truth sometimes matters. Did he ever begin to question his version of events?

He says the manner of his father’s death was the subject of bitter dispute, which caused him to doubt his own memory. Some family members insisted that Knausgaard’s father, contrary to his son’s recollection, had not been surrounded by bottles of alcohol when he died, there was nothing unseemly about the scene, and he had peacefully suffered a heart attack.

“I began to doubt everything,” he says. “Did I exaggerate it? Did I exploit it for my own benefit? Because I’ve done that in my life before. I used to talk about my father to make myself more interesting. And I knew that the shocking details in the story are good literature in themselves almost – it’s a good story.”

Then he received a letter from a health worker who, reading the first volume A Death in the Family, recognised the house Knausgaard’s father died in, and got in touch to confirm the accuracy of his description.

“And that,” he says, “was such a relief.”

Knausgaard lives with his second wife, the poet and novelist Linda Bostrom, and their four children (aged one to 11) in a small village 10 minutes’ drive – in a downbeat old van – from Ystad. He tells me they moved there because it had a good school. There’s a pause before he adds: “It closed down the year after we came here.”

It’s a typically wry observation. He does a good line in self-deflation, but there’s nothing insincere or ironic about him. For all his ability to laugh at himself, he is a serious presence both on the page and in person, someone, you feel, who is acutely aware of the existential tensions between life and art.

He met Bostrom at a writer’s conference while he was still married to his first wife, the journalist Tonje Aursland. He made a pass at her, which she rejected, and, in a drunken state of demoralisation, he deliberately cut up his face with broken glass. He later left Aursland and moved to Sweden, but she only learned of the initial episode with Bostrom when she read the second volume, A Man in Love, where it is recorded with characteristically scrupulous candour. Deeply upset, she made a radio documentary in which she confronted Knausgaard.

As uncomfortable as that experience was, still more troubling was his family’s reaction to the first volume, a mordant reflection on the ignominious demise of Knausgaard’s father. An uncle threatened to sue, and one side of the family stopped talking to the author.

It had been Knausgaard’s four-year failure to write a fictionalised account of his father that led him to embark on his autobiographical outpouring. He declared himself nauseated by the very idea of fabricated plots and characters. Again the Norwegian is not alone in that sentiment. Many writers have expressed their belief that in the age of information saturation, made-up stories have lost their power. In HHhH, his novel about the assassination of Reinhard Heydrich, Laurent Binet is positively disgusted by the prospect of committing any form of embellishment or factual inaccuracy.

But Knausgaard took this distaste for the unreal and the manipulated to new extremes. He abandoned a careful approach to prose and structure, and wrote in a blind creative frenzy, producing upwards of 20 pages a day. John Updike, who rummaged through his own life for fictional source material, used to speak of writing as “unpacking” his bag. With Knausgaard it was more like upending the contents and shaking them on to the page.

Karl Ove Knausgaard (left) with his father and older brother Yngve, circa 1970. Photograph: Courtesy of Karl Ove Knausgaard

“My initial thought was empty everything,” he says. “Put everything out there so there is nothing left, which of course is impossible.”

He sat at his keyboard in what he calls an “autistic” state, oblivious to the impact his words would have on the world, and in particular those close to him. The first two volumes were written together and published before he began the third. But such was the pain that the first two inflicted on family and friends that he pulled back in the third, fourth and fifth. In the sixth he returned to full disclosure, cataloguing the breakdown his wife, Linda, suffered during the fall-out over the first two volumes of My Struggle.

If he was more restrained about others in the middle volumes, about himself he remained completely unsparing throughout. In Dancing in the Dark, the latest to be translated, he documents his prolonged and calamitous attempts to dispose of his virginity during his late teenage years, a struggle sorely undermined by a chronic condition of premature ejaculation.

It’s very funny but also excruciatingly exposing. I ask him if, having detailed his father’s physical humiliations (including incontinence), it made it easier to address his own.

“No,” he says firmly, “I didn’t make that connection. I read everything to a friend while I was writing. I’d never told anyone about my sexual shortcomings, not girlfriends of course, not my brother. It was a complete secret. And I read it to my friend and he just laughed and didn’t stop laughing. And I thought, OK, it’s out. It can’t be any worse than this.”

Knausgaard presents an oddly vulnerable figure. A timid child who tiptoed around his brooding father (as described in the third volume, Boyhood Island), he grew into a cripplingly self-conscious teenager, and then, by his own reckoning, an adult who is seldom happy in the company of others. When I tell him that, for all his stated insecurities, he looks like the very picture of strapping Norse masculinity, he replies drily.

“Yeah, but I don’t behave like it. That’s the problem.”

Karl Ove Knausgaard (right) with brother Yngve in Bergen, circa 1990. Photograph: Asbjorn Jensen

The issue of masculinity is a recurring theme in My Struggle. He says he formed his views on male identity as a child growing up in the 1970s. He was teased by the other boys for being a “jessie” or gay.

“It put such doubt in me that I’ve never really recovered from it,” he says. “I have all these notions of what it is to be a man. You shouldn’t cry for instance [he spends many pages crying or trying to conceal his tears] and you shouldn’t talk about feelings. I don’t talk about feelings but I write a lot about feelings. Reading, that’s feminine, writing, that’s feminine. It is insane, it’s really insane but it still is in me.”

In Sweden, he says, there is a widely held conviction that gender identity is socially constructed rather than biologically determined. You’d think that for a such a lachrymose man, riven by longstanding doubts, his adopted home would be an ideal refuge. Yet he can’t help but reject Sweden’s social idealism, and even hanker after the tormenting certainties of his youth.

“Those small things like giving a hug to man,” he says, almost wincing, “I try to avoid it. Because I can see the situation is coming and I try to prepare. But I remember the first time I did it, I was 16 and I was at the gymnasium and it was a cosmopolitan thing, an international thing, a modern thing, but I never felt at ease with it at all.”

It’s this sort of comment, he acknowledges, that has led to a misunderstanding among some readers and critics. “I’m not saying what I think people should do. People should do whatever they like. I really don’t care about that whatsoever. I’m just writing about my own discomfort in the culture. But I think there are a lot of men who have read this and felt like I’m expressing something they can relate to.”

Knausgaard’s home is a quadrant of low-slung, pretty but unassuming buildings, one of which – his study – is really a glorified garden shed. It’s divided into two book-lined rooms. In one there is a drum kit, guitar, bass guitar and amps. In the other, an armchair and sofa on which we sit. With their mountains of unwashed coffee cups and other detritus, both rooms display a reassuring indifference to order. He apologises, explaining that he spent “about an hour” that morning tidying up for my visit.

This scene – the acclaimed writer in his rock’n’roll lair discussing his latest work with a member of the international press – is doubtless one about which the young Knausgaard dreamed. The book is full of longing – for escape, experience and recognition – as well as paralysing inhibition and vaunting ambition.

At one stage in Dancing in the Dark he reacts to his brother’s dismissal of his authorial aspirations with an internal outburst: “I’ll bloody show the whole sodding fucking world who I am and what I am made of. I’ll crush every single one of them. I’ll render every single one of them speechless. I will. I will. I bloody well will. I’ll be so big no one is even close. No one. No. One. Never. No bloody chance. I will be the bloody greatest ever.”

But every escape contains its own confinement, and the escape from obscurity to a global reputation has brought unwanted attention and preoccupying commitments to the business of bookselling. He cancelled a promotional trip to the UK earlier this year. Was he beginning to feel the weight of his phenomenon imprisoning him all over again?

He pauses and lights a cigarette.

“Like my friends say, I’m crying in my limo,” he jokes, letting out a bashful smile. “No, it’s like this. When this massive breakthrough came in Norway it was completely unexpected. And then it was like a dream. It was very difficult on many levels but basically it was what I wanted as a writer.”

Knausgaard has been through several stages of adjustment to the attention and success the book has brought. The first was dealing with the familial repercussions. He has a strong moral instinct, someone who thinks of himself as a man who tries to do the right thing. Yet in hurting those close to him, he couldn’t help but believe that he had done the wrong thing. The compensation – and it was considerable – was writing something that showed the whole sodding fucking world who he was and what he could do.

I think shame is an essential mechanism in social life. It makes people behave in an appropriate way

However, the fact that so many people did indeed take notice occasioned the next stage of anxiety.

“We were raised to believe that if many people like something, it’s not good,” explains Knausgaard, dragging on another cigarette.

His two books before My Struggle had been well received in Norway but the readership was limited to what he calls “people in the system”: students, academics, other writers and those with a particular interest in literature. With the six-part memoir, the world outside the system got involved.

He seems to have come to terms with the fact that popularity isn’t necessarily antithetical to quality. What bothers him now is the ceaseless call of book promotion and the self-promotion and self-analysis that demands.

“Now it’s different because I’ve been talking about it for five years. And I do not talk about books. I talk about myself. And that’s to be trapped in something, because it never ends.”

But after a short time in his company it’s clear that there’s much in the book that continues to pique his own curiosity. He answers questions not with terse reluctance but genuine intellectual engagement.

One aspect of Knausgaard’s writing that is distinctly unusual for a modern writer in western Europe is its constant grappling with the question of shame.

Time and again in the book, Knausgaard displays a talent for experiencing the maximum shame in even the most unpropitious circumstances. What to most people would be cause for a minor outbreak of sheepishness can lead Knausgaard to suffer agonies of mortification. Was he aware of the extremity of his responses?

“When I wrote my first novel my editor wrote the sleeve note and he called it ‘a monument of male shame’. And it had never occurred to me that I was writing about male shame. It was so much a part of me that I didn’t see it, didn’t recognise it as shame.”

Once he accepted the diagnosis, he began to explore its literary potential.

“Writing is a way of getting rid of shame. When you write the whole idea is to be free. And what are you free from? From people looking at you. I think shame is an essential mechanism in social life. It regulates everything and makes people behave in a decent and appropriate way to each other. But I have kind of too much, an overdose. I’m so restricted I can’t do anything.”

All of which makes it that much more remarkable that he was able to dredge up his sexual and moral failings for public delectation. He argues that when he’s writing, he’s not thinking about the reader. His focus is entirely on getting to the truth of his story. Authors often claim to write for themselves, but they usually have one eye on the reader, if only in terms of structuring their narrative, selecting when to release information for maximum impact.

There’s little sense of those ploys in My Struggle. Nor does Knausgaard intrude on the truth of the past with the hindsight of the present, even when it comes to the question of musical taste – a subject that is notorious for historical revisionism. We see the young Knausgaard proudly displaying his love of Simple Minds and other now unfashionable bands without any nods or winks from the author to the reader. Was that difficult?

“Yes it’s very hard to write from that point of view because of the vanity,” he admits. “In a book about childhood it’s obvious there is a difference between the writer and the kid, but when you’re 17, 18 it’s not so obvious anymore. I remember my editor said, ‘Can’t you once or twice make a reflection here?’ But I wanted to see if it was possible to identify with him at that age. That’s a risk because you are never so stupid as when you are 17. Never. When you are eight you’re not stupid. You’re many things but not stupid. But at 17 you are.”

As the drum kit and guitars testify, Knausgaard has not lost his passion for rock music. But as he got older, his interest turned increasingly to writing and writers. And his preferences, we learn, are the poets and thinkers who stem from the German Romantic tradition: the poet Holderlin, most obviously, but also it could be argued the philosophers Nietzsche and Heidegger. In them he seems to find a galvanising rejection of liberal complacency and post-enlightenment pieties.

In an essay on Knausgaard, the novelist Hari Kunzru suggested that this desire for the kinds of spiritual renewal propagated by these writers made the “Hitlerian provocation of the title… more than a flippant joke”. He worried that Knausgaard wanted to “drink from the same wells of Nordic tradition as his predecessor”.

In fact Knausgaard includes a 400-page essay on Hitler in the sixth and final volume. So was his affinity for German Romanticism merely aesthetic or was there something more ideological at play?

Knausgaard in Bergen, circa 1989. Photograph: Asbjorn Jensen

“I’m certainly interested in it but I also recognise that if you take these things to the bitter end you end up in Germany in 1934,” he says, reaching for another cigarette. “And that’s why I was writing about it in the last volume, to find out what’s good in Nazism and what’s not. Which is a kind of refreshing take on it. There is a longing for something not contemporary in me, and in many people of course. But I have a completely anti-ideological point of view. Everything in ideology is the opposite of a novel, which to me is the anti-totalitarian language per se.”

The phrase “what’s good in Nazism” would be enough to set alarm bells ringing in many readers, but Knausgaard is making a subtle and important point. The Germans – and Norwegians – who waved flags in support of the Nazis didn’t see themselves as followers of genocidal sadists. They thought that they were part of something larger than them, a community, a movement, a happening.

In the final volume, Knausgaard says, he compares this feeling to that felt by Norwegians when they came out into the streets to protest against the attacks by the ultra-nationalist terrorist Anders Breivik, who killed 77 Norwegians in 2011.

“One hundred thousand people are out there, everyone is united. I said if you want to think about Nazism, what that was, you must think of it like this. It was something good, we are out on the streets for something good. The timing for saying that was not so good,” he concedes, “but I still think it is like that.”

Knausgaard says he broke down crying when he heard about what Breivik had done. Exiled in Sweden, he experienced a powerful solidarity with his fellow Norwegians.

“I felt so strongly identified with a kind of ‘we’ and I had never felt that before. This has happened to ‘us’. And it was so strong. And I’m very interested in those strong forces in society. To be a good writer you must be out of the ‘we’. You can’t be a part of it.”

Knausgaard found it particularly disturbing that Breivik was roughly from his generation, and what’s more that in his manifesto there were some echoes of his own thinking.

“I could see the train of thought in the way that he criticised certain things that I criticise. For instance, the feminisation of society.”

It wasn’t until after he’d finished writing about Breivik that he realised that he formed a coherent part of the thesis of My Struggle.

“The feeling of lack of presence, lack of physicality, the feeling that the world is disappearing into images and stories and his description of what he experienced on the island [Utoya]. It’s like he’s not in this world, it’s all images, no bodies just pixels. He turned away from the world to do that, I think for three years, only by himself, playing games on his computer, completely antisocial, and then he could do it.”

In a way, of course, Knausgaard also turned away from the world. “To do something good you have to step out of society, almost out of humanity, if that’s possible,” is how he has put it. Alone in his study, he was immune to the consequences of his actions, of his writing. He believes that the description of his intimates in his book amounted to an “unmoral” act. And he has even compared apologising for it after publication to killing someone and then saying sorry.

But he did not kill anyone, and nor did he disappear into a deranged fantasy of his own making. He wrote his autobiographical novel to connect with reality, his own reality, it’s true, but one that speaks to a great many other people.

His struggle was a tremendous act of recovery. He reclaimed not only the story of his life, but a sense of artistic purpose, and not incidentally a book title that had been for too long sullied by history’s most vile tyrant. In the process, however, his writing became overwhelmed by its subject, or rather the means of dealing with the subject.

He announced at the end of volume six that he would “enjoy, really enjoy, the thought that I am no longer an author”. But, he tells me, he has been writing ever since. Last summer he co-wrote a diary of the World Cup, a correspondence with the writer Fredrik Ekelund, in which he muses on his affection for the cynical approach of Argentina – for a long time, “Argentina” was the planned title of My Struggle.

“It’s written in the same style [as My Struggle],” he says of the World Cup book. “I write essays like that, and long articles for the newspaper in the same way. What I have to do is get out of that way of writing. I get so tired of my own voice. It’s like it’s producing something like a machine. I need to get somewhere else. I can’t keep using that technique. So every day I try to write one page about one thing, and I will do so for a year.”

That’s how Knausgaard intends to write his next novel, one day at a time. He has shown the world what he’s made of. And now, like all great artists, he must show us all over again. In a cluttered study in a small village near Ystad, the struggle continues.

Dancing in the Dark is published by Harvill Secker (£17.99). To order a copy for £14.39 click here