'Strange things happen in this world," Haruki Murakami says. "You don't know why, but they happen." It could be a guiding motto for all of his fiction, but he is talking specifically about a minor character in his new novel, Colorless Tsukuru Tazaki and His Years of Pilgrimage. The character is a jazz pianist who seems to have made a pact with death, and is able to see people's auras.

"Why that pianist can see the colours of people, I don't know," Murakami muses. "It just happens." Novels in general, he thinks, benefit from a certain mystery. "If the very important secret is not solved, then readers will be frustrated. That is not what I want. But if a certain kind of secret stays secret, it's a very sound curiosity. I think readers need it."

The world's most popular cult novelist is sipping coffee in the sunny library of an Edinburgh hotel, which – perhaps disappointingly for admirers of his more fantastical yarns – is not reached through a labyrinthine network of subterranean tunnels. Murakami is relaxed and affable, rather than forbiddingly gnomic. "I'm not mysterious!" he says, laughing.

Tsukuru Tazaki, as the author calls his own novel for short, sold a million copies in two weeks when it came out last summer in Japan. (Murakami was born in Kyoto to two literature teachers, and grew up in the port city of Kobe. These days he lives near Tokyo, having spent periods in Greece, at Princeton and Tufts universities – where he wrote his masterpiece, The Wind-Up Bird Chronicle – and recently in Hawaii.) It contains passing mysteries like the pianist who sees auras, but it is also a mystery novel in a larger sense. Tsukuru, its 36-year-old protagonist, is still in mourning for the years before he went to university, when he was part of an inseparable group of five friends – until one day they told him, without explanation, that they never wanted to see him again.

"In the first place I had the intention to write a short story," Murakami says. "I just wanted to describe that guy, 36 years old, very solitary … I wanted to describe his life. So his secret was not to be dissolved; the mystery was going to stay a mystery."

But he hadn't reckoned on the inciting power of a woman to move the story forward, as Murakami's female characters so often do. "When I wrote that short-story part," he continues, "Sara, [Tsukuru's] girlfriend, came to him and she said, 'You should find out what happened then', so he went to Nagoya to see his old friends. And the same thing happened to me. Sara came to me and said, 'You should go back to Nagoya and find out what happened.' When I was writing the book, my own character came to me and told me what to do … The fiction and my experience happened at the same time, in parallel. So it became a novel."

Murakami has often spoken of the theme of two dimensions, or realities, in his work: a normal, beautifully evoked everyday world, and a weirder supernatural realm, which may be accessed by sitting at the bottom of a well (as does the hero of The Wind-Up Bird Chronicle), or by taking the wrong emergency staircase off a city expressway (as in 1Q84). Sometimes dreams act as portals between these realities. In Tsukuru Tazaki there is a striking sex dream, at the climax of which the reader is not sure whether Tsukuru is still asleep or awake. Yet Murakami hardly ever remembers his own dreams.

"Once I talked to a very famous therapist in Japan," he says, "and I said to him that I don't dream much, almost nothing, and he said: 'That makes sense.' So I wanted to ask him: 'Why? Why does it make sense?' But there was no time. And I was waiting to see him again, but he died three or four years ago." He smiles sadly. "Too bad."

His novels thus far have generally divided into two types. There are the overtly magical-realist romances (A Wild Sheep Chase, The Wind-up Bird Chronicle, 1Q84), and the works on a smaller canvas, in which hints of the supernatural remain mostly beneath the mournful, mundane surface (South of the Border, West of the Sun; Sputnik Sweetheart). With its unresolved mysteries, tales-within-tales and maybe-dreams, Tsukuru Tazaki seems almost a hybrid of both styles. "I had been thinking my novels are divided into two categories, as you said," he agrees. "So it's just like Beethoven's symphonies, you know, odd numbers and even numbers. Three, five, seven, nine is kind of a big symphony, and two, four, six, eight is a kind of intimate work. I think my novels do the same thing. What do I think about this Tsukuru Tazaki? Yeah, it might be a new category."

Such musical comparisons come naturally to Murakami, who along with his wife, Yoko Takahashi, ran a jazz bar called Peter Cat in Tokyo in his twenties, which he opened while still a drama student at Waseda University. Murakami sold the bar and concentrated on writing full-time after the publication of his second novel, Pinball. Since then, his life has been one of writing and long-distance running – as chronicled in his memoir What I Talk About When I Talk About Running – and also collecting records. His novels almost always feature a thematic piece of music (his breakout Japanese bestseller, Norwegian Wood, was named after the Beatles song). The unusual harmonies of Thelonious Monk's "'Round Midnight" were perfect for this novel's haunted pianist, he thought: "Thelonious Monk's tune is full of mysteries. Monk plays some very strange sounds during the chords. Very strange. But to him it's a very logical chord. But when we are listening to his music it doesn't sound logical."

A still from the film version of Murakami's Norwegian Wood (2011).

Meanwhile, Tsukuru and his former friends listen to "Le Mal du Pays", a piece from Liszt's set of piano suites Years of Pilgrimage (hence the novel's subtitle). It was the soundtrack to the novel's composition. "I wake up early in the morning and I play a record, a vinyl record, when I'm writing. Not so loud. After 10 or 15 minutes I forget about the music, I just concentrate on my writing. But still I need some kind of music, good music. When I was writing Tsukuru Tazaki I was listening to Liszt, the Years of Pilgrimage, and that song, 'Le mal du pays', remained in my mind somehow, so I just wanted to write something about that song. That's a beautiful record." Listening to it, Tsukuru feels as if "he'd swallowed a hard lump of cloud".

The solitary adult Tsukuru works as a designer of railway stations. "There is a reason I'm interested in railway stations," Murakami begins to explain, not unmysteriously. It dates back to his early 20s, when he was looking for a good location in Tokyo to open his jazz bar. "I heard a certain railway company was rebuilding a station," he says. He wanted to know where the new entrance would be, so his bar would be near it. "But that's a secret, you know, because people are speculating." At the time Murakami was studying drama, but he went to the railway company and pretended to be a student of railways, befriending the man who was in charge of the rebuilding project. "He didn't tell me the new location of the entrance to the station. But he was a nice guy. We had a good time together. So when I wrote this book I remembered that episode.

"I have collected so many memories, in my chest, the chest of my mind," he says with satisfaction. "I think everybody has a lot of memories of his or her own, but it's a special gift to find the right drawer. I can do that. If I need something, I can point to the right drawer."

Tsukuru considers himself uninteresting, an "empty vessel", but Murakami almost can't help giving his hero an aesthetic sensibility. At one point, Tsukuru sees a chair in an office: "The chair was a simple Scandinavian design of chrome and white leather. Beautiful, clean, and silent, with not an ounce of warmth, like a fine rain falling under the midnight sun." So is Tsukuru really more interesting than he thinks he is? With the novelist's humane affection focused on them, might anyone be?

"I don't know," Murakami says. "I have many similarities to [Tsukuru]. I see myself as a kind of ordinary guy. I don't think of myself as an artist, mostly. I guess I'm just engineering something." A builder, like Tsukuru? "Yeah, right!" He chuckles. "I like to write. I like to choose the right word, I like to write the right sentence. It's just like gardening or something. You put the seed into the soil at the right time, in the right place."

That kind of engineering is exhausting, though: a daily trip to the "basement of the mind" and back up again. "You can say that it's a kind of unconscious, subconscious … you have to go down there and come back to the surface. You have to dedicate yourself to that work. You have no extra space to do something else."

Murakami's style is simple, even apparently casual, on the surface, and Tsukuru Tazaki, like many of his previous novels, has divided critics into those who find it banal and those who perceive greater depth in its vividness and precision of imagery. Like most simple styles, of course, his is the result of lots of hard work. "I take time to rewrite," he explains. "Rewriting is my favourite part of writing. The first time is a kind of torture, sometimes. Raymond Carver [whose work Murakami has translated into Japanese] said the same thing. I met him and I talked with him in 1983 or 84, and he said: 'The first draft is kind of torture, but when you rewrite it's getting better, so you are happy, it's getting better and better and better.'" There is never a deadline for a Murakami novel – "I don't like deadlines …when it's finished, it's finished. But before then, it is not finished." Sometimes he can't tell when he should stop rewriting, but "my wife knows. Yes. Sometimes she decides: 'You should be finished here.'" He smiles and imitates his own obedient response: "'OK!'"

Right now, Murakami is not writing anything. "After 1Q84," he says, "I was so exhausted … Usually when I'm exhausted by writing a big novel, I write a set of short stories. But not that time … I didn't have any strong energy to descend" – he mimes going down into the basement. "You have to be strong to descend into the darkness of your mind." But after finishing Tsukuru Tazaki, Murakami wrote six short stories in three months; they were published this summer in Japan, under a title meaning "Men Without Women". He might, he thinks, begin another novel next year. A long one, like an odd-numbered Beethoven symphony? "I think maybe a big book, yes."

One of Murakami's favourite writers … Kazuo Ishiguro. Photograph: Matt Carr/Getty Images

Asked to name some of his favourite writers working today, Murakami enthuses about Kazuo Ishiguro ("I think he dedicates himself to the writing … When he's not writing he goes around the world, but when he's writing he goes nowhere"), Cormac McCarthy ("always riveting"), and the Norwegian novelist Dag Solstad, whom he is currently translating into Japanese from English ("He's a kind of surrealistic writer, very strange novels. I think that's serious literature"). As he has translated Raymond Chandler, I ask him about modern crime writers too. "I like Lee Child," he announces decisively, and laughs. So do I, I say. "Oh you like him? That's good! So far I have read 10 of them." What do you like about them? He moves his hands in the air as though running his fingers over an invisible piano keyboard, and grins. "Everything's the same!"

Murakami doesn't read many of his Japanese contemporaries. Does he feel detached from his home scene? "It's a touchy topic," he says, chuckling. "I'm a kind of outcast of the Japanese literary world. I have my own readers … But critics, writers, many of them don't like me." Why is that? "I have no idea! I have been writing for 35 years and from the beginning up to now the situation's almost the same. I'm kind of an ugly duckling. Always the duckling, never the swan.

"But I think, in a sense, we are playing different games," he continues. "I began to think that way. It's very similar, but the rules are different. The equipment's different, and the fields are different. Like tennis and squash. " Does he think he'd be accepted if he won the Nobel prize, as many people now expect will happen? "Uh, I don't want to speculate," he says, and laughs. "That's a very risky topic. Maybe I would be hanged from a lamppost, I don't know!"

How long does Murakami think the game of literature can last? "I think serious readers of books are 5% of the population," he says. "If there are good TV shows or a World Cup or anything, that 5% will keep on reading books very seriously, enthusiastically. And if a society banned books, they would go into the forest and remember all the books. So I trust in their existence. I have confidence."

What would he still like to achieve for himself, as a writer? "Honestly, I don't have any idea," he replies. "Scott Fitzgerald was my idol when I was young. But he died when he was 40-something. I love Truman Capote, but he died at 50-something. And Dostoyevsky is my ideal writer, but he died at 59. I'm 65 right now. I don't know what's going to happen! So I have no role model. I have no idea – when I am 80 years old, what will I write? I don't know. Maybe I'm running and writing …" "That would be great. But nobody knows."

He says he tries to think of himself as a kind of craftsman, a tinker. "I'd like to be a perfect tinker. So I have to write good sentences – honest and beautiful and elegant and strong sentences."