Taking in folklore, history and the world’s first novel, here is some of the best reading about an endlessly inventive country

Japan has a birthday this year. It’s 150 years since rebel samurai overthrew the old Tokugawa Shogunate, marched – or, rather, palanquined – a teenage emperor into the newly named city of “Tokyo”, and made him their figurehead as they set about transforming their country. Western warships had recently been menacing Japanese shores, not so much offering friendship as insisting on it at the point of a gun. If Japan’s new leaders were to avoid becoming next on colonialism’s to-do list, a rapid programme of modernisation was called for: factories and weapons; mines and offices; trains, trams, trade.

How do you persuade a population used to thinking in regional rather than national terms, and who have next to no idea who you are, to cooperate in all this? To pay taxes, to join your army, to send their children to new national schools? One way is to tell stories. About Japan as a place especially blessed, perhaps even by the gods. About a country destined one day to become a beacon of modernity in Asia – if only people would put the effort in now.

In Japan Story, I set out to trace the extraordinary influence of these two tales in shaping modern Japan and its image around the world, across a tumultuous century-and-a-half. I wanted to explore, too, the fascinating range of alternative stories that people in modern Japan have told about their country: visions of what they hoped it might become, playing out across politics and music, art and philosophy, family and work, dance and religion, literature, folklore and film.

Here are 10 books that offer a taste of this rich and plural, endlessly inventive place:

1. Legends of Tōno by Kunio Yanagita

Collected by Japan’s first folklorist in the early years of the 20th century, these are traditional tales of the strange, the supernatural and the monstrous, told by people from the northern village of Tōno. Yanagita worried that the corruptions of the modern city – from office drudgery to an unpleasant me-first individualism – would soon claim these rural people too, so he wanted to capture their way of living and relating to the world before it was too late.

2. Kokoro by Natsume Sōseki

Often billed as Japan’s answer to Charles Dickens, Sōseki was a shrewd, sophisticated chronicler of his country’s early dealings with the modern west. He claimed that anyone trying to live a civilised life amid Japan’s hasty and superficial attempts to play industrial catch-up would inevitably lose their minds. Which he himself did, while studying in London. Sōseki poured all his angst and insights into his great psychological novel about “The Heart of Things”: the story of a group of Tokyoites caught between the old ways and the new.

3. Rashōmon and Seventeen Other Stories by Ryūnosuke Akutagawa

Akutagawa was already a star author when he took his own life in 1927, at the age of just 35. The “vague anxiety” about the future that he described in his suicide note seemed later to mark a tipping point for Japan: from an era of trial-and-error democratisation and cosmopolitanism into something darker and more inward-looking, leading eventually to terrible conflict. This collection features an excellent introduction to Akutagawa and his times by a star author of a later era: Haruki Murakami. More importantly, it features the short story Spinning Gears: a terrifying (self-)portrait of a man at the end of his tether: rifling through bookshop shelves “like a compulsive gambler”, riding Tokyo trains and taxis back and forth, trying to make life tolerable a little while longer. Not much longer, as it turned out. Akutagawa died soon after finishing this final story.

4. The Tale of Genji by Murasaki Shikibu

The observational style used by Sōseki and Akutagawa, sometimes painfully acute, owed a great deal to a tradition in Japan going back almost a millennium: to Sei Shōnagon’s The Pillow Book and a little later to The Tale of Genji – seen by some as the world’s first novel. Written by a lady-in-waiting at Japan’s 11th-century imperial court in Heian (modern Kyoto), it tells the story of an impossibly perfect prince, offering along the way a series of razor-sharp observations on the psychological foibles and social failings of those around him.

5. Kyoto: A Cultural and Literary History by John Dougill

“City of Genji” has its own section in this excellent guide to Kyoto: the ideal blend of history, culture, religious practice and belief, architecture, and the everyday. Dougill has a knack of throwing in comparative western references at just right the moment, from Geoffrey Chaucer to what was happening in Europe when some big event in Kyoto’s history came to pass.

Facebook Twitter Pinterest A notoriously difficult genre … Toshiro Mifuner and Richard Chamberlain in the 1980 TV adaptation of James Clavell’s Shogun. Photograph: Cine Text/Allstar/Sportsphoto Ltd

6. Shogun by James Clavell and The Shogun’s Queen by Lesley Downer

Two fabulous examples of a notoriously difficult genre, featuring as a joint entry here because they tell the story of Japan’s first Shogun and one of its last. Clavell traces the journey of an English sailor in late 16th-century Japan as he becomes part of a feudal lord’s bid for control of the whole country. It is based on the friendship of William Adams with Tokugawa Ieyasu, the first of the Tokugawa Shoguns. Downer explores the power of women in shaping the final years of the Shogunate, with her take on the story – enormously popular in Japan – of Atsuhime: a young samurai girl from south-western Japan who ends up at the very centre of the action in Edo (modern-day Tokyo) in the 1850s, as foreigners start to crowd around and the world begins to fall in.

7. Embracing Defeat by John Dower

One of the classic works of modern Japanese history. A vivid, all-encompassing account of a country picking itself up off its knees in the wake of the second world war. Dower traces everything from epic destruction – the aftermath of Hiroshima, Nagasaki, and the firebombing of Tokyo – down to the everyday inventiveness of starving people. And he explores the US’s radical attempt to recreate Japan in its own image, during the years of occupation from 1945 to 1952.

8. A Tokyo Romance by Ian Buruma

A memoir of Japanese counterculture in the 1970s, by someone who experienced it as an impressionable and often rather overwhelmed young man. Buruma hangs out with a theatre troupe trying to push against the plush, hushed soullessness of modern kabuki performances, returning instead to the itinerant “riverbed beggar” tradition out of which it first grew. The legendary choreographer Tatsumi Hijikata, creator of Ankoku Butō – the “Dance of Darkness” inspired by Japan’s shamanic tradition – is delighted to discover that this young man’s name sounds a lot like “bloomers”. He insists on calling him “Pants” thereafter.

Top 10 books to help you survive the digital age Read more

9. Dogs and Demons: The Fall of Modern Japan by Alex Kerr

Frustration, anger, and incredulity course through this powerful book by one of the best-known western critics of late 20th-century Japan’s construction boom: propping up an ailing economy by way of enormous and, for Kerr, largely unnecessary infrastructure projects. Roads to nowhere and bridges to uninhabited islands; sterile concrete tetrapods littering what ought to be beautiful beaches. Kerr has since found success as a restorer of traditional homes, bringing in tourism to help revive parts of rural Japan that had been on the verge of dying out.

10. Strange Weather in Tokyo by Hiromi Kawakami

To finish: a contemporary love story, between a woman in her late 30s and her old high-school teacher. Set in one of Tokyo’s numerous small bars, the drama is marinated in beer, saké, miso soup, humour, poetry, and wonderfully warm, comforting conversation.

• Japan Story: In Search of a Nation by Christopher Harding is published by Penguin, priced £25. It is available from the Guardian bookshop for £22, including free UK p&p.