"Human life is limited but I would like to live forever." So Yukio Mishima wrote on the morning of 25 November 1970. By noon, he was dead: eviscerated and decapitated, by his own choice if not entirely his own hand. Flanked by young adherents, he had penetrated the Tokyo headquarters of Japan's Eastern Army, made a wild and whirling speech exhorting the troops to impose martial law in the emperor's name, and then performed ritual seppuku. Mishima was Japan's most famous writer, but henceforth nothing would define his life quite so much as the leaving of it.

Before quitting his opulent home that November day, Mishima sealed and posted to his publisher the manuscript of The Sea of Fertility, a tetralogy of novels over which he had laboured for five years. If he had meant the work to be his magnum opus, it was doomed to be occluded by his suicide. These four books – Spring Snow, Runaway Horses, The Temple of Dawn and The Decay of the Angel – are a saga of 20th-century Japan: a story of national decline that nonetheless proposes redemption through the endurance of a certain soul, forceful enough to be reborn ad infinitum.

Evaluations of Mishima have long been clouded by ambiguities: he was an inwardly divided man who divided public opinion accordingly. His was a life of ceaseless self-reinvention – a dutiful son and law student who became, aged 24, the scandalising author of the homoerotic Confessions of a Mask; a wan aesthete who bulked himself up into a muscle-man poseur; a homosexual man who married and had children; and, through the 1960s, a vociferous advocate of an honour-bound imperial Japan who was also a media showboater with gaudy "western" tastes.

As a writer, Mishima offered dazzling performances, finical executions of ideas. His reputation-making works – The Sound of Waves (1954), The Temple of the Golden Pavilion (1956) – were highly polished if somewhat minor-key masterpieces. With The Sea of Fertility (named after a vast basalt-dark lunar plain once imagined to be a body of water), he chose a broad canvas like his hero Thomas Mann. Therein he would exhibit all his gifts – an eye for detail and scene-making, a sensuous regard for the physical, and a cool detachment that could be terrifying in its terseness. The usual red thread through a generational saga is, of course, a single family. But with trademark boldness, Mishima opted for the transmigration of a soul – the device expressing his own avidity for life, one that physical existence could never fully requite.

The tetralogy opens with Spring Snow. It is 1912, Meiji Japan giving way to "Taisho democracy", the milieu that of a fading aristocracy resigned to westernisation (shades of Lampedusa's The Leopard in its sense of how sweet was life before the revolution). Adolescent law student Shigekuni Honda is a stolid sidekick to his capricious friend Kiyoaki Matsugae, a baron's son of distant samurai descent. While Honda's dutiful future seems preordained, Kiyoaki is a dreamer, gripped by the sense that life's elusive fineness is slipping away by the second. He longs to chase the impossible, to "bend the world" into the shape of his ideals.

Kiyoaki's desires eventually alight on the beautiful Satoko Ayakura. Though friends since childhood, they have grown into a mutual indifference. Yet when Satoko gets engaged to a prince, Kiyoaki is suddenly consumed by passion, and the two fall into an illicit affair that proves the undoing of them both. Honda accepts the mantle of go-between for the lovers, but can only watch as Satoko renounces the world for a remote, wintry nunnery – Kiyoaki drives himself to pneumonia in a hopeless effort to retrieve her. Dying, clutching Honda's hand, Kiyoaki murmurs that they will meet again someday, "beneath the falls". On this sorrowful note, Spring Snow concludes.

Book II, Runaway Horses, finds Honda in 1932: an appeals court judge locked in the routines of career and an arid marriage, dully certain that his savour of life died with Kiyoaki. One day his duties take him to a kendo tournament in a shrine town, where he is struck by the sinuous ferocity of a young fighter, Isao Iinuma (son of Kiyoaki's former tutor). Later, showering alongside Isao beneath a waterfall, Honda spies a trio of dark moles on the youth's left flank, identical to a pattern on Kiyoaki's body. This is all the evidence Honda requires to accept that his friend, as if by magic, is returned in a new skin; that "passionate love of years past" has earned rebirth as "passionate dedication in the present".

The enchantment soon darkens, though, as Honda learns that Isao is a fledgling fanatic, inspired by Meiji-era samurai morals, who holds capitalism responsible for Japan's modern degeneracy and has formed a cabal that has plans to assassinate members of the country's financial elite (zaibatsu). The plot is designed as a suicide mission yet Isao exults in this, certain that true samurai fall like cherry blossom even amid the gore of self-disembowelment. However, as the fateful day nears Isao is betrayed and arrested. Honda resigns from the judiciary to defend the boy, telling himself he is shielding "Kiyoaki" from a second early death, though at heart he envies Isao's burning ardour. (Here as always Mishima is dispassionately brilliant in noting ignoble human impulses – the twinge of jealousy, the flicker of disgust.) Honda wins the case but finally fails to divert Isao from his pathological course of suicide-murder.

Runaway Horses is disturbing material, also a harbinger of Mishima's own act of "patriotic" self-slaughter. Yet the novel offers a critique of Isao's zealotry. This hyper-masculine, virginal youth is shown to be haunted by femininity and begins to feel a yearning for more intimate relations with the world. When a female admirer gives him his first kiss he is "startled by the sweetness". Later, in prison awaiting trial, he dreams of being transformed into a woman: having steeled his body into a weapon he now feels himself "filled with a mist of soft, languid flesh". Through these strange, elegant, erotic passages, Mishima prefigures the next incarnation of Kiyoaki: having explored his inner fascist, he turns – for Book III – to his inner female.

The Temple of Dawn first locates Honda in 1940, touring Thailand on business, there encountering seven-year-old Siamese princess Ying Chan – known as "Moonlight" and believed to be mentally ill. Perhaps unsurprisingly, Honda is barely surprised when the princess seizes his leg and cries, "Mr Honda! How I've missed you!", insisting that he take her back to her native Japan. By now, Honda has ruminated extensively over theories of metempsychosis – indeed The Temple of Dawn is weighed down by Mishima's frequent résumés of Honda's learning, suggestive perhaps of an author's struggle to convince himself.

Come 1952, Honda is retired in a fancy villa near Mount Fuji. But a pall of dissipation hangs over his existence, as it does over defeated post-second world war Japan. More ruefully conservative than ever, Honda accepts the fallen state of his world but remains sure that the regenerated Kiyoaki "grasped the splendour of the unseen on the other side" – an ecstasy Honda knows he cannot share. Whereupon "Princess Moonlight" sashays back into his life – fully grown, sweet-natured and voluptuously lovely. But Mishima doesn't take us inside Ying Chan's mind as he did with Kiyoaki and Isao. Rather we see her through Honda's eyes – see, indeed, a great deal of her, for Honda, ostensibly eager to locate Kiyoaki's three moles on "the dusky sky of her brown skin", has felt his sterile heart beat harder in Ying Chan's presence. She is one more embodiment of all that Honda cannot possess – what Marguerite Yourcenar, in a superb essay on Mishima, described as "youth successively incarnated in the most ardent, the hardest, or the most seductive forms".

Honda's desire for Ying-Chan turns to debacle and he will be far from her side when she, too, meets a preordained death aged 20. A pervasive aura of desolation carries the reader into the tetralogy's final instalment, The Decay of the Angel, a work of pitch-black pessimism that sees the Japan of 1970 nearly consumed by a rising tide of modern detritus. The septuagenarian Honda places his faith in one more seeming renewal of Kiyoaki – an arrogant, ingrate youth named Toru – but in the process he is harried and abused almost to death, so leaving him to wonder whether he ever read the augurs correctly, or has been living an epic delusion all along.

The Sea of Fertility concludes gracefully with Honda making a bone-weary pilgrimage to Gesshu nunnery, hopeful of an audience with Kiyoaki's lost love Satoko. This scene does much to redeem the otherwise rushed quality of The Decay of the Angel: Mishima was certainly distracted in the writing, done mostly during a seaside family holiday in August 1970 when he was already making secret plans to die his own excruciating death, much in the manner of his fictional Isao.

Why did Mishima do it? Evidently he did feel ashamed of the Japan of Coca-Cola and MacArthur's constitution that he painted so critically in The Sea of Fertility. But his drive to death had deeper roots, exposed by the tetralogy's nihilism – its sense that beauty is both life's raison d'être and proof of its futility. The calculated air of Mishima's art, its blade-edged chill and bleakness, makes him for many a hard writer to love. He offers few consolations. But his best work, unnerving as it may be, still casts a spell; and I suspect it will retain its dark radiance.

• Richard T Kelly's novel The Possessions of Doctor Forrest is published by Faber.