One of the more surprising objects in the Bodleian Libraries’ fascinating exhibition of JRR Tolkien papers and memorabilia, “Tolkien: Maker of Middle-Earth”, is a letter in 1967 from a young man named Terence Pratchett, expressing admiration for Tolkien’s late story Smith of Wootton Major. Tolkien and Pratchett seem to stand at opposite ends of a literary spectrum; although Pratchett described Tolkien as being like Mount Fuji in Japanese prints, a ubiquitous background presence in all later “fantasy” writing, Middle Earth and Discworld are radically different creations.

Pratchett’s Discworld is funny, deflationary, bursting with self-consciousness and ironic ingenuity, stuffed full of puns, wordplays and parodies; Tolkien’s Middle Earth is intensely serious, and its verbal ingenuity is bound up with the monumental labours of its creator in shaping not only a set of actual languages but a linguistic history and historical geography for most of them and their speakers.

You could sum up the difference by saying that Pratchett is knowing – but for Tolkien, “knowingness” would unmake his whole project. As he made clear, part of his ambition was to provide something like a mythology for England; and mythologies cannot be knowing in this way, conscious of their literary pedigree. A myth is not a fantasy and Tolkien would certainly not have seen himself as writing “fantasy novels”.

The narrative of The Lord of the Rings and the “legendarium” of The Silmarillion and other writings are presented as a set of imaginative structures in and through which people can think and feel with the same consistency, intelligence and growing wisdom as they did through the stories of Olympus, Troy, Asgard or the Arthurian cycle.

This is not an entertainment. It has to take itself seriously – which explains why Tolkien had no time for his friend CS Lewis’s Narnia. Narnia was a world that characters could drop in or out of; it was cheerfully eclectic in its use of mythical and legendary raw material, and the stories were narrated with a good deal of the tongue-in-cheek waggishness of the great Edwardian children’s writers whom Lewis loved. For Tolkien, all this was embarrassingly trivial.

As he argued in his essay “On Fairy-Stories”, imaginative writing is to do with how human beings exercise their destiny as images of God, as sub-creators. Lovingly attending to a world unfolding out of the mind is one way of reflecting the ultimate and original act of imagining, which is the creation of the universe itself. It is a singularly bold statement of what any writer might seek to achieve, and it makes sense of Tolkien’s almost obsessive concern with teasing out the connections between the various bits of his imagined world, elaborating more and more fully the stories and records, the grammars and vocabularies, of his imagined societies.

The narrative of his major work is painstakingly detailed about landscape and weather, dates, phases of the moon and so on: he mapped his story on to an actual period in the real world, giving the events of The Lord of the Rings a clear chronology and thoroughly calculated travelling times. Using your creations either for entertainment or for edification would be to make them less than they could be, to show a sort of disrespect to what you had made. If the reader is looking for signs of Tolkien’s Catholic commitments in his work, this is where they are to be found, in a conviction about creation, rather than in any Lewis-like theological allusions.

****

The exhibition’s sumptuous catalogue has a couple of very good essays on the sources of both Tolkien’s imagined philology and on his creative use of themes from Scandinavian mythology in particular; and the exhibition displays numerous examples of the beautiful scripts he devised, and of his designs for Elvish heraldry and decorative arts and for detailed maps of Middle Earth. The stylised designs based on the flora of Middle Earth – his “moon-grass” and “wind-feather” – are especially beautiful. He evidently learned from his mother – some of whose handwriting can be seen here – many of the skills he used in his own scripts, from the fluent and regular hand he routinely wrote for daily purposes, through the lettering used in his “Father Christmas” letters for his children, to the scripts, Elvish, Runic and English, which he designed for the covers and title pages of The Lord of the Rings.

One of the insights afforded by this exhibition is how extensively Tolkien (1892-1973) worked as a visual artist, not only in depicting settings for his stories but more generally as a painter or draughtsman of archetypal and mythical landscapes. As a young man he had considerable skill in sketching and watercolour (and this skill was not lost as the years went on, if his 1938-ish sketch of a willow tree is anything to go by), but these imaginary landscapes are very different indeed; the palette is unsettlingly acidic, and discordant or violent at times, and many images, such as the famous cover picture for The Hobbit (1937), are severely stylised, to the point of regimentation. It is in The Father Christmas Letters that he seems to be most harmonious as an artist, where he describes and illustrates the comic adventures of Father Christmas and his helpers at the North Pole.

The photographs reproduced emphasise his devotion to his family. Unlike CS Lewis and Charles Williams, with whom he is regularly bracketed because of their association in the “Inklings” group of Oxford writers, his personal life was largely uncomplicated and conventional – except in the unconventional degree of his romantic devotion to his wife, Edith, as expressed in a mythologised version of their love, “The Lay of Beren and Luthien” (the two names appear on their gravestone).

Their long courtship, almost derailed by the interference of Tolkien’s clerical guardian, extended through the period of his wartime service (he fought in the Battle of the Somme and lost some of his closest friends). The relationship took on a powerful emotional charge, which – his biographers suggest – put some strain on the marriage when it was finally allowed to happen; and Edith Tolkien did not always find the atmosphere of religious and imaginative intensity in which her husband lived easy to cope with. But the marriage was manifestly resilient and fundamentally secure, and the couple’s four children seem to have grown up in an atmosphere of warmth. Tolkien may have been a fairly typical English father of his generation in the actual time he spent with his children, but there is no denying his devoted affection and the labour he put into the letters and stories written for them.

****

So, how do we now respond to Tolkien’s imagined world, a world that is hierarchical, notoriously short on female agents, and which was accused by the poet Edwin Muir of being populated exclusively by different-sized schoolboys? As with Lewis, the complaint about implied misogyny is regularly coupled with worries about racial stereotyping, the romanticising of violence and the reduction of moral issues to cosmic battles between absolutes.

It is worth noting that Peter Jackson’s superbly visualised film versions of Tolkien’s novels if anything intensify some of these problems. But things are not quite that simple. The text certainly offers accounts of battles delivered with traditional relish, and it undoubtedly presents the enemy in conventionally non-European costume and colouring; and yes, the female presence is mostly restricted to elf queens, giant spiders and a couple of gossipy homebodies (apart from the would-be warrior princess Éowyn).

But Muir’s comment is unjust (and even if it were not, it is worth remembering that Lord of the Flies is about schoolboys). Like many dismissive readings, it tends to take the heroic narrative too much at face value and fails to spot just how Tolkien consistently complicates the ethical climate of the story.

CS Lewis’s original review of The Lord of the Rings (1954-1955) perceptively notes two features that are often missed. First, the story repeatedly reminds us that all this is taking place in a post-heroic age: the great days of elves and humans are over, and the elves are on their way across the sea, never to return. For all the triumph of the king’s return, some things can never be restored. Straightforward fortissimo heroics are rare and often ineffectual.

Back in the Shire at the end of the book, the young hobbits, Merry and Pippin, enjoy a grand reputation as champions against the forces of darkness; but the focal figure of Frodo (not, in the book, a youth at all but – like the exiled king, Aragorn – a taciturn middle-aged figure) fades from popular view and suffers physically and mentally as a result of his earlier struggles. Second, throughout the narrative, “noble” figures succumb to temptation, are corrupted by passion and ego, and have their judgement clouded by partisan loyalty.

The careful construction of the narrative in its later sections alternates between a world of spectacular battles and sieges, and the grindingly slow progress of the hobbits and Gollum towards the volcanic cavern where the Ring must be destroyed. The dynamic of the relation between Frodo, his servant Sam, and Gollum – surely one of Tolkien’s most disturbing and original creations, at once monstrous and pathetic – is one of his subtlest achievements.

He deals with complex emotional rivalries, with the “homosocial” intensities of patron and client, or master and servant relationships – which seem a long way from our contemporary social or sexual politics, but which still offer illuminations about power, projection and desire. In short, one thing that Tolkien does not do is to tell his story as though conventional heroics solved anything at all.

This is where he ends up writing, despite himself, a story that is more of a novel than a myth. Myths have no authors, it has been said. Even with the apparatus of invented language and ethnography, Tolkien’s history and “legendary” are haunted by the self-awareness of a particular type of 20th-century author: English, Catholic, academic, intensely aware of the devastation of a very specific England by industrialisation and urbanisation, more stoical than optimistic, yearning for a shared social narrative that would reaffirm certain solidarities of faith and mutual respect; deeply conservative but just as deeply opposed to unexamined power and the tyranny of profit.

****

The centre of the moral world of The Lord of the Rings is a mixture of common-sense ethical humility and a carefully concealed doctrine of divine grace. Once again, not enough readers reflect on the fact that at the crucial moment of the story, Frodo himself fails: he gives way to temptation, a temptation of genuinely apocalyptic implications. We have seen him wrestling with the addictive power of the Ring, and can understand how smaller yieldings finally make this culminating disaster possible. We have been warned that no one can safely use the power vested in the Ring, even fleetingly.

In the story, we have met some characters who somehow manage to ignore the seduction of the Ring’s promise – not only the egregious Tom Bombadil (one of Tolkien’s more annoying creations, he nonetheless serves an interesting function as a figure whose alignment with the natural order saves him from any dreams of absolute control), but also Sam, Frodo’s loyal servant.

Yes, Sam is an idealised version of a socially ambivalent and archaic stereotype. Forget this for a moment and look at his instinctive realisation that fantasies of high-octane power, celebrity and control are poisonous. He is anything but perfect: his stubborn parochialism and his taunting of Gollum are failings, with bad consequences. But he retains some fundamental instinct of moral realism. This helps him share Frodo’s burden without collapsing. Frodo’s empathy for Gollum (rooted in a shared understanding of the Ring’s terrible seduction), finally leads to a genuinely shocking denouement; but Gollum, furious, alienated by Sam, recklessly greedy for the Ring, saves Frodo from his self-inflicted catastrophe and dies as a result.

Somehow, the tangled web of interaction between these three ends in “salvation”. Some force overrules and rescues them – but only through the weaving together of a whole set of flawed agencies, mixed motives, compassion, prejudice, courage and craving. Tolkien is seeking to model the way in which the creator works not by intervening but by interweaving. It is this starkly unexpected conclusion to the quest and the journey that makes the book most clearly a Christian fiction.

But even for the non-religious reader, this diagnosis of power is a reason for treating Tolkien more seriously than many are inclined to. Look beyond the unquestionable flaws: the blandly patriarchal assumptions, the recurrent patronising of the less “elevated” characters, the awkwardness of the would-be High Style of narrative and dialogue, the pastiche of Scott or Stevenson at their worst; beyond even the fantastically elaborated histories and lores and languages of Middle Earth.

The work is ultimately a fiction about how desire for power – the kind of power that will make us safe, reverse injustices and avenge defeats – is a dream that can devour even the most decent. But it is also a fiction about how a bizarre tangle of confused human motivation, prosaic realism and unexpected solidarity and compassion can somehow contribute to fending off final disaster. Not quite a myth, but something of a mythic structure, and one that – in our current climate of political insanities and the resurgence of varieties of fascistic fantasy – we could do worse than think about.

Some, including Michael Moorcock, have accused Tolkien himself of implicit fascism because the story ends with everyone going back home and order being restored. But this is a travesty of the narrative’s logic. The return home is a return to a bitter conflict with exploitation, malice and petty tyranny. And, as we have seen, the whole story is haunted by memories of loss, awareness of fallibility and, above all, scepticism about anyone’s fitness to wield absolute power.

Tolkien’s work is indeed more than just “fantasy”. Terry Pratchett has a part to play, because parody and carnival are excellent parents for political scepticism; but Tolkien’s dogged concern about the terrible dangers of our desire for final solutions and unchallengeable security is even more necessary. It is a good moment to dust off Tolkien’s legacy and pay him the compliment of attending to what he struggled to bring to birth in the huge labour of “sub-creation” to which this exhibition witnesses.

Rowan Williams, former Archbishop of Canterbury, is a contributing writer and a lead book reviewer for the New Statesman.

“Tolkien: Maker of Middle-Earth” is at the Bodleian Libraries, Oxford until 28 October