"Far off, like a perfect pearl, one can see the city of God. It is so wonderful that it seems as if a child could reach it in a summer's day. And so a child could. But with me and such as I am it is different. One can realise a thing in a single moment, but one loses it in the long hours that follow … "

– Oscar Wilde, "De Profundis"

Oscar Wilde wrote "De Profundis" in Reading gaol where he was serving two years hard labour for being himself; he was homosexual. He was sent to prison in 1895 after one of the most notorious trials in English history. Wilde's fatal amour, Lord Alfred Douglas, was son of the Marquess of Queensberry, who was a bully, womaniser, gambling addict, cycling bore and amateur boxer (to him we owe the Queensberry rules). In his personal life there was no such thing as fair play. Queensberry was a vicious pugilist detested by his family. A caricature of masculinity, he loathed the cult of art and beauty that Wilde championed, and under the guise of saving his son from sodomy, he set about bringing down Wilde at the height of his fame.

The tragedy is that he succeeded. Wilde became the most infamous man in Britain. Queensberry bankrupted him. Even his copyrights and his library were sold. On release in 1897 he was forced to live abroad, separated permanently from his wife and children, who changed their name. Three years later he was dead.

Wilde loved his sons and had been a devoted father. He loved his wife, Constance Holland, too; in his domestic affairs, and perhaps only there, Wilde was unexpectedly conventional. He liked women, but in common with Victorian men of his class, heterosexual and homosexual alike, his interests and his excitements happened outside of the home – with other men.

Unlike other men, Wilde was flamboyant, outspoken and provocative. The all-male environments of school, university, the army, gentlemen's clubs and public life operated on a tacit code of concealment – whether of mistresses or misdemeanours. The "love that dare not speak its name" was a crime, yet in the eyes of society, Wilde's real crime was being found out. The Victorians didn't invent hypocrisy, but in an era of industrious taxonomy, they were the first to reclassify that sin as a virtue.

Disgraced and imprisoned, sleeping on a plank bed, his health broken, Wilde wrote a long letter to Alfred Douglas, later published as "De Profundis". It is a meditation on faith and fate, suffering and forgiveness, love and art. The strange thing is that in this, his last real piece of work, Wilde takes us back in tone and spirit to his first authentic work – the fairy stories or children's stories he wrote immediately after the birth of his two boys, Cyril and Vyvyan.

The work Wilde is remembered for was written over a period of less than 10 years. The Happy Prince and Other Tales was published in 1888. That volume marks the beginning of Wilde's true creativity. He had lectured extensively in the US – but that would not have won him any lasting legacy, any more than his journalism or his poems. He had published a great many poems, but Wilde was a bad poet – he rarely found the right words and he was old-fashioned. Read him next to Emily Dickinson, Walt Whitman or WB Yeats, and you will see for yourself. We don't read his poetry now – it is dated and dead; too much Arcady and Hellenic Hours. The early plays suffer from the same verbal excess. Wilde at his worst wrote in purple. At his best he is dazzling.

The birth of his children seems to have regenerated Wilde as a writer. The tedious Hellenism vanished. The purple-isms faded. There are still overwritten images – Dawn's grey fingers clutching at the stars – and he never gives up his fondness for a biblical moment, usually appearing as precious stones or pronouns (thee and thy), but his style did change. The writing became freer and sharper, and also more self-reflective, without being self-absorbed.

Academic criticism of Wilde's work has too often dismissed these fairy stories as a minor bit of sentimentalism scribbled off by an iconoclast during a temporary bout of babymania. But since JK Rowling's Harry Potter and Philip Pullman's His Dark Materials, children's literature has been repositioned as central, not peripheral, shifting what children read, what we write about what children read, and what we read as adults. At last we seem to understand that imagination is ageless. Wilde's children's stories are splendid. In addition, it seems to me that they should be revisited as a defining part of his creative process.

The fate of the Happy Prince echoes Wilde's own reversal of fortune from fame and money to destitution and exile. Illustration: Grahame Baker-Smith

"The Happy Prince" starts a new note in Wilde's writing: loss. Emotions as poses are a distinctive feature of Wilde's work ("I can sympathise with everything except suffering"), and the poems had their fair share of lamentation, but from now on, loss is not a pose; it is real.

Fairytales always involve reversals of fortune. This works in both directions: beggars become kings, palaces collapse into hovels, the spoilt son eats thistles. Wilde's own reversal of fortune from fame and money to destitution and exile shares the same rapid drama. Fairytales are also and always about transformation of various kinds – frogs into princes, coal into gold – and if they are not excessively moralistic, there is usually a happy ending. Wilde's fairytale transformations turn on loss. Even "The Star-Child", in which meanness and vanity are overcome by compassion, ends with a kingdom that lasts only three years.

Wilde had a streak of prophecy in him. The children's stories can be read as notes from the future about Wilde's fate. It is as though the little child in him was trying to warn him of the dangers his adult self would soon face. "Every single work of art is the fulfilment of a prophecy", he writes in "De Profundis".

"The Happy Prince" is the story of a gilded and jewelled statue on a pedestal high above the town. One day, a Swallow late-flying to Egypt, after an unsatisfactory dalliance with a reed ("She has no conversation"), rests at the feet of the Happy Prince, who tells him of all the suffering he can see. He asks the Swallow to take the ruby from his sword and give it to a poor family. The Swallow does so. The Prince begs him to stay and to strip him bit by bit of all his gold and jewels to distribute to others. The weather is getting colder and the Swallow knows he should fly to the sun. But as he takes the Prince's jewelled eyes, he realises that he must stay, for now the Prince is blind. This is a lovely echo of King Lear, when the blinded Gloucester is not abandoned by his son Edgar – just as Cordelia never abandons the love-blind Lear.

Winter comes. The Swallow dies at the feet of the Happy Prince, no longer sparkling with jewels and gold. The Mayor has the statue pulled down – proposing one of himself in its place. As the workmen melt down the Prince they find they cannot melt his heart. They throw it on the rubbish heap next to the body of a swallow.

I don't think anything could be closer in description than this to the rubbishing of Wilde and his genius by a society obsessed by appearances and indifferent to imagination. The soul is often described as a bird – and if Wilde is the Happy Prince, then the Swallow is his soul, that returns to him and will not leave him. The Reed, shallow-rooted, flirtatious, blown about by every wind, is certainly Douglas.

Wilde believed in the soul. He played with ideas of the separation of self and soul. This is the pivot of his chilling story The Picture of Dorian Gray, but he explored this sinister theme for the first time in his fairy story "The Fisherman and his Soul". A young man wants to be rid of his Soul so that he can marry a Mermaid. He gets a magic knife from a witch and cuts away his Soul. But his Soul returns to him once a year seeking reconciliation. The story ends in death, but not in tragedy – at least not in Wilde's worldview, where Love is the supreme value. And it is Love that asks for the supreme sacrifice.

In "The Nightingale and the Rose", a Nightingale colours a white rose red with her own heart's blood so that a poor student will have the most beautiful flower in the world to give to his beloved. His beloved rejects him and his rose, and the rose is thrown in the gutter, where it is broken by a cart-wheel. As Wilde says to Douglas in "De Profundis": "Having got hold of my life you didn't know what to do with it … and so you broke it."

Indifference to gift and sacrifice is a theme of these fairy stories. In "The Birthday of the Infanta", the haughty princess humiliates the Dwarf who loves her. In "The Devoted Friend", a selfish Miller befriends Little Hans, a simple soul who tends his garden. The Miller likes making speeches about friendship, and never tires of telling Little Hans that the fruit and flowers he takes from the garden every day are only tokens of friendship. "At present you have only the practice of friendship; some day you will have the theory also." The Miller never gives anything in return, and when he has taken all there is, and Little Hans is dead of cold, hunger and exhaustion, the Miller thinks how selfish it is of Little Hans to die – not the kind of thing a friend would do at all.

When Wilde was out of prison and living on £3 a week, Douglas, who had sponged off Wilde for years, inherited £20,000. He refused to settle any money on Wilde. It was the painter Whistler, though, who was the model for the selfish Miller. Whistler was a sarcastic, self-interested friend whom Wilde ridiculed in his story "The Remarkable Rocket". In this tale about a batch of fireworks destined for a royal wedding, the Rocket assumes he is the star of the show: "The only thing that sustains one through life is the consciousness of the immense inferiority of everybody else." Whistler was uncomfortable with Wilde's success and gradually became more vicious towards him. As Queensberry closed in, Whistler abandoned Wilde. Later, when Wilde was released and struggling to write, Whistler suggested he try writing "The Bugger's Opera". But as Wilde noted, Whistler always spelt Art with a capital I.

Wilde had a lifelong interest in Catholicism, although he was only baptised on his deathbed. He had a theory that Christ was the perfect example of what an artist should be – a true individual, a political and social radical, someone who enjoyed the company of the poor and the outcast – as Wilde did. Someone who could forgive and in whom love was transfigured as imagination. What is the resurrection if not the triumph of imagination over experience? His most overtly religious story is "The Selfish Giant". In a world that is always winter, long before CS Lewis created Narnia, Wilde's Giant is both fairytale giant and Victorian industrialist. Wilde hated the hoarding and excesses of his epoch's materialism – not because he was a socialist, but because his whole endeavour, his cult of art and beauty, was a fight against the coarsening of the soul.

When the Giant builds a wall around what is "his" and drives the children away, he drives away the springtime, too. His only friends are hail and snow and bitter wind. Then one day, by a miraculous intervention, unsought and undeserved, the children creep back through a hole in the wall, led by the most mysterious child of all, whom the Giant comes to love. It is this Christ-child who returns for the Giant as he dies, covering his body in white blossom – the living opposite of the snow that had for so long covered the garden. The little child is wounded in the hands and feet when he returns – but in answer to the Giant's cries of outrage and revenge, the child explains that they are "the wounds of love".

In An Ideal Husband, Lord Goring tells the devoted but puritan Lady Chiltern, "I have sometimes thought that perhaps you are a little hard in some of your views on life … It is love, and not German philosophy, that is the true explanation of this world, whatever may be the explanation of the next."

And so when Richard Dawkins says in all seriousness, "so many of the stories I read allowed the possibility of frogs turning into princes; whether that has a sort of insidious effect on rationality, I'm not sure," we know he has made a category error.

Reason and logic are tools for understanding the world. We need a means of understanding ourselves, too. That is what imagination allows. When a child reads of a Nightingale who bleeds her song into a rose for love's sake, or of a Selfish Giant who puts a wall round life, or of a Fisherman who wants to be rid of his Soul, or of a statue who feels the suffering of the world more keenly than the Mathematics Master who scoffs at his pupils for dreaming about Angels, the child knows at once both the mystery and truth of such stories. We have all at some point in our lives been the overlooked idiot who finds a way to kill the dragon, win the treasure, marry the princess.

As explanations of the world, fairy stories tell us what science and philosophy cannot and need not. There are different ways of knowing. "Bring me the two most precious things in the city," said God to one of His Angels; and the Angel brought Him the leaden heart and the dead bird.

• The Selfish Giant and Other Stories by Oscar Wilde is published by the Folio Society for £24.95.

• This article was amended on 17 October 2013. In the original Wilde was described as being born and raised a Catholic. This has been corrected.