Praised by AS Byatt as an author who "in some countries would be taught as their major writer", Robert Irwin has published six novels including The Arabian Nightmare, The Mysteries of Algiers, Exquisite Corpse, and Satan Wants Me. He is the author of 10 works of non-fiction including The Arabian Nights: A Companion, Night and Horses and the Desert: The Penguin Anthology of Classical Arabic Literature and For Lust of Knowing. Memoirs of a Dervish: Sufis, Mystics and the Sixties is published this month by Profile Books. He has lectured on Arabic and Middle Eastern history at the Universities of London, Cambridge and Oxford, and is the Middle East editor for the Times Literary Supplement.

Buy Memoirs of a Dervish by Robert Irwin from the Guardian bookshop

"A quest is a journey in the course of which one advances spiritually and mentally, as well as physically travelling miles. The quester leaves the familiar for the unknown. The nature of the goal may not be clear at first and may only become fully apparent at the end of the quest. It is an excellent plot device and ideally everyone's life should have a plot.

"My own quest began in the 1960s when I travelled out from the home counties in search of the meaning of life and self-knowledge. I hitchhiked across North Africa and in a zawiya (a kind of Sufi monastery) in Algeria I saw miraculous things and experienced ecstasy, but, though the books listed below are narratives of successful quests, my own Memoirs of a Dervish is uniquely an account of ultimate spiritual failure."

Malory composed his great romance in the late 15th century. Four long chapters are devoted to the quest of the Knights of the Round Table for the sublimely mysterious Sangreal. Only when it is has been found can the blight be lifted from the land and the Maimed King be restored to health. The precise nature of Grail, with its origins in both Celtic mythology and the narrative of Calvary, is unclear. The quest for the Grail is the quest for meaning of the Grail. Paladins, such as Lancelot and Gawain, are doomed to fail, for the final vision of the Grail and the solution of its riddle is reserved for Galahad who is perfectly (and chillingly) pure.

This book is not strictly a quest narrative, but is the biography of a 13th-century Sufi mystic from Muslim Spain. Ibn 'Arabi's journey to Mecca was no ordinary pilgrimage, for it was there he saw that the Black Stone, the Ka'ba, was the point of contact between the invisible and the visible and, while he was circling it, he encountered "the Evanescent Young Man, the Speaking-Silent One, He Who is neither living nor dead". In Mecca Ibn 'Arabi also communed with the seven abdals (the righteous persons who secretly serve the Qutb, or mystical pole of the world). He later proceeded on a "Night Journey" to receive the Seal of Prophethood.

3. The Way of a Pilgrim

This extraordinarily moving book opens with the anonymous narrator attending a sermon and being struck by a single sentence from the Epistle of St Paul to the Thessalonians: "Pray without ceasing'. From then on the pilgrim walks across Russia and Siberia constantly mouthing the prayer 'Lord Jesus Christ have mercy on me'. He has many remarkable encounters with both ordinary folk and spiritually enlightened teachers and, as he travels, he is filled with love of God and of his fellow creatures. The manuscript of this work was discovered in the Monastery of Mount Athos and published in Russia in 1884. It is one of the great classics of Russian Orthodox spirituality.

4. A New Model of the Universe by PD Ouspensky

The quest of Peter Demianovich Ouspensky (1878-1947) began in a Moscow schoolroom in 1890 or 1891 when he struggled with a textbook of mathematical problems. Suddenly he found himself envisaging an escape from the confinement of logic and everyday awareness. The manuscript A New Model of the Universe was written in Russian sometime before 1914 but first published in English in 1931. Ouspensky's account of his hunt for hidden knowledge covered the Legend of the Holy Grail, the mystery of Christ, myths of non-human races, the Fourth Dimension, the coming of the Superman, the symbolism of the tarot, lucid dreaming and much else. That Ouspensky ever reached his goal is questionable, but he had found plenty of interesting things on the way.

First published in 1922, Hesse's slender novel chronicled the quest of the Brahmin Siddhartha to conquer suffering and fear. His path was not straightforward, for asceticism and meditation did not of themselves bring fulfilment. It was necessary for him to abandon the ascetic way and immerse himself in the worlds of money and sex in order to fully understand what it is to be human. Only after many years was he truly able to renounce the world and find ultimate fulfilment toiling as a simple ferryman.

Underlying all the satire, snobbery, nostalgia and hedonism, a strong spiritual theme provides Waugh's novel with both its structure and its underlying meaning. Brideshead Revisited is really a platonic, Roman Catholic allegory. From the moment he encountered Sebastian Flyte in 1920s Oxford, Charles Ryder, without realising it, had embarked on a quest that would bring him to God (though this is only implied, for the novel ends before the inevitable culmination). The beauty of Sebastian and Julia Flyte, as well as of Brideshead, are only foreshadowings of the ultimate source of all beauty.

"Just as a white summer-cloud, in harmony with heaven and earth, freely floats in the blue sky from horizon to horizon, following the breath of the atmosphere – in the same way the pilgrim abandons himself to the breath of the greater life that wells up from the depth of his being and leads him beyond to the furthest horizons to an aim which is already present within him though yet hidden from his sight." Govinda travelled through Tibet prior to the Chinese invasion of 1950, but his book is no mere travelogue; it is an account of a spiritual progress. Though there are encounters with the miraculous, Govinda was no kind of mystagogue and his prose is lucid and beautiful.

8. The Thousand-Petalled Lotus: The Indian Journey of an English Buddhist by Sangharakshita

Born in Tooting Broadway in 1925, Derek Linwood was conscripted in 1949 and posted to India where he deserted and became a Buddhist monk. He wandered from village to village with a begging bowl, meditated in caves and made the cause of the untouchables his own. The Thousand-Petalled Lotus was reviewed in the Times Literary Supplement: "Non-fiction it may be, but he makes his points with all the finesse and resonance of a novelist ... his deft prose should invite comparison with EM Forster; for within the limits of his factual narrative he gives us an invaluable perspective on that 'passage' by which we reach a traditional culture." Moreover, since Sangharakshita has razor-sharp mind, this book can be read with profit even by those who have no interest at all in religion.

9. Pilgermann by Russell Hoban

Hoban has described how this novel originated in a quasi-mystical experience that came to him one night in 1980 when he was camped outside the Crusader castle of Montfort in Galilee: "The stare into the darkness, the hooded eagleness of the stronghold high over the gorge, the paling into dawn of its gathered flaunt and power precipitated Pilgermann into his time and place and me into a place I hadn't even known was there." Superficially, a picaresque, action-packed novel in which a Jew somehow gets caught up in the First Crusade and travels out to Syria, Pilgermann deals with ultimate mysteries and hidden patterns. By the way, Hoban's marvellous children's book, The Mouse and his Child deals with a different kind of quest – the quest of two tin toys to become self-winding.

"I read a book one day and my whole life was changed." This is the opening sentence of Pamuk's third novel, The New Life. As it happens, the book that lay on the table was also entitled The New Life. As a result of reading this book, which he sensed has been written for him alone and which overwhelmed him with its intensity, the narrator threw up his job, abandoned his family and set out on a series of bus journeys that criss-crossed Anatolia in a quest to discover more about the origins of the book. As Maureen Freely wrote of it, "The New Life is not a novel but something too powerful to be contained in such a 'flimsy foreign toy'. Suspended in the gap between life and death, it expresses longings that owe more to Sufism than any Western tradition."