Whenever modern civilisations contemplate their own mortality, there is one ghost that will invariably rise up from its grave to haunt their imaginings. In February 1776, a few months after the publication of the first volume of The Decline and Fall of the Roman Empire, Edward Gibbon commented gloomily on the news from America, where rebellion against Britain appeared imminent. "The decline of the two empires, Roman and British, proceeds at an equal pace." Now, with the west mired in recession and glancing nervously over its shoulder at China, the same parallel is being dusted down. Last summer, when the Guardian's Larry Elliott wrote an article on the woes of the US economy, the headline almost wrote itself: "Decline and fall of the American empire".

Historians, it is true, have become increasingly uncomfortable with narratives of decline and fall. Few now would accept that the conquest of Roman territory by foreign invaders was a guillotine brought down on the neck of classical civilisation. The transformation from the ancient world to the medieval is recognised as something far more protracted. "Late antiquity" is the term scholars use for the centuries that witnessed its course. Roman power may have collapsed, but the various cultures of the Roman empire mutated and evolved. "We see in late antiquity," so Averil Cameron, one of its leading historians, has observed, "a mass of experimentation, new ways being tried and new adjustments made."

Yet it is a curious feature of the transformation of the Roman world into something recognisably medieval that it bred extraordinary tales even as it impoverished the ability of contemporaries to keep a record of them. "The greatest, perhaps, and most awful scene, in the history of mankind": so Gibbon described his theme. He was hardly exaggerating: the decline and fall of the Roman empire was a convulsion so momentous that even today its influence on stories with an abiding popular purchase remains greater, perhaps, than that of any other episode in history. It can take an effort, though, to recognise this. In most of the narratives informed by the world of late antiquity, from world religions to recent science-fiction and fantasy novels, the context provided by the fall of Rome's empire has tended to be disguised or occluded.

❦

Consider a single sheet of papyrus bearing the decidedly unromantic sobriquet of PERF 558. It was uncovered back in the 19th century at the Egyptian city of Herakleopolis, a faded ruin 80 miles south of Cairo. Herakleopolis itself had passed most of its existence in a condition of somnolent provincialism: first as an Egyptian city, and then, following the conquest of the country by Alexander the Great, as a colony run by and largely for Greeks. The makeover given to it by this new elite was to prove an enduring one. A thousand years on – and some 600 years after its absorption into the Roman empire – Herakleopolis still sported a name that provided, on the banks of the Nile, a little touch of far-off Greece: "the city of Heracles". PERF 558 too, in its own humble way, also bore witness to the impact on Egypt of an entire millennium of foreign rule. It was a receipt, issued for 65 sheep, presented to two officials bearing impeccably Hellenic names Christophoros and Theodorakios and written in Greek.

But not in Greek alone. The papyrus sheet also featured a second language, one never before seen in Egypt. What was it doing there, on an official council receipt? The sheep, according to a note added in Greek on the back, had been requisitioned by "Magaritai" – but who or what were they? The answer was to be found on the front of the papyrus sheet, within the text of the receipt itself. The "Magaritai", it appeared, were none other than the people known as "Saracens": nomads from Arabia, long dismissed by the Romans as "despised and insignificant". Clearly, that these barbarians were now in a position to extort sheep from city councillors suggested a dramatic reversal of fortunes. Nor was that all. The most bizarre revelation of the receipt, perhaps, lay in the fact that a race of shiftless nomads, bandits who for as long as anyone could remember had been lost to an unvarying barbarism, appeared to have developed their own calendar. "The 30th of the month of Pharmouthi of the first indiction": so the receipt was logged in Greek, a date which served to place it in year 642 since the birth of Christ. But it was also, so the receipt declared in the Saracens' own language, "the year twenty two": 22 years since what? Some momentous occurance, no doubt, of evidently great significance to the Saracens themselves. But what precisely, and whether it might have contributed to the arrival of the newcomers in Egypt, and how it was to be linked to that enigmatic title "Magaritai", PERF 558 does not say.

We can now recognise the document as the marker of something seismic. The Magaritai were destined to implant themselves in the country far more enduringly than the Greeks or the Romans had ever done. Arabic, the language they had brought with them, and that appears as such a novelty on PERF 558, is nowadays so native to Egypt that the country has come to rank as the power-house of Arab culture. Yet even a transformation of that order barely touches on the full scale of the changes which are hinted at so prosaically. A new age, of which that tax receipt issued in Herakleopolis in "the year 22" ranks as the oldest surviving dateable document, had been brought into being. This, to almost one in four people alive today, is a matter of more than mere historical interest. Infinitely more – for it touches, in their opinion, on the very nature of the Divine. The question of what it was that had brought the Magaritai to Herakleopolis, and to numerous other cities besides, has lain, for many centuries now, at the heart of a great and global religion: Islam.

❦

It was the prompting hand of God, not a mere wanton desire to extort sheep, that had first motivated the Arabs to leave their desert homeland. Such, at any rate, was the conviction of Ibn Hisham, a scholar based in Egypt who wrote a century and a half after the first appearance of the Magaritai in Herakleopolis, but whose fascination with the period, and with the remarkable events that had stamped it, was all-consuming. No longer, by AD 800, were the Magaritai to be reckoned a novelty. Instead – known now as "Muslims", or "those who submit to God" – they had succeeded in winning for themselves a vast agglomeration of territories: an authentically global empire. Ibn Hisham, looking back at the age which had first seen the Arabs grow conscious of themselves as a chosen people, and surrounded as he was by the ruins of superceded civilisations, certainly had no lack of pages to fill.

PERF 558 … the receipt for 65 sheep, issued in year 22, written in Greek and Arabic. Photograph: National Museum In Vienna

What was it that had brought the Arabs as conquerors to cities such as Herakleopolis, and far beyond? The ambition of Ibn Hisham was to provide an answer. The story he told was that of an Arab who had lived almost two centuries previously, and been chosen by God as the seal of His prophets: Muhammad. Although Ibn Hisham was himself certainly drawing on earlier material, his is the oldest biography to have survived, in the form we have it, into the present day. The details it provided would become fundamental to the way that Muslims have interpreted their faith ever since. That Muhammad had received a series of divine revelations; that he had grown up in the depths of Arabia, in a pagan metropolis, Mecca; that he had fled it for another city, Yathrib, where he had established the primal Muslim state; that this flight, or hijra, had transformed the entire order of time, and come to provide Muslims with their Year One: all this was enshrined to momentous effect by Ibn Hisham. The contrast between Islam and the age that had preceded it was rendered in his biography as clear as that between midday and the dead of night. The white radiance of Muhammad's revelations, blazing first across Arabia and then to the limits of the world, had served to bring all humanity into a new age of light.

The effect of this belief was to prove incalculable. To this day, even among non-Muslims, it continues to inform the way in which the history of the Middle East is interpreted and understood. Whether in books, museums or universities, the ancient world is imagined to have ended with the coming of Muhammad. Yet even on the presumption that what Islam teaches is correct, and that the revelations of Muhammad did indeed descend from heaven, it is still pushing things to imagine that the theatre of its conquests was suddenly conjured, over the span of a single generation, into a set from The Arabian Nights. That the Arab conquests were part of a much vaster and more protracted drama, the decline and fall of the Roman empire, has been too readily forgotten.

❦

Place these conquests in their proper context and a different narrative emerges. Heeding the lesson taught by Gibbon back in the 18th century, that the barbarian invasions of Europe and the victories of the Saracens were different aspects of the same phenomenon, serves to open up vistas of drama unhinted at by the traditional Muslim narratives. The landscape through which the Magaritai rode was certainly not unique to Egypt. In the west too, there were provinces that had witnessed the retreat and collapse of a superpower, the depredations of foreign invaders, and the desperate struggle of locals to fashion a new security for themselves. Only in the past few decades has this perspective been restored to its proper place in the academic spotlight. Yet it is curious that long before the historian Peter Brown came to write his seminal volume The World of Late Antiquity – which traced, to influential effect, patterns throughout the half millennium between Marcus Aurelius and the founding of Baghdad – a number of bestselling novelists had got there first. What their work served to demonstrate was that the fall of the Roman empire, even a millennium and a half on, had lost none of its power to inspire gripping narratives.

"There were nearly twenty-five million inhabited planets in the Galaxy then, and not one but owed allegiance to the Empire whose seat was on Trantor. It was the last half-century in which that could be said." So begins Isaac Asimov's Foundation, a self-conscious attempt to relocate Gibbon's magnum opus to outer space. First published in 1951, it portrayed a galactic imperium on the verge of collapse, and the attempt by an enlightened band of scientists to insure that eventual renaissance would follow its fall. The influence of the novel, and its two sequels, has been huge, and can be seen in every subsequent sci-fi epic that portrays sprawling empires set among the stars – from Star Wars to Battlestar Galactica. Unlike most of his epigoni, however, Asimov drew direct sustenance from his historical model. The parabola of Asimov's narrative closely follows that of Gibbon. Plenipotentiaries visit imperial outposts for the last time; interstellar equivalents of Frankish or Ostrogothic kingdoms sprout on the edge of the Milky Way; the empire, just as its Roman precursor had done under Justinian, attempts a comeback. Most intriguingly of all, in the second novel of the series, we are introduced to an enigmatic character named the Mule, who emerges seemingly from nowhere to transform the patterns of thought of billions, and conquer much of the galaxy. The context makes it fairly clear that he is intended to echo Muhammad. In an unflattering homage to Muslim tradition, Asimov even casts the Mule as a mutant, a freak of nature so unexpected that nothing in human science could possibly have explained or anticipated him.

Parallels with the tales told of Muhammad are self-evident in a second great epic of interstellar empire, Frank Herbert's Dune. A prophet arises from the depths of a desert world to humiliate an empire and launch a holy war – a jihad. Herbert's hero, Paul Atreides, is a man whose sense of supernatural mission is shadowed by self-doubt. "I cannot do the simplest thing," he reflects, "without its becoming a legend." Time will prove him correct. Without ever quite intending it, he founds a new religion, and launches a wave of conquest that ends up convulsing the galaxy. In the end, we know, there will be "only legend, and nothing to stop the jihad".

There is an irony in this, an echo not only of the spectacular growth of the historical caliphate, but of how the traditions told about Muhammad evolved as well. Ibn Hisham's biography may have been the first to survive – but it was not the last. As the years went by, and ever more lives of the Prophet came to be written, so the details grew ever more miraculous. Fresh evidence – wholly unsuspected by Muhammad's earliest biographers – would see him revered as a man able to foretell the future, to receive messages from camels, and to pick up a soldier's eyeball, reinsert it, and make it work better than before. The result was yet one more miracle: the further in time from the Prophet a biographer, the more extensive his biography was likely to be.

Herbert's novel counterpoints snatches of unreliable biography – in which Paul has become "Muad'Dib", the legendary "Dune Messiah" – with the main body of the narrative, which reveals a more secular truth. Such, of course, is the prerogative of fiction. Nevertheless, it does suggest, for the historian, an unsettling question: to what extent might the traditions told by Muslims about their prophet contradict the actual reality of the historical Muhammad? Nor is it only western scholars who are prone to asking this – so too, for instance, are Salafists, keen as they are to strip away the accretions of centuries, and reveal to the faithful the full unspotted purity of the primal Muslim state. But what if, after all the cladding has been torn down, there is nothing much left, beyond the odd receipt for sheep? That Muhammad existed is evident from the scattered testimony of Christian near-contemporaries, and that the Magaritai themselves believed a new order of time to have been ushered in is clear from their mention of a "Year 22". But do we see in the mirror held up by Ibn Hisham, and the biographers who followed him, an authentic reflection of Muhammad's life – or something distorted out of recognition by a combination of awe and the passage of time?

❦

There may be a lack of early Muslim sources for Muhammad's life, but in other regions of the former Roman empire there are even more haunting silences. The deepest of all, perhaps, is the one that settled over the one-time province of Britannia. Around 800AD, at the same time as Ibn Hisham was drawing up a list of nine engagements in which Muhammad was said personally to have fought, a monk in the far distant wilds of Wales was compiling a very similar record of victories, 12 in total, all of them attributable to a single leader, and cast by their historian as indubitable proof of the blessings of God. The name of the monk was Nennius; and the name of his hero – who was supposed to have lived long before – was Arthur. The British warlord, like the Arab prophet, was destined to have an enduring afterlife. The same centuries which would see Muslim historians fashion ever more detailed and loving histories of Muhammad and his companions would also witness, far beyond the frontiers of the caliphate, the gradual transformation of the mysterious Arthur and his henchmen into the model of a Christian court. The battles listed by Nennius would come largely to be forgotten: in their place, haunting the imaginings of all Christendom, would be the conviction that there had once existed a realm where the strong had protected the weak, where the bravest warriors had been the purest in heart, and where a sense of Christian fellowship had bound everyone to the upholding of a common order. The ideal was to prove a precious one – so much so that to this day, there remains a mystique attached to the name of Camelot.

Nor was the world of Arthur the only dimension of magic and mystery to have emerged out of the shattered landscape of the one-time Roman empire. The English, the invaders against whom Arthur was supposed to have fought, told their own extraordinary tales. Gawping at the crumbling masonry of Roman towns, they saw in it "the work of giants". Gazing into the shadows beyond their halls, they imagined ylfe ond orcnéas, and orthanc enta geweorc – "elves and orcs", and "the skilful work of giants". These stories, in turn, were only a part of the great swirl of epic, Gothic and Frankish and Norse, which preserved in their verses the memory of terrible battles, and mighty kings, and the rise and fall of empires: trace-elements of the death-agony of Roman greatness. Most of these poems, though, like the kingdoms that were so often their themes, no longer exist. They are fragments, or mere rumours of fragments. The wonder-haunted fantasies of post-Roman Europe have themselves become spectres and phantasms. "Alas for the lost lore, the annals and old poets."

So wrote JRR Tolkien, philologist, scholar of Old English, and a man so convinced of the abiding potency of the vanished world of epic that he devoted his life to conjuring it back into being. The Lord of the Rings may not be an allegory of the fall of the Roman empire, but it is shot through with echoes of the sound and fury of that "awful scene". What happened and what might have happened swirl, and meet, and merge. An elf quotes a poem on an abandoned Roman town. Horsemen with Old English names ride to the rescue of a city that is vast and beautiful, and yet, like Constantinople in the wake of the Arab conquests, "falling year by year into decay". Armies of a Dark Lord repeat the strategy of Attila in the battle of the Catalaunian plains – and suffer a similar fate. Tolkien's ambition, so Tom Shippey has written, "was to give back to his own country the legends that had been taken from it". In the event, his achievement was something even more startling. Such was the popularity of The Lord of the Rings, and such its influence on an entire genre of fiction, that it breathed new life into what for centuries had been the merest bones of an entire but forgotten worldscape.

It would seem, then, that when an empire as great as Rome's declines and falls, the reverberations can be made to echo even in outer space, even in a mythical Middle Earth. In the east as in the west, in the Fertile Crescent as in Britain, what emerged from the empire's collapse, forged over many centuries, were new identities, new values, new presumptions. Indeed, many of these would end up taking on such a life of their own that the very circumstances of their birth would come to be obscured – and on occasion forgotten completely. The age that had witnessed the collapse of Roman power, refashioned by those looking back to it centuries later in the image of their own times, was cast by them as one of wonders and miracles, irradiated by the supernatural, and by the bravery of heroes. The potency of that vision is one that still blazes today.