In the spring of 2006, Gerard Russell was a bored British diplomat stewing in the heat of the Green Zone in Baghdad, “a five-mile 21st-century dystopia filled with concrete berms and highway bridges that ended in midair where a bomb had cleaved them”. Then he received a call from the high priest of the Mandeans.

The Mandeans, he knew, claimed descent from Seth, son of Adam, and believed themselves to be the last followers of John the Baptist. They were also said to be the last surviving remnant of the Gnostic sects that once proliferated across the Middle East in late antiquity. In addition, Arab scholars had long recognised them as “the last Babylonians”. It was, writes Russell, “rather like being summoned to meet one of the knights of the Round Table”. He arranged to meet the high priest and his entourage in Al-Rasheed hotel on the edge of the Green Zone.

The encounter had, however, a sad conclusion. The high priest told Russell how the American invasion of Iraq had unleashed a firestorm on his people. The Mandeans had been protected by Saddam Hussein, who saw them as a link to the ancient Babylonian empire that the Ba’athists claimed as the precursor of the modern nation state of Iraq. But during the anarchy after his fall, and the US occupation that followed, life had become impossible. The high priest spoke of the long series of forced conversions, bombings, killings and kidnappings for ransom that had affected his flock since 2002. Now he wanted to transport the entire community to the west: “There are only a few hundred of us left in Iraq,” said the high priest. “And we want to leave. We want your country to give us asylum.”

The slow and still continuing unravelling of the vast multiethnic, multireligious diversity of the Ottoman empire has been the principal political fact of both the Middle East and the Balkans ever since the mid-19th century. Under the capricious thumb of the sultans, the different faiths, tribes and ethnicities of the Ottoman empire had lived, if not in complete harmony, then at least in a kind of pluralist equilibrium: an interwoven patchwork of different communities living separately, yet side by side. But with the Ottoman retreat from the Balkans in the early 19th century, and the eventual collapse of the rest of that empire in the aftermath of the first world war, that patchwork was ripped apart.

Everywhere, pluralism was replaced with a ferocious polarisation. Almost all the former Ottoman lands suffered bouts of savage bloodletting, and some of these – Turkey 1919-21, Palestine 1948, Cyprus 1963-4, Lebanon 1975-90, Bosnia 1991-2, Iraq from 2002 and most recently in Syria from 2011 – grew into civil wars of startling violence and fought along religious faultlines.

In the aftermath of each of these wars, from Sarajevo to Baghdad, in dribs and drabs and occasional tragic exoduses such as occurred with the Yezidis last summer, ethnic and religious minorities have fled to places where they can be part of a majority: the Pontic and Smyrna Greeks to Greece; the Anatolian Armenians to Armenia; the various Jewish communities to Israel – in each case creating religious nationalisms operating in two directions. Those too few for that, such as the Mandeans and Yezidis, have tended to abandon the region altogether, seeking out places less heavy with history, such as North America or Australia. The recent Isis-driven departure of the Yezidis and Chaldean Christians of Mosul is only the latest chapter in a process that began with the secession of Serbia and Greece from Ottoman control in the 19th century, and the subsequent explusion of their Turkish minorities: in 1878, for example, about 130,000 Bosnian Muslims migrated from Sarajevo to areas under Ottoman rule.

Islam has traditionally been tolerant of minorities: the relatively gentle treatment of Christians under Muslim rule contrasts strongly with the fate of Jews and Muslims in, say, 15th-century Spain, forced to flee or convert and even then pursued by the cruelties and tortures of the Inquisition. As Aubry de la Motraye, a 17th-century Huguenot exile escaping religious persecution in Europe, admiringly put it, “there is no country on earth where the exercise of all religions is more free and less subject to being troubled than in Turkey”. The same broad tolerance that gave homes to the Jews expelled from Spain and Portugal also protected the survivors of other religions that preceded Islam: not just Judaism and eastern Christianity, but also Yezidis, Samaritans and Mandeans, as well as relative latecomers such as the Druze and the Alawites.

All this came to an abrupt end after the first world war, and the establishment of a series of ethno-religious Ottoman successor states such as Serbia, Turkey and Israel. Here, citizenship was often conflated with a religious and ethnic identity. In each of these, majoritarianism was the rule, and minorities felt increasingly unloved and unwelcome.

This process has only accelerated in the 21st century, especially in Iraq, Syria and Egypt, where the slow decline of communism and Arab nationalism, “Islamism’s secular competitors”, has taken place in parallel with the rise of fundamentalist Salafi Islam. As Russell observes: “In Egypt, the past 50 years have seen much more violence against the Copts than the previous 50 years had. Iraq, a country ruled in the 1950s by a man of mixed Shi’a-Sunni parentage, is now a maelstrom of communal violence.”

In each case, the situation of the minorities has grown increasingly untenable: the Chaldeans, Mandeans and Yezidis have all had to flee Iraq, the last Armenians have left Syria and the Copts are now haemorrhaging out of Egypt. To the east of Ottoman lands, among the Zoroastrians of Iran and the Kafir Kalash of Afghanistan, there has been a similar process of growing violence culminating in emigration.

There have been good studies of individual parts of the process, but Heirs to Forgotten Kingdoms, Russell’s brilliant and constantly engaging account of his travels through the disappearing religions of the Middle East, is the first attempt to pull all these diverse threads together. His descriptions of dogged believers clinging on in their last shrines may be terse compared with the fully drawn pen portraits one gets from Ryszard Kapuściński or Colin Thubron, but where Russell excels is in his ability to link the past and the present, and to draw from a well of historical and theological scholarship, and his deep erudition in Arabic and Farsi, to show why we must take note of these unlikely survivors from the ancient Middle East, why their emigration matters and how this is likely to affect contemporary politics.

The opening Mandean chapter shows the full range of Russell’s strengths. The Mandeans are usually looked on as the last of the Gnostics, yet he convincingly shows that many of their customs and traditions date from many centuries earlier: their scriptures are written “in a language very close to that used by Jewish scholars who compiled the Babylonian Talmud”. Their music and hymns and their avoidance of meat and alcohol, as well as their suspicion of sex, link them to the Manicheans, whose founder, Mani, was born into a Mandean household. Their use of astronomy and some of their spells, which still invoke the goddess Libat or Ishtar, are inheritances from Babylon.

It is a similar case with the Yezidis. They have long been accused by their Muslim and Christian neighbours of being devil worshippers. This is a crude caricature of a much more interesting and complex esoteric theology, whose worship of the peacock angel, Melek Taus, draws on elements of Assyrian and Sumerian religious beliefs and whose bull slaughter is an inheritance from that early competitor of Christianity, Mithraism. At the centre of their belief is their faith that Melek Taus, having rebelled against God, “extinguished the fires of hell with his tears of repentance and was restored to favour as the chief of all the angels”.

The book, which opens with one dystopia, Baghdad, ends with another: the urban wasteland of Detroit, the unlikely destination of many of these exiles. On the way, Russell takes us on a fascinating and timely journey through the beliefs and predicaments of seven fascinating but little-known religions; as well as the Mandeans and Yezidis, we meet the last of the Iranian Zoroastrians, the Druze and Samaritans lodged uneasily between Israel and the Arabs, the increasingly persecuted Coptic Christians of Egypt and the Kafir Kalash of the Hindu Kush. It’s a long time since I read a travel book that taught or illuminated so much, but its importance is greater than that. Tragically, this book puts on record for the last possible time a once-plural world that is on the verge of disappearing for ever.

• William Dalrymple’s Return of a King: The Battle for Afghanistan is out in paperback.

• To order Heirs to Forgotten Kingdoms for £16 go to bookshop.theguardian.com or call 0330 333 6846.