The British weren't quite as standoffish in India as the history books may suggest - many married locals in the early 19th century, although their families later learned to keep quiet about it. When William Dalrymple began to research the subject, he was surprised to find he too had Indian blood

Towards the end of the autumn of 1801, a major scandal broke out in Calcutta over the behaviour of James Achilles Kirkpatrick, the British resident (in effect, ambassador) at the court of Hyderabad. Some of the stories circulating about Kirkpatrick were harmless enough. It was said that he had given up wearing English clothes for all but the most formal of occasions, and now habitually swanned around the British residency in what one surprised visitor had described as "a Musselman's dress of the finest texture". Another noted that Kirkpatrick had hennaed his hands in the manner of a Mughal nobleman, and wore Indian "mustachios, though in most other respects he is like an Englishman".

These eccentricities were, in themselves, hardly a matter for alarm. The British in India - particularly those at some distance from the thoroughly Anglicised presidency towns of Calcutta, Madras and Bombay - had long adapted themselves to Mughal customs, shedding their Britishness like an unwanted skin, and wearing Indian dress, writing Urdu poetry, taking harems and adopting the ways of the Mughal governing class that they slowly came to replace, a process that Salman Rushdie, talking of modern multiculturalism, has called "chutnification". Although by 1801 this had become a little unfashionable, it was hardly something which could affect a man's career. But other charges against Kirkpatrick were of a much more serious nature.

First, there were consistent reports that Kirkpatrick had "connected himself with a female" of one of Hyderabad's leading noble families. The girl in question, Khair un-Nissa, was said to be little more than 14 years old at the time. Moreover, she was a Sayyeda, a descendant of the prophet, and thus, like all her clan, kept in the very strictest purdah. Despite these powerful taboos, the girl had somehow managed to become pregnant by Kirkpatrick and was said to have given birth to his child. Worse still, the girl's grandfather was said to have "expressed an indignation approaching to frenzy at the indignity offered to the honour of his family by such proceedings, and had declared his intention of proceeding to the Mecca Masjid [the principal mosque of the city]" where he threatened to raise the Muslims of the Deccan against the British.

Finally, and perhaps most alarmingly for the authorities in Bengal, it was said that Kirkpatrick had formally married the girl, which meant embracing Islam, and that he had become a practising Shi'a Muslim. These rumours had led some of his colleagues to wonder whether his political loyalties could still be depended on. More than a year earlier, the young Colonel Arthur Wellesley, the future Duke of Wellington, had written to Calcutta that he had heard that Kirkpatrick now seemed to be so solidly "under the influence" of the Hyderabadis that "it was to be expected that he would attend more to the objects of the Nizam's court than those of his own government"; that Kirkpatrick might, in other words, have gone over to the other side, to become, to some extent, a double-agent.

I first came across Kirkpatrick's story on a visit to Hyderabad in February 1997. I thought it was most extraordinary, and by the time I left the city I was captivated. It seemed so different from what one expected of the British in India. Little did I know then that it was to be the start of an obsession that would take over my life for the next five years.

I had been working in the India Office library on the papers of Kirkpatrick for several months before members of my own Scottish family started popping up in the story. At first they sounded a remarkably dour and unpromising lot. James Dalrymple was the first of my kinsmen to make an appearance, but entered stage left as the principal gooseberry of the plot, doing all he could to keep Kirkpatrick apart from his beloved, and scheming with Khair's grandfather to stop the two from seeing each other. Dalrymple's sister-in-law, Margaret, was an even less promising proposition, described by Kirkpatrick as "an affected, sour, supercilious woman".

My relations suddenly became a lot more interesting, however, with the appearance in the story of a Muslim princess with the somewhat unexpected name of Mooti Begum Dalrymple, a woman whose name had certainly been rigorously removed from all the family records I had seen at home. Mooti turned out to be the daughter of the Nawab of the nearby port of Masulipatam, and was married to James Dalrymple. It seems to have been a measure of the strangeness of their marriage that the two agreed to split the upbringing of their children according to sex: the boys were sent to Madras to be brought up as Christians, eventually to be sent back to East Lothian and reabsorbed into Scottish society, while the only girl from the marriage, Noor Jah Begum, was brought up as a Hyderabadi Muslim and remained in India, where she eventually married one of her father's sepoy officers.

Kirkpatrick's children, who were roughly the same age as my long-lost cousin Noor Jah Begum, also made a similarly strange journey across cultural frontiers: brought up as Muslims in Hyderabad with the names Sahib Allum and Sahib Begum, they were shipped off to London where they were baptised and took the names James and Kitty Kirkpatrick. There, Kitty's tutor fell in love with her, but was turned down; he was, after all, only a tutor. This, in retrospect, was a mistake on Kitty's part, as the heartbroken tutor was the young Thomas Carlyle, who later went on to immortalise her as Blumine, the Rose Goddess, in his novel, Sartor Resartus.

The period seemed to be full of unexpected collisions and intermixings. With brothers and sisters in cross-cultural marriages apparently routinely divided between Christianity and Islam, this was not an era when notions of clashing civilisations would have made sense to anyone. The world inhabited by Sahib Begum/Kitty Kirkpatrick was far more hybrid, and had far less clearly defined ethnic, national and religious borders, than we have all been conditioned to expect. It is certainly unfamiliar to anyone who accepts at face value the usual rigid caricature of the Englishman in India, presented over and over again in films and television dramas, of the imperialist incarnate: the narrow-minded sahib in a sola topee, dressing for dinner in the jungle while raising a disdainful nose at both the people and the culture of India.

As I progressed in my research, it was not long before I discovered that I had a direct Indian ancestor, was the product of a similar interracial liaison from this period, and had Indian blood in my veins. No one in my family seemed to know about this, though it should not have been a surprise: we had all heard the stories of how our beautiful, dark-eyed, Calcutta-born great-great-grandmother, Sophia Pattle, with whom the painter Sir Edward Burne-Jones had fallen in love, used to speak Hindustani with her sisters and was painted by Frederick Watts with a rakhi - a Hindu sacred thread - tied around her wrist. But it was only when I poked around in the archives that I discovered that she was descended from a Hindu Bengali woman from Chandernagore, who had converted to Catholicism, taken the name Marie Monica, and married a French officer. No wonder her contemporaries in Calcutta had made jokes about her name: Pattle was not a version of Patel, but it was easy to see from her appearance and behaviour why people thought it might be.

I am sure that I am hardly alone in making this sort of discovery. The wills of East India Company officials, now in the India Office library, clearly show that in the 1780s, more than one-third of the British men in India were leaving all their possessions to one or more Indian wives, or to Anglo-Indian children - a degree of cross-cultural mixing which has never made it into the history books. It suggests that, 200 years before Zadie Smith made it on to the telly and multiculturalism became a buzzword politically correct enough to wake Norman Tebbit and the Tory undead from their coffins at party conferences, the India of the East India Company was an infinitely more culturally, racially and religiously mixed place than modern Britain can even dream of being.

The wills of the period also suggest perhaps surprising ties of intense affection and loyalty on both sides, with British men asking their close friends to be executors and to care for their Indian partners, referring to them as "well beloved" or "worthy friend", and even - as Kirkpatrick's will has it - "the excellent and respectable Mother of my two children for whom I feel unbounded love and affection and esteem".

In the more loving relationships of this period, Indian wives often retired with their husbands to England. The Mughal travel writer, Mirza Abu Taleb Khan, who published in Persian an account of his journey to Europe in 1810, described meeting in London several completely Anglicised Indian women who had accompanied their husbands and children to Britain. One of them in particular, Mrs Ducarroll, surprised him every bit as much as Kirkpatrick tended to surprise his English visitors: "She is very fair," wrote Khan, "and so accomplished in all the English manners and language, that I was some time in her company before I could be convinced that she was a native of India." He added: "The lady introduced me to two or three of her children, from 16 to 19 years of age, who had every appearance of Europeans." A great many such mixed-blood children must have been quietly and successfully absorbed into the British establishment, some even attaining high office: Lord Liverpool, the early-19th-century prime minister, was of Anglo-Indian descent.

Much, however, depended on skin colour. As a Calcutta agent wrote to Warren Hastings, the governor-general of India, when discussing what to do with his Anglo-Indian step-grandchildren: "The two eldest - [who] are almost as fair as European children - should be sent to Europe. I could have made no distinction between the children if the youngest was of a complexion that could possibly escape detection; but as I daily see the injurious consequences resulting from bringing up certain [darker-skinned] native children at home, it has become a question in my own mind how far I should confer a service in recommending the third child" to proceed to England. It was decided, in the end, that the "dark" child should stay in India, while the others were shipped to Britain.

The future of such children depended very much on the whims of their parents. One of the most unashamedly enthusiastic British embracers of Mughal culture during this period was General Sir David Ochterlony: every evening, all 13 of his Indian wives went around Delhi in a procession behind their husband, each on the back of her own elephant. But beneath this enviably carefree-sounding exterior seems to have lain the sort of tensions that affect anyone who straddles two very different and diverging worlds.

One of the most moving of Ochterlony's letters concerns his two daughters, and the question of whether he should bring them up as Muslim or Christian. If Christian, they would be constantly derided for their "dark blood", but Ochterlony also hesitated to bring them up as Muslims. A letter, written to another Scot in a similar position, who has opted to bring up his children as Muslim Indians, ends rather movingly: "In short my dear M[ajor] I have spent all the time since we were parted in revolving this matter in my mind but I have not yet been able to come to a positive decision."

This period of intermixing did not last: the rise of the Victorian Evangelicals in the 1830s and 40s slowly killed off the intermingling of Indian and British ideas, religions and ways of life. The wills written by dying East India Company servants show that the practice of marrying or cohabiting with Indian bibis quickly began to decline: from turning up in one-in-three wills between 1780 and 1785, they are present in only one-in-four between 1805 and 1810. By 1830, it is one-in-six; by the middle of the century, they have all but disappeared.

Biographies and memoirs of prominent 18th-century British Indian worthies that mentioned their Indian wives were re-edited in the mid-19th century so that the consorts were removed from later editions. The mutiny of 1857 merely finished off the process. Afterwards, nothing could ever be as it was. With the British victory, and the genocidal spate of hangings and executions that followed, the entire top rank of the Mughal elite was swept away and British culture was unapologetically imposed on India.

The story of mixed-race families such as my own and the Kirkpatricks seems to raise huge questions about Britishness and the nature of empire, faith and personal identity; indeed, about how far all of these matter, are fixed and immutable - and to what extent they were flexible, tractable and negotiable. It is significant, moreover, that all this surprises us as much as it does: it is as if the Victorians succeeded in colonising not just India but also, more permanently, our imaginations, to the exclusion of all other images of the Indo-British encounter. Yet at a time when east and west, Islam and Christianity, appear to be engaged in another major confrontation, this unlikely group of expatriates provides a timely reminder that it is very possible - and has always been possible - to reconcile the two worlds and build bridges across cultures. Only bigotry, prejudice, racism and fear drive them apart. But they have met and mingled in the past; and they will do so again.

· The White Mughals: Love & Betrayal in Eighteenth Century India by William Dalrymple is published by HarperCollins. For more information, go to www.williamdalrymple.uk.com