If you only read the British media, you would think there are no British immigrants anywhere in the world. Instead, there are only legions upon legions of expats.

The word conjures up groups ripening in the sun on recliners by the pool, lounging in bars festooned with union flags, combing foreign supermarket aisles for Marmite and HP Sauce, and decrying bronzed natives and their lack of work ethic.

For a year or two I fancied I was an expat myself. I came from India to study at St Andrews on a bursary. I mingled with classmates who had multiple passports, whose parents were expats in Zurich, Dubai, New York and Tokyo. But as I marvelled at the ease with which they glided into France, took trains in Croatia and made friends with Bedouins in Jordan, I was having protracted arguments with customs, who jabbed at my documents every time I tried to nip over to Ireland or France.

Such treatment made me realise I would never been an expat – only an immigrant. It seems it's impossible to be an Indian expatriate. Even Lakshmi Mittal, the richest man in Britain, and an Indian passport holder, wouldn't dare to call himself an expat.

So what is the difference? It seems expats have a special prerogative. It is an entitlement with far-reaching consequences. Not long ago, I won a free trip to the Caribbean. On my flight was a senior executive from a large London-listed company heading back to his tax-haven paradise in the Bahamas. He had been an expat for nearly 25 years. The Londoner loathed Britain and its tax regime. He foamed at the mouth and gesticulated wildly as he nursed a glass with fluctuating levels of scotch.

I learned of his disgust at the dross pouring into his once-beloved country and the horrifying prospect of them benefiting from his tax money. I nodded gamely as he told me how he hated immigrants and wished they would all bugger off to where they came from. Then he leaned back, closed his eyes, clucked his tongue, and said: "It's a good thing we have tame natives in the Caribbean. None of that PC nonsense."

The British in Spain number close to a million, and they positively abhor being called immigrants. The most common argument given by the expat community is that they contribute to the local economy, take nothing, and create jobs. They are not job-seeking flotsam and, after all, where would the Manuels and the Josés be without their money.

The irony of complaining about immigrants in their own country while living the life of an expat escapes the British. I don't blame them. It is mere cognitive dissonance born out of centuries of dominion over large swaths of the world. From Delhi to Darfur, the correct way to drape a napkin on your knees or manipulate dinner cutlery was the British way. The Indian just off the plane at Heathrow, the Pakistani out for his maiden walk on Oxford Street, the Lithuanian finding his way around St Pancras are all too acutely aware that this is not their country. They must ingratiate themselves as soon as possible. Our expat has no such dislocations.

Australia, Canada, America, New Zealand and scores of pins and flags on the world map were once firmly and exclusively Anglo-Saxon in identity. Not any more. The last few decades have seen planes and boats disgorge people of other, newer ethnicities, and they and their children have gained momentum in their adopted countries. In face of this rising demographic pressure there is an even more urgent need to distinguish themselves as British.

Meanwhile, in France, the term immigré has negative connotations, almost always indicating either a clandestine or a Magrebhi. In Greece, being a metanastis is as good as being unwanted cargo from Africa. The term expatriate is a stamp of superiority and is reserved for those who have the right passport – and look the part.

I, sadly, will never fit that mould.

• Follow Comment is free on Twitter @commentisfree