They might not be found on the campaign trail and they wouldn't be caught dead with a placard, but Scotland's reclusive aristocrats have expressed grave fears about the prospect of independence.

In a rare series of interviews from their ancestral castles, a string of Scottish grandees confided in the high-society magazine Tatler about their concerns if Scotland votes yes next week – not least the prospect of a dreaded castle tax.

"This place eats money. But what if Salmond imposes a mansion tax? We're done for," said Eleanor, the Duchess of Argyll, who lives with her elephant polo champion husband Torquhil at Inveraray Castle in west Scotland. Their grey-stone castle, which appeared in the Downton Abbey Christmas special in 2012, is the jewel in the crown of their 60,000-acre estate.

One nameless aristocrat, whose family was reputedly involved in the 1707 Act of Union, railed about the Scottish National Party supporters seeking to break away from the union: "The buggers are out to get us!" The crossbench peer Lord Adrian Palmer, of the Huntley and Palmers biscuit dynasty, was not enamoured by Alex Salmond's party. "I mean, [the pro-independence movement] simply do not know what they're doing," he scoffed.

Felicia Morris, a high-flying lawyer dubbed "the Queen of the London-based Scots", was also alarmed by the prospect of an independent Scotland. "Everyone's very worried," she told the magazine. "It's being talked about incessantly at dinner parties."

Their chief concerns, according to the upmarket title, are not about the future of the sterling or what happens when Royal Bank of Scotland relocates its registered headquarters to London. Their worries are slightly closer to home: "Will their 80,000-acre estates be parcelled out to crofters? Might SNP leader Alex Salmond bring in a swingeing castle tax? Will treasonous Scots cast off the Queen as their head of state? It's causing disquiet among the ranks, if not the file."