Attempts by the Barclay brothers, owners of the Daily Telegraph, to obtain a £1bn VAT windfall from the British taxpayer, are leading to criticism of the way their offshore commercial empire is structured.

The much-delayed broadcast of an investigation by the BBC's Panorama programme into the Barclays' tax affairs focuses on the Ritz hotel, another of their assets. The hotel in London has paid no corporation tax for the past 17 years by legally claiming reliefs.

Yet the Barclays are engaged in a lengthy legal battle to obtain a £1bn VAT refund. They claim they are entitled to compound interest on a tax rebate stretching back 30 years, on behalf of the Littlewoods catalogue group which they took over in 2002.

The original refund of £204m has long been paid to them, along with a cheque for £268m in simple interest. But the twins have taken their claim for a huge extra payoff as far as the European court of justice.

Richard Murphy, from the Tax Justice Network, told the BBC that trusts set up by the Barclays met offshore: "These meetings are taking place in Monaco, but there is no doubt that sitting right in the middle of the meetings are the Barclay brothers who are therefore able to exercise control of these companies."

The MP Nadine Dorries told Panorama: "They're incredibly wealthy men who don't pay British tax. I think it is just utterly appalling."

The family is also behind anonymous offshore firms that recently bought up the London headquarters of the financially ailing Spiritualist Association of Great Britain (SAGB), the Guardian has learned. The Barclays are now trying to sell the Belgravia mansion for £26m, making a substantial capital gain.

Another mansion, Forbes House, in Knightsbridge, has similarly been purchased through offshore entities in Jersey and the British Virgin Islands (BVI). The £48m investment is reportedly intended to provide an opulent home in London for Sir David Barclay's son, Aidan.

A recent Guardian investigation, Offshore Secrets, showed how thousands of offshore entities in the BVI are being used to purchase expensive property in London. This keeps ownership secret, and opens the way to a series of tax advantages. The government is promising to end some loopholes which allow such offshore-owned property to be traded free of capital gains tax and stamp duty.

SAGB fell on hard times in 2010. With the days of Sir Arthur Conan Doyle and other famous spiritualists long gone, the organisation could no longer afford to maintain its home at 33 Belgrave Square. SAGB sold the building in 2011 to a BVI-registered offshore entity, Platinum Prime Property Investments Ltd, for £6m, then Platinum Prime immediately resold the property for £21m to Rose Season Enterprises.

Loan documents reveal that behind Rose Season is the GL Monaco Corporation, controlled by Sir David Barclay.

London property sources say that earlier this year, the mansion was on offer again by the Barclays for £26m, marking a potential capital gain of almost £5m. Currently, such offshore residential property gains would be tax free.

Forbes House, at 10 Halkin Street, is the former headquarters of the Society of Motor Manufacturers. It has been bought with a £48m loan by Forbes House Ltd, registered in the BVI, and planning applications have been made in the name of a subsidiary, the Jersey-registered Halkin St Development Ltd.

The Barclay twins, who grew up in west London, now officially reside in the tax havens of Monaco and Sark in the Channel Isles, where they have built a castle. They say they do no business in Britain. Aidan Barclay manages much of the UK business for his father and his uncle.

The Telegraph newspapers are held offshore, via two Jersey entities, the May Corporation and Press Holdings, then ultimately by a Bermuda-registered entity called B UK Ltd. Aidan Barclay is a director of the latter entity, according to company documents. It is not clear which of these offshore companies is regarded as UK tax resident, and thus liable for British taxes. Barclay declines to say.

A recent high court judgment ruled that much of the Barclay twins' business empire, including the Telegraph newspapers, is physically managed by Aidan and a group of four senior executives from a office at 20 St James's Street in central London.

In a judgment during the current battle by Barclay-controlled entities to seize control of the Claridges hotel group in London, Mr Justice Richards said "the affairs of these companies and many other business interests of the Barclay brothers, are managed from an office in St James's Street in London where a number of senior executives are based".

These included Rigel Mowatt, an accountant in charge of the Telegraph as well as Aidan Barclay. When the latter was ill, executives "looked instead to Sir David Barclay for decisions", said the judge.

One occasion when a twin, Sir Frederick Barclay, came to London to do business, is described in a witness statement by Patrick McKillen, the owner of a shareholding in the Claridges group, who is trying to fight off their bid for control. The Barclays were preoccupied with forcing Ritz hotel guests to wear ties, said McKillen.

"He [Sir Frederick] initially ignored me and instead asked the concierge why I had been allowed to enter the hotel without wearing a tie. I thought this was a joke, however Sir Frederick continued to reprimand the concierge. I felt he intended to cause me embarrassment. It was not a good way to start the meeting," McKillen said.

Sir David Barclay told the BBC in a statement: "We have not attended office, management or board meetings in the UK since leaving the country." He added: "My brother and I have no editorial, political or economic power in the UK."

Barclay's son Aidan told Panorama: "The Barclay family members and their companies abide by the law and pay the taxes required by UK law and the laws of other relevant countries."

Aidan Barclay added that the Littlewoods directors had obligations to recover as much cash as possible: "It would be a dereliction of their duties not to pursue repayments which are properly due from HMRC [HM Revenue & Customs]."

He declined to comment on the London property acquisitions.

• This article was amended on 25 January 2013 because the original said, incorrectly, "SAGB sold the property to an anonymous BVI entity, Rose Season Enterprises, for £21m, although less than £6m seems to have been paid immediately". This has been corrected.