At the heart of a stunning 50-acre estate by the banks of the river Deveron in Aberdeenshire sits the granite-clad Victorian mansion Blairmore House, home to four generations of the prime minister's family.

Built in the 1880s by Alex Geddes, a Scotsman who became known as the Chicago grain king, the estate holds decades of David Cameron's family history. The union of the Geddes and Cameron families was celebrated in the grounds in 1905, and the nearby chapel remembers forebears killed in the first world war. David's father, Ian Donald Cameron, was born in 1932 at Blairmore House. But soon after that, the old place was sold.

So it was perhaps for sentimental reasons that the offshore fund Ian Cameron helped to establish in the tax haven of Panama shares the name. Blairmore Holdings Inc, just like Blairmore House, is a monument to wealth obtained overseas.

Valued today at £25m, the Panamanian fund was established in 1982 while David was still at Eton, the school that his father attended. At the time, Ian Cameron still worked at Panmure Gordon, the City broking firm at which three generations of the family were senior partners.

The family's banking history goes back even further, to the 1860s, when Sir Ewen Cameron joined the industry. He later helped the Rothschild banking dynasty sell war bonds during the Russo-Japanese war. While at Panmure Gordon, Ian was a bond specialist too, showing determination to overcome his physical disability – he was born with deformed legs – and make partner at the firm by the age of 30.

What little we know of the historic activities of Blairmore Holdings comes from a shareholders' prospectus issued in 2006, less than a year after David Cameron became leader of the opposition.

The prospectus, designed to attract investors with a licence to take risks, notes that Ian Cameron, a director of the company, "was instrumental in the formation of Blairmore Holdings Inc" and explains that the fund is designed to "provide investors with steady long-term capital growth over and above the global rate of inflation".

It states: "The affairs of the fund should be managed and conducted so that it does not become resident in the United Kingdom for UK taxation purposes."

Explaining that the firm has access to banking services in Panama as well as auditors and trading offices in the Bahamas, Blairmore Holdings was certainly keen to convince investors that the business would be beyond the reach of Her Majesty's Revenue & Customs.

In recent months, the role of Britain's tax inspectors has come under renewed scrutiny. On Wednesday, David Cameron raised the subject again, telling parliament: "We have increased staffing at HMRC to ensure that we crack down on tax avoidance."

He then put pressure on the opposition to condemn Ken Livingstone for his tax affairs: "The man whom they are putting forward to be mayor of London has set up a company to funnel all this money into and is potentially paying a lower tax rate than the people who would work for him at the GLA. It is completely disgraceful."

There is no suggestion that Ian Cameron used his Panamanian investment fund to evade income tax, but the political debate around tax avoidance so far has focused only on individuals – not businesses like Blairmore that operate outside the tax jurisdiction of the largest world economies.

Before his death in 2010, Ian Cameron was intimately involved in several such companies: Blairmore Holdings, located in Panama; Blairmore Asset Management, a short-lived fund located in Geneva; and Close International, a Jersey-based investment vehicle founded in 1979. Of the countries in which the businesses were based, only Jersey has acceded to the requirements of the OECD global transparency tax standard.

All three businesses were operated by a clutch of British directors, who followed Ian Cameron from Panmure Gordon to Smith & Williamson, and who also found their clients among the British establishment.

Both Close International and Blairmore Holdings are listed in accounts of a clutch of Church of England charities and family trusts whose funds are managed by Smith & Williamson, among them an "educational charity" managed on behalf of Conservative peer Lord Vinson which donated to right-leaning thinktanks.

Of the three firms in which the prime minister's father was involved, Blairmore Holdings appears to be the largest and most successful. For his role as senior director, Ian Cameron was paid $20,000 (£12,400) a year. Due to the opaque nature of Panama's company records, it is impossible to know the full extent of his earnings from the fund, the size of his shareholding or what happened to the shares after his death.

What is certain is that the prime minister, who once described himself as the "heir to Blair", was not the heir to Blairmore.

On his father's death in October 2010, David Cameron inherited £300,000 in cash – just under the inheritance tax threshold at that time – while his two sisters split the proceeds of Ian Cameron's £1m Kensington mews house between them. In 2006, his elder brother, Alexander, a QC, moved into the £2.5m family home in Berkshire while the prime minister's parents downsized to the cottage next door.

In total, Ian Cameron left an estate of £2.7m but, under normal probate law, if there were assets outside England and Wales they would have to be administered in the territories in which they are registered. This leaves questions about the extent of Ian Cameron's offshore investments unanswered by the will.

Blairmore Holdings, meanwhile, continues to go from strength to strength. According to the company's own listing with the Financial Times, the value of the fund's investments has increased by 8.5% in the last three years.

The Blairmore legacy Ian Cameron left is unlike that of his forebears – it's not in the rolling hills of Scotland, but in a glass and steel office block in Panama City.

Additional reporting by John Burn-Murdoch

• This article was amended on 24 April 2012. The original said the family's banking history went back to the 1860s, when Sir Ewen Cameron helped the Rothschild banking dynasty sell war bonds during the Russo-Japanese war. He joined the banking industry in the 1860s and helped sell war bonds after the Russo-Japanese war began in 1904. This has been corrected.