A young adviser to Margaret Thatcher who is now a minister for David Cameron explicitly suggested that Scotland be used as a testing ground for the introduction of the poll tax, the flagship policy that was eventually to topple her as prime minister.

Oliver Letwin, now a Cabinet Office minister, emerges in official papers publicly released on Tuesday as the man who single-handedly kept the idea of the poll tax alive in the mid 1980s despite attempts by two senior ministers to strangle it at birth.

Downing Street files for 1985 and 1986 released at the National Archives show that it was Letwin, then just 29 but already working in Thatcher’s policy unit, who argued that the poll tax – then known as the residence charge – could be introduced in two stages.

“If you are not willing to move to a pure residence charge in England and Wales immediately, you should not introduce a mixture of taxes but should rather use the Scots as a trail-blazer for the real thing,” Letwin told Thatcher in a memo written in November 1985. Letwin, whose parents knew Thatcher, had got his job on the recommendation of the cabinet minister Sir Keith Joseph.

“If the Scottish experiment worked, it could make a pure residence charge look sensible rather than extreme, and thereby pave the way for its introduction in England and Wales,” said Letwin, confirming the long-made but never confirmed charge that Thatcher treated people in Scotland as “guinea pigs” for England and Wales.

The reference to “a mixture of taxes” was no idle speculation as even Ken Baker, then environment secretary and Thatcher’s most ardent poll tax supporter, was having sufficient doubts to lend his support to a compromise solution.

The 1985 cabinet papers show those doubts were fuelled by the combination of dire warnings from the then chancellor, Nigel Lawson, and the then home secretary, Douglas Hurd and by environment department tables of “winners and losers” which showed the devastating impact of a flat-rate poll tax not only on the poor but also on Conservative voters in the English shires. While Lawson warned that the introduction of the poll tax would be “politically catastrophic” for the Conservative party, Hurd even went so far as to compare the likely enforcement difficulties to the problem of collecting BBC licence fees in west Belfast.

Lawson bluntly told the rest of the cabinet that the poll tax was “completely unworkable and politically catastrophic” in a cabinet committee memorandum in May 1985 which he later described as “the most strongly worded attack” he had launched on any policy proposal while he was in government.

He described the official “specification report” published for the first time today as giving a “horrifying picture of the impact” with more than 7 million losers: “A pensioner couple in inner London could find themselves paying 22% of their net income in poll tax, whereas a better off couple in the suburbs pay only 1%. We should be forced to give so many exceptions and concessions (inevitably to the benefit of high spending authorities in inner London) that the flat-rate poll tax would rapidly become a surrogate income tax.”

On 30 September 1985, Hurd tackled the problems of collection and enforcement of a local government tax that was to be widely attacked as making a ‘duke pay the same as a dustman’. He said it would be represented as a tax on the right to vote even though it would be based on a separate register from the electoral register.

The home secretary warned it would be impossible to check whether everybody living in a certain house had been declared to pay the tax. “One can expect to find evasion and fiddling among all classes of people,” he warned before predicting the residents’ charge would be seen by the public “as being widely evaded and would bring the new system quickly into disrepute”.

The difficulties of trying to operate the television licence system in west Belfast illustrated the problems that it might encounter in difficult areas, he added.

The cabinet papers also show that the poll tax was not the brainchild of the Adam Smith Institute, as has been widely credited, but of Lord Victor Rothschild, who had headed Edward Heath’s Downing Street thinktank in the 1970s. Thatcher’s local government ministers, Patrick Jenkin and William Waldegrave, turned to Rothschild when Thatcher asked them to find an alternative to the rates.

Lawson described this in his memoirs as a ‘fatal invitation’ – “Rothschild prided himself on having no political judgement. He was above that sort of thing. William Waldegrave seemed to regard this as an advantage.” The Downing Street files show that Thatcher herself saw Rothschild as the author of the poll tax and at one point asked him to “take a fatherly interest in the infant’s development.”

The Cabinet papers show that in April 1985 after a Chequers summit where Thatcher had pressed ahead despite Lawson’s opposition, Rothschild had written to the prime minister telling her: “The community charge (as it was officially to become) is, I believe, a winner. But I am nervous lest it is accidentally or deliberately misinterpreted, for example: ‘Tories hit the poor again’, ‘No compassion for the have-nots’.”

When further Department of Environment ‘winners and losers’ figures came in showing there would be 7.5m losers or 44% of the population, including significant numbers in low-spending English shire counties, even Ken Baker was prepared to consider alternatives.

Rothschild again intervened with what he called Operation Gynandromorph – “I don’t know if the prime minister did Greek”, he asked before explaining it meant hybrid – involving a new mixed tax based on the square footage of a property and the number of people in the home to deal with the problem of a poor pensioner couple living in overcrowded accommodation.

The idea was rapidly seized on by Baker and Waldegrave as a possible answer. In November 1985 Letwin told Thatcher however that while it dramatically reduced the number of losers it was still “electorally dangerous”. Instead he seized on the situation in Scotland where George Younger, the Scottish secretary, was desperate for an alternative to the rates after a politically disastrous domestic rating revaluation had laid waste to Tory fortunes north of the border.

Letwin told Thatcher: “You have an apparent dilemma: a sudden move to residence charges in England and Wales is too dangerous; but a gradual transition via a mixed tax is unattractive. We believe that George Younger may offer us a way out of this dilemma. He is extremely keen to use Scotland as trail-blazer for the pure residence charge,” reported Letwin.

What sold it to Thatcher however was the fact that as Letwin pointed out it would also unite the cabinet as “since the radicals would be able to trumpet the Scottish experiment, and the sceptics would be comforted by the absence of any turbulence on the English domestic front.”

Letwin told Thatcher, using a helpful table showing just how deeply split the cabinet was on the issue. Thatcher could only count on the definite support of five ministers to press ahead the poll tax in England and Wales first – including Younger, Baker, Nicholas Ridley, Nicholas Edwardes and Lord Whitelaw. Three more ‘might be pro’ – Norman Tebbit, Leon Brittan and Sir Keith Joseph. Four – Lawson, Hurd, Michael Heseltine and Lord Hailsham – were listed as definitely against. The remaining ten, including Ken Clarke and Geoffrey Howe, were put down as waiverers. Letwin was proved right in that the Scottish pilot scheme united the cabinet and kept the idea of the poll tax alive.

By January 1986, Baker had published a green paper on scrapping the domestic rates with the poll tax at its centrepiece. The steamroller was to prove unstoppable all the way to the mass riots of 1990 after its introduction into England and Wales – and its role in triggering Thatcher’s downfall six months later.