Whitehall says that unless Scottish public spending and debts are heavily cut, the currency could be threatened by speculators

The Treasury has warned that it could refuse to agree a formal currency union with an independent Scotland unless Scottish public spending was heavily restrained and the country cut its debts to reassure the markets.

In a detailed critique of Alex Salmond's proposals to create a new sterling area, published on Tuesday by Chancellor George Osborne, the Treasury said it had significant doubts about whether a currency pact would be in the UK's interests.

It said if Salmond won next year's independence referendum, a currency union would expose the UK to greater risks from speculators and downgrading on financial markets. The Scottish economy would be a tenth of the size of the UK's and its heavy dependence on volatile North Sea oil and gas receipts and on financial services would leave it more vulnerable to economic shocks. Its status as a newly separate sterling economy may also unnerve investors and the markets.

That, the Treasury report said, would mean there was "a fundamental asymmetry in the degree of exposure to fiscal and financial risk as a sterling union would comprise two members of very different sizes".

It added: "Even with constraints in place, the economic rationale for the UK to agree to enter a formal sterling union with a separate state is not clear."

The UK government believes a newly independent Scottish government would also be required to formally commit to joining the euro as a condition of its EU membership, raising further doubts about the durability of a sterling pact.

The Treasury and the Bank of England, which Salmond hopes will become Scotland's central bank and lender of last resort, would then insist on "rigorous oversight of Scotland's economic and fiscal plans by both the new Scottish and the continuing UK authorities. These constraints would need to reflect the difference in the degree of exposure to fiscal risk."

The Bank might even refuse to continue to allow Scotland's three banks, Royal Bank of Scotland, Bank of Scotland and Clydesdale, to continue issuing sterling banknotes. The three banks have about £3.8bn of their own notes in circulation, underpinned by equivalent cash and securities deposits in the Bank of England.

The paper, called Scotland analysis: currency and monetary policy, concluded: "If financial markets perceive that a currency union (or a fixed exchange rate regime) is not economically or politically durable, or only a transitional arrangement, speculative activity can put immediate pressure on the arrangement."

It said Scotland's finances and economy would therefore be weakened under independence; its other options – joining the euro immediately, creating its own currency or simply using sterling without any deal – were also flawed and risky.

Describing the UK as "one of the most successful monetary, fiscal and political unions in history", it argues: "All of the alternative currency arrangements would be likely to be less economically suitable for both Scotland and the rest of the UK."

Trailed heavily in Scotland over the weekend, the Treasury report – its first formal intervention in the independence debate, is being circulated widely within City and financial institutions, as London seeks to shore up its case within the business world against independence.

But its analysis is being resisted by Salmond and the Scottish government, which rushed out its own currency report in response. It believes the very close economic and business ties, and the tightly integrated economies cited by the Treasury, are very good reasons for agreeing a sterling pact.

John Swinney, the Scottish finance secretary, said its analysis of Scotland's balance of payments showed the Scottish economy was consistently healthier and less indebted than the UK's as a whole over the last 30 years, despite higher per head state spending.This was due to oil receipts, which were expected to increase, and added £40bn last year to the UK's balance of payments. The Scottish government's fiscal commission, which included the Nobel prize-winning economist Joseph Stiglitz, had said a sterling pact was clearly in the UK's interests, Swinney added.

"A sterling zone, with the pound as a shared currency will provide the full flexibility to set tax and spending decisions to target key opportunities and challenges in Scotland," Swinney said.

"A sterling zone is also in the overwhelming economic interests of the rest of the UK every bit as much as it is in the interests of Scotland. An independent Scotland using the pound will mean sterling's balance of payments will be massively supported by Scotland's huge assets, including North Sea oil and gas – which alone swelled the UK's balance of payments by £40bn in 2011-12.

"Far from holding the Scottish economy back, independence would allow a future Scottish government to tailor its economic, investment, taxation and capital spending strategies far more intelligently to suit the country's interests: this would strengthen rather than weaken its economy."