An independent Scotland could win special concessions on joining the EU, the UK government has said, but it warns that would mean long and complex talks to win a deal with every member state.

In a softening of its stance on Scottish membership of the EU, a Foreign Office study being released on Friday by William Hague, the foreign secretary, will confirm that Holyrood could still keep sterling and the UK's existing opt-outs on borders and social policy.

The Foreign Office paper on the EU and international issues will say there are precedents for new EU member states getting special deals and winning delays on certain EU policies, but it warned that seeking favourable treatment would be highly uncertain, and would lengthen and delay the entire process for Scotland.

Speaking to the BBC's Today programme, Hague said he did not think a future independent Scotland would achieve a "seamless process of EU membership".

"That's not how it looks when you look at the facts. It's very clear that any part of a member state that leaves, that member state has to reapply for membership of the European Union," said Hague.

"What the Scottish government aspires to, in terms of keeping the UK pound rather than joining the euro, keeping the common travel area with the UK and Ireland rather than joining Schengen, having different arrangements under the Common Agricultural Policy, would be a very difficult process of negotiation."

Asked whether he believed an independent Scotland could be forced to join the euro, Hague said: "Every EU member state that has joined from the 1990s has been required by the EU to commit itself to joining the euro.

"The UK and Denmark have – right back since the Maastricht treaty – a permanent opt-out, but other states have had to commit themselves to joining the euro and all that that entails, and to joining the Schengen travel area. So Alex Salmond [the Scottish first minister] can say that it would be different in the case of Scotland, but what a very difficult negotiation that would be, and at what price for Scotland."

He also said a newly independent Scotland would not be automatically entitled to the EU bailout.

"We have a rebate as the United Kingdom from the EU, painfully negotiated in the 1980s. That does not transfer to any part of the United Kingdom that leaves the European Union, so the financial cost to Scotland of EU membership would be much greater, even if successfully negotiated. Scots would be paying more to get less from the European Union – there is no doubt about that."

David Lidington, the Europe minister, said he thought it was implausible for an independent Scotland to join the EU as a full member state in the 18-month timescale set out by Salmond.

Some countries insisted on referendums on treaty changes, while others have democratic concerns of their own – and full unanimity was required from all 28 member states on every line of the deal, he said.

"There will be some countries that have secession movements of their own that makes them cautious about giving the green light. There will be countries in eastern Europe who have a very practical concern that they are on a slow path to getting agricultural receipts from the EU, and I doubt they would be happy for Scotland to leapfrog them and get all the agricultural entitlements that Scotland currently has, so I think that there's a lot of nitty-gritty negotiation. Nobody can say confidently that if Scotland walks away from the UK there is a smooth automatic path to EU membership."

Salmond has insisted his government will not join the euro, and will resist pressure to accept the euro in the long run, and will also insist on staying within the UK-Ireland "common travel area" rather than joining the Schengen agreement on open borders.

The FCO paper said Scotland would be in "uncharted territory" if it wanted a permanent opt-out from joining the euro and Schengen – the two policies most fiercely guarded by the European commission and other member states.

Adding that getting permanent opt-outs was now extremely rare, it stated: "Requests from the candidate for unprecedented concessions and exceptions to the acquis [the body of EU law], particularly if they concern important and sensitive policy areas, have the potential to lengthen the process considerably and are not guaranteed a successful outcome."

Despite these warnings, the FCO analysis moves the UK government closer to the Scottish government's insistence that the EU is unlikely to adopt a strict, orthodox position on an independent Scotland's application.

José Manuel Barroso, the current commission president, and other commissioners, have repeatedly warned that an independent Scotland would be seen as a new state outside the EU as soon as it became independent, and would need to start the membership process from scratch.

That hardline stance has been upheld by the Spanish prime minister, Mariano Rajoy, who is resisting demands for Catalonia's independence and said late last year that any part of an EU member state that voted to leave that state would also be leaving the EU.

Rajoy has refused to confirm whether he would veto Scotland's application, but said: "It's very clear to me, as it is for everybody else in the world, that a country that would obtain independence from the EU would remain out of the EU, and that is good for Scottish citizens to know and for all EU citizens to know."

Other experts on the EU, including Sir David Edward, the former UK member of the European court of justice, insist there is no realistic prospect of Scotland being barred from the EU after independence, particularly if independence is the democratic wish of Scotland's voters.

Edward believes the UK government would face strong economic, moral and political pressures to support Scotland's continued membership, because of the "unacceptable" economic and security damage to the UK economy if there was no continuity post-independence.

The FCO paper, part of a series of Scotland Analysis documents on independence from the UK government, said, however, that a number of other member states would resent giving a newly independent state privileges they did not have, or worry about the precedent set for other independence movements.

"Some member states may be unwilling to grant opt-outs to an independent Scottish state on measures which they have had to adopt themselves," it said, adding: "Scotland's negotiations to join the EU could be complex and long, and the outcome could prove less advantageous than the status quo."

Nicola Sturgeon, Scotland's deputy first minister, said earlier this week the case for allowing Scotland to remain in the EU was common sense and in keeping with the EU's history and traditions.

"It is time to get real in this debate. To stop pretending the European Union is something it isn't," said Sturgeon. "The EU is not in the business of throwing out its citizens, of ignoring democratic processes, of reducing co-operation and cutting the size of the EU.

"It is engaged in a process of enlargement, not contraction. It is founded on the principles of democracy and mutual co-operation among countries and citizens who share its objectives."