A new regime is just taking shape in Brussels, readying to run the European Union for the next five years. Already it is clear that the three big items sitting in the in-trays for the new commission, the new parliament and the new president of the European council include what is known in Brussels as "the British question" – whether the UK stays in the EU and on what terms.

The British question could be very soon superseded or joined by "the Scottish question" – the riddle of Scotland's future in Europe should it opt to go it alone and quit the UK.

The Scottish question has senior EU officials and diplomats breaking out in a sweat. British officials in Brussels are besieged by anxious questions as to what is going to happen. There are few clear answers to the Scotland and Europe dilemma. Only many more questions. On the whole, Brussels is praying for a no victory. The two top outgoing officials in Brussels, José Manuel Barroso, head of the European commission, and Herman Van Rompuy, president of the European council have both been less than generous when speaking of Scotland's EU prospects.

A yes to independence will take the EU into completely uncharted territory, legally and politically.

"But they're not going to vote yes, are they?" asked a former EU commissioner. "That would be just awful. It will be very messy."

Britain's relationship with the rest of the EU has always been troubled and ambivalent. It is currently unusually jumpy. The combination of Alex Salmond and David Cameron makes for extreme volatility and uncertainty.

If Salmond wins his vote and Cameron wins his for a second term next year, the bizarre situation may arise where a new country called Scotland is clamouring to be let in to the EU after having forfeited 41 years of membership at the same time as a shrunken UK is heading for the EU door marked Brexit.

"We are already part of the EU set-up. We want to continue to be," said Alyn Smith, MEP for the SNP. "But there are no articles in the EU treaties which cover Scotland's position in the EU. There is no roadmap for this stuff."

The SNP has been all over the place on Scotland's European destiny. The policy has rested on a set of assertions articulated mainly by Salmond and Nicola Sturgeon whose veracity appears highly dubious.

First it was claimed that Scotland would automatically remain in the EU, inheriting its UK membership. Highly unlikely. Then it was asserted that Scotland would be put on a fast-track to membership under a different article in the Lisbon treaty from the one dealing with accession and the only process that has been used so far to admit new members. This is also extremely improbable.

But the problem is compounded by the confusion at the Brussels end.

"The EU has been a player in the debate," said Nicola McEwen, an Edinburgh University political scientist specialising in nationalism, secessionism and constitutionalism. "Because of the lack of clarity in the procedures, the EU interventions can be very political, even partisan."

When it comes to the terms of membership, there are further problems. Amid the current brouhaha in Edinburgh and London over the pound sterling and a post-independence sterling currency union, it is forgotten that any country joining the EU has to commit to joining the euro single currency – eventually. Why Scotland should be the exception to this rule is less than clear.

But Smith is forthright. "The euro is off the table. Forget about it."

Single currency membership would require another referendum in an independent Scotland, he said, though there would be no need for a referendum on Scotland separately entering the EU.

Salmond also hopes that Scotland would continue to benefit from the exceptional terms the UK has extracted from Brussels over the last 30 years. Paradoxically, he wants to retain the privileges secured by the Scottish bogey, Margaret Thatcher, in the form of the infamous British EU budget rebate, a refunding mechanism that intensely annoys the rest of the EU. There will be little enthusiasm to extend it to a new country called Scotland. The same goes for the opt-outs, say on the single currency, also won by the hated Tories at Maastricht more than 20 years ago.

There can be political fixes to several of these potential sticking points in the tradition of the Brussels fudge, perhaps some kind of transitional arrangement put in place between leaving and rejoining. And Scotland would probably be put on a fast track to EU membership. After all, membership bids revolve around negotiations on the 35 chapters of the EU's body of law and regulations – the acquis communautaire. By definition, this entire body of law already applies in Scotland. On the face of it, there is little to negotiate.

But first you have to apply to join. Getting 28 other governments to agree to the membership bid can take a while. Politically, it will be impossible for Brussels to disavow the freely and democratically expressed will of the Scottish people.

But the EU is not a union of citizens, when push comes to shove despite the regular rhetoric from EU leaders. It is a legal construct, a union of states represented by their governments, but bound by legal agreements between states. Any of those 28 other governments will have a veto on a Scottish bid which would also need to be ratified by national parliaments and secure an absolute majority in the European parliament.

Governments with secessionist problems at home, notably the Spanish, the Cypriots, and some of the east Europeans, may baulk at moves deemed to be encouraging separatism. At the very least, this could delay Salmond's timetable of 18 months to membership – another assertion that struggles to withstand scrutiny. Especially since it will coincide with negotiating with London the terms of disengagement from Britain.

An independent Scotland knocking on Brussels' door puts the EU in completely unknown territory, hence the lack of definitive answers to the riddle. No country has left the EU. No EU member state has ever broken up. Brussels is trembling at the prospect of a precedent.

The nearest analogy would be Greenland, which joined as part of Denmark at the same time as Britain in 1973, but then quit the EU following a referendum in 1985, the departure triggered by a fisheries dispute. Fisheries will also be an EU issue for Scotland, if it comes to that, since the vast bulk of the UK's fisheries belong north of the border.

The last country to join, for example, Croatia last year, had its EU funding of fishing truncated, with the money being staggered and taking years to reach the full level it is due. There were several other transitional and temporary curbs placed on EU benefits for all the east European countries who joined over the past decade. They will not be keen to see a richer country, Scotland, enjoy better terms.

While the legal problems abound, in the end it will come down to politics and political will. The likeliest prognosis is that an independent Scotland will have a certain future in Europe. But that getting there will not quite be the plain sailing predicted by the government in Holyrood.

"After the vote, the headache will be prolonging the negotiations," said McEwen. "Political considerations are likely to prevail over the legal ones."