The division over police and judicial powers is the latest example of British 'exceptionalism' that is causing rancour and dismay on the continental side of the Channel

In a drab committee room at the offices of the European council last week, senior EU officials and Irish diplomats gathered to discuss tactics and strategy towards Britain. Specifically, the Irish, taking over the six-month EU presidency on 1 January, were keen to work out how to respond to a decision in London to absent Britain from scores of policies, laws and instruments governing EU police and judicial co-operation.

The coalition in London, which is split on the issue, has not yet decided how it wants to exercise its Lisbon treaty opt-out from the battery of measures on everything from counter-terrorism to the European arrest warrant and pooled data collection before then trying to opt back in selectively to the bits that it favours. Pre-emptively, the eurocrats are already trying to anticipate the British pitch and mulling their options for a tough negotiation.

The division over police and judicial powers is the latest example of British "exceptionalism" in the EU. The opt-out en bloc was secured by Sir Jon Cunliffe, Gordon Brown's EU adviser and a Treasury mandarin steeped in the arcane detail of European regulation and negotiation for more than two decades, during the drafting of the Lisbon treaty. The opt-out was the price Britain demanded to avoid a UK referendum on the new EU pact.

Cunliffe manoeuvred Britain away from a referendum. Now, as David Cameron's ambassador to the EU, he will have to deal with the impact in Brussels of another referendum campaign, playing a central role in the perilous brinkmanship certain to define the UK's relations with the rest of Europe over the next few years as Britain struggles, perhaps once and for all, to settle its European destiny.

The UK's "exceptional" status is already underlined by its absence from the Schengen passport-free travel zone, the single currency and the new single European banking supervisor. There is also the perennially contested British rebate from the EU budget.

Now the opt-out on police and justice will markedly extend the sense of British detachment and estrangement from its continental partners. And there's a lot more. All the signs are that Cameron wants to go much further and take advantage of the single currency crisis to rewrite the terms of British participation in the EU club.

On the continental side of the Channel, the mood is one of exasperation, regret, rancour and dismay over Britain's special pleading. "You have one country in the EU that doesn't want to belong to anything. I'm not sure it can go on like this. People are running out of patience," said a senior EU official.

A second very senior official in Brussels, more in sorrow than in anger, contrasted Britain's impact on the EU over the decades with what he saw as current marginalisation. "British influence over 40 years has been incredibly strong. The French complain that the EU is too liberal, too Atlanticist, too open. That's because the British have embedded these ideas in Europe. But there's now a perception that the Brits don't have much influence over European affairs. They're wilfully semi-detached, that's the truth."

As for Cameron's apparent hopes of delivering a looser relationship that would put Britain firmly on the EU periphery but still within the union, Guy Verhofstadt, an ardent European federalist and former Belgian prime minister who leads the liberals in the European parliament, echoed the views of many in Brussels and beyond. "Europe à la carte doesn't work," he said. "Britain is seen as a troublemaker. If Cameron goes forward with his bid for second-class membership, maybe he can get it. But the British will see that it's not in their interest. The Eurosceptics want a new relationship. Something like Norway. Fine. You pay, but no say."

If domestic politics are forcing Cameron's hand on Europe, it is three years of turmoil over the viability of the euro that have presented him with the opportunity to demand a new settlement. That is a risky gambit, exploiting Europe's worst ever crisis for national ends. With European leaders consumed by crisis management over the euro, it is less than clear that they have any appetite left for accommodating a Britain seen in places as petulant and scheming to take advantage of their troubles.

"If Cameron thinks he can use the euro crisis to blackmail the others, that's the wrong way to go about it. It will fail," said Robert Cooper, the veteran UK diplomat who retired as one of the EU's top foreign policy strategists a few months ago. "If the others saw that Cameron was an active, co-operative European who is in trouble domestically, they might help. But the general feeling is that the UK is absent. Whereas previously it was one of the serious players, it is no longer."

Germany's finance minister, Wolfgang Schäuble, who has shaped his country's European policy for more than 20 years, said: "Britain is currently going through a process of reflection about its relationship with its European partners. I understand and respect the fact that this discussion must primarily take place in Britain. However, the UK has been part of the European community since 1973. Europe needs the UK and the United Kingdom needs Europe."

To secure his new deal, Cameron needs the agreement of 26 other governments as well as the assent of the European commission as the guardian of the EU treaties. It's a tall order. While the other Europeans would be making big concessions to Britain, it is not clear what London brings to the table in return, apart from a negative offer – the threat of a veto to block integrationist steps that other countries want to take without Britain.

"Britain feels that it is unique," said Herman Van Rompuy, the president of the European council,. "Working together across different fields inevitably involves give and take, the sum of which is positive."

The so-called Brexit question – the issue of whether Britain is heading for the EU exit door – has been a few years in gestation in Brussels, but in the past couple of months has developed more urgency as a greater awareness of Cameron's predicaments at Westminster has dawned on EU policymakers.

A year ago Cameron deployed his nuclear weapon, a veto, at a summit of EU leaders to prevent Germany's fiscal pact becoming EU law. He won plaudits on the Tory backbenches but in Europe made only enemies, gained nothing in return, and the others went ahead with the pact anyway outside EU law. He also riled the EU's key power brokers early on by cutting Conservative ties with the mainstream centre-right in Europe, realigning the party with nationalist east Europeans further to the right, some of them marginal, others viewed as extreme.

The turning point, sounding the alarms in Brussels and Berlin, may have come in October when Labour and Conservative rebels combined to defeat the prime minister in the House of Commons, tying his hands over the UK position on the next seven-year EU budget.

"That vote got a lot of people asking what's going on," said an EU ambassador in Brussels. "People are more careful about the UK now because they see that politics in the UK are changing. It's different now. On the British side there used to be a large pro-Europe section of the Conservative party. Now I count very few of them putting the case for Europe."

Charles Grant, director of the Centre for European Reform thinktank, views that problem as more entrenched and systemic. "For two decades Britain's EU debate has been one-sided: Eurosceptic politicians and commentators have set the agenda, while few politicians (or business leaders) have argued the merits of the EU," he wrote in a recent paper. "Pro-EU politicians have seen the short-term advantages of saying little about an unpopular subject. So they have lost the argument by default."

Since the Commons vote in October, however, Cameron has arguably enjoyed his best period in EU politics as prime minister, with two good summits from which he emerged with credit and allies, and with the striking of an allegedly new "Berlin-London axis" with the German chancellor, Angela Merkel, who has taken to emphasising the need to keep Britain engaged in and on Europe. "Germany is serious about protecting the UK. They've finally woken up to British politics," said the ambassador.

Several witnesses of recent summits say that Merkel's ascendancy is now complete and that her confidence in and command of a dominant role has grown, but that President François Hollande of France has been lacklustre and quiet while Cameron has shone. "Cameron is very effective on the issues he's interested in. He was fast and quick on Syria at the last summit," said a second EU ambassador.

Another diplomat witness said: "Merkel respects him. You can see that in the body language. And there is no chemistry between her and Hollande. It's a bit like it used to be with Tony Blair. Everyone needs to listen to Cameron. After Merkel, he is the most impressive."

But Cooper, a veteran of Brussels and Downing Street, points to the contrast between Cameron and Blair not just in their European policies, but in their methods and tactics. While Blair devoted time and energy to cultivating influential EU networks, Cameron has neglected that, he said, and paid the price by forfeiting influence.

"Blair was very skilful working systematically inside the [EU] machine. Cameron has not done that. So when the UK lobbies for something or has good ideas, others are likely to say yes we agree with you but we don't want to be seen to be supporting you."

Among EU policymakers outside Britain, there is a sense of mystification over where Cameron may be taking the UK, uncertainty over whether he is leading or being led, and a concern that the dynamic could veer out of control and produce unwanted consequences. "Avoiding an accident would be good," said one of the ambassadors. "To manage that, awareness of where you want to end up is the best antidote. But the chances of avoiding an accident are getting higher."

Perhaps. But Verhofstadt and many others outside Britain take the view that Cameron's new deal will fail to materialise, presenting the UK with a blunt, existential choice. "Before Cameron is able to repatriate powers, there will be a new EU convention leading to a new treaty. Every country will have a choice: join that or stay outside. A moment will come around 2016 when every country will have to decide whether it is in or out."