Prime minister strikes at heart of the European project in speech setting out his plans for an 'in or out' referendum by 2017

David Cameron has outlined the scale of his ambition to transform the terms of Britain's membership of the EU by calling for the UK to be exempted from its founding principle: the creation of an ever-closer union.

In his long-awaited speech on the EU, the prime minister cast himself as a modern-day heretic as he pledged to challenge established thinking.

Speaking at the London headquarters of Bloomberg, Cameron confirmed plans to hold an in-out referendum after the next election, as he warned: "The biggest danger to the European Union comes not from those who advocate change, but from those who denounce new thinking as heresy. In its long history Europe has experience of heretics who turned out to have a point."

The prime minister said that nothing would be off the table when he puts forward demands for the repatriation of a series of powers to Britain if he wins the 2015 general election. A new settlement would then be put to voters in a referendum by the end of 2017.

"I believe in confronting this issue – shaping it, leading the debate. Not simply hoping a difficult situation will go away," he said.

The prime minister concluded by saying that he would campaign with all his "heart and soul" for Britain to remain in the EU if he succeeded in renegotiating its membership terms. "When the referendum comes, let me say now that if we can negotiate such an arrangement, I will campaign for it with all my heart and soul," he said.

But Cameron declined to be drawn on whether he would campaign for a no vote if he failed to secure changes in the negotiations.

Downing Street had indicated in recent weeks, as the speech was repeatedly delayed, that the prime minister would not set out a shopping list of demands. But he made clear that he wanted to challenge the central tenet of the EU: the pledge in the founding treaty of Rome in 1957 to create an "ever-closer union".

The prime minister said: "We understand and respect the right of others to maintain their commitment to this goal. But for Britain – and perhaps for others – it is not the objective. And we would be much more comfortable if the treaty specifically said so, freeing those who want to go further, faster, to do so, without being held back by the others."

Cameron said this could be achieved, in part at least, by fully implementing the Laeken declaration of 2001, which said power should be passed back to member states if that is their desire. "It was put in the treaty," he said of the 2001 agreement. "But the promise has never really been fulfilled. We need to implement this principle properly."

The prime minister also spelled out his wish to extend Britain's opt-out from aspects of the working time directive. "It is neither right nor necessary to claim that the integrity of the single market, or full membership of the European Union requires the working hours of British hospital doctors to be set in Brussels irrespective of the views of British parliamentarians and practitioners," he said.

Cameron said he thought he would be able to table his demands in treaty negotiations that he expects to be held in coming years to agree new governance arrangements for the eurozone. But he said he was prepared to follow the example of Harold Wilson, who renegotiated Britain's membership terms outside a treaty ahead of the 1975 referendum.

"If there is no appetite for a new treaty for us all then of course Britain should be ready to address the changes we need in a negotiation with our European partners," he said. Cameron had to repeat this line after initially fluffing his words.

The prime minister insisted that his strategy was designed to ensure that Britain remains an active and influential member of the EU. "If we leave the EU, we cannot of course leave Europe. It will remain for many years our biggest market, and forever our geographical neighbourhood. We are tied by a complex web of legal commitments.

"We would have to think carefully too about the impact on our influence at the top table of international affairs. There is no doubt that we are more powerful in Washington, in Beijing, in Delhi because we are a powerful player in the European Union.

"That matters for British jobs and British security. It matters to our ability to get things done in the world. It matters to the United States and other friends around the world, which is why many tell us very clearly that they want Britain to remain in the EU.

"We should think very carefully before giving that position up. If we left the European Union, it would be a one-way ticket, not a return."

But Cameron declined to rule out campaigning in favour of a no vote in his referendum if he failed to secure his demands. Nick Robinson, the BBC's political editor, asked the prime minister: "Do you not owe it to the British people and indeed to Europe to be clear? If you cannot get a better deal are you saying that you would vote no, for Britain to get out of Europe? And if you fudge the question, which you just did, aren't we entitled to assume that you want people to think that but you are scared to say it?"

The prime minister said: "I would answer that very directly. Who goes into a negotiation hoping and expecting to fail? That might be the approach you take. That is not the approach I take … There is every chance of success.

"But in the end the choice will not be for the politicians. The choice will be for the British people. The fact is we are not comfortable with the state of our membership today. It does need to change. I am setting out the path for how we change that, how we put that to the British people and in the end they will decide. But do I believe Britain's future is better off inside a changed EU? Yes I do and I and that is what I will be fighting for."