We hear about the tactics of a referendum on membership of the European Union but little about the points of principle and substance that it raises. We need to look at these, too, otherwise we could sleepwalk into something stupid.

Why does David Cameron want a referendum on Europe?

That is simple. It is for the same reason as Harold Wilson proposed one in 1975: to deal with his divided party by appealing to higher authority. There is no popular demand for a referendum, but if you ask people in opinion polls whether they want to have a vote on EU membership, you can get a positive answer; if you backed a poll with a media campaign, you could probably get the same answer on many questions.

We have not seen large demonstrations demanding a referendum. Indeed, most voters do not care much about the EU: it comes somewhere between tenth and 15th in the issues voters list as important. Even for Ukip voters, the EU is not the most important question.

The big demand for a referendum comes from those in the Conservative Party who want to leave the EU but can’t see a way to get a majority in parliament for it. Cameron himself probably doesn’t want to take Britain out of Europe; hence his policy of trying to put this off until after the next election, in the hope that something may turn up.

Cameron’s position, though not noble, is understandable – it reflects weakness. It is less easy to understand why some in the Labour Party want to imitate it. Is it because they are afraid of the proposition that “the British people should have the right to choose”? This leads to a second question.

Are referendums a good way to make decisions?

This is also easy to answer: no. It is shameful that few political leaders are ready to say so. Democracy is not just about voting. It is also about debate and about responsibility.

Debate is necessary to understand complex issues. We invented representative democracy because debate is time-consuming and it is not practical in a modern state to assemble the whole population in market squares to debate issues. (In Athens the people were able to do this because citizens were few and they had helots and women to do the work.) Under the system of government “by the people”, the people choose the government and then hold it accountable when they don’t like what it does. If referendums are “more democratic” than decisions by parliament, why not make decisions about taxation or electricity prices by referendum, as has been tried in California (and then the lights went out)? When bad decisions are made in this way, who takes responsibility?

For years, both parties resisted calls for a referendum on capital punishment because they feared there would be a majority in favour of it. Over time and through long debates, parliament became convinced by the evidence that capital punishment had no deterrent value and that innocent people had been hanged. Yet they feared that, in a referendum, the debate would be shallow and voters would follow prejudice rather than the evidence.

The referendum on the Alternative Vote (AV) showed how difficult it can be to generate a serious debate on matters that are important but complicated where mastery of the detail demands time.

Yes, but shouldn’t we decide constitutional questions by referendum?

We seem to be drifting towards this idea. Recently on the Today programme in a discussion about some question of British institutions (it may have been the size of the House of Lords), one of the presenters said: “But isn’t this the sort of thing we’d have to have a referendum on?” We don’t have to have a referendum on anything unless parliament decides to call one.

If we did decide that constitutional change required a referendum, we would have to start by defining what was and what was not a constitutional question. Many laws – on race relations, capital punishment, the franchise and electoral systems, abortion – might or might not be part of a constitution.

Our present system of making no distinction between constitutional and other law gives us a flexible system. Usually constitutions are written as though they were going to last for ever; they never do. Some of the most important parts of the US constitution are in the amendments to it; yet now it seems impossible to secure further amendment. So the Supreme Court ends up doing a job that belongs in the political and not in the legal arena.

The sovereignty of parliament is a good principle because it allows maximum space for political decision-making and maximum opportunity for debate on issues that are always complex. It is alarming that no one, including those in the party of Edmund Burke, seems ready to defend this principle.

In 1975 they did better. A powerful speech was made opposing the Wilson referendum by the new leader of the opposition, Margaret Thatcher. Since then parliament seems to have lost confidence in itself. So, we might ask:

What has gone wrong with parliament?

This is less easy to answer. There is a growing feeling of separation between the mass of the people and the so-called political class. The Conservative Party’s very obsession with the EU illustrates this. Far from giving a voice to the people, the point of a referendum is to give a voice to a section of the Conservative Party.

One reason why people feel less represented by the House of Commons is that the two big parties are less representative of the people than they used to be. In the 1950s the Conservative Party had three million members and the Labour Party one million, together with an organic link to a broadly based trade union movement. Both parties were social as well as political organisations. They reflected a society more sharply divided and less diverse than today’s; but between them they were representative of the population in a way that their successors today are not.

Having parties that are dominant (because of the electoral system) but weak (because they are disconnected from society) together with a chamber strong on party discipline and adversarial politics is not an attractive combination. It is only on the rare occasions when this breaks down – as recently over Syria – that we get a little of the thrill of democracy in action.

And so what is to be done?

We ought to understand democracy as an evolutionary process. We are lucky to have a constitution that makes change easy. Constitutions need to keep in step with an evolving society.

In the 65 years between 1880 and 1945 we went through several constitutional revolutions. In the 65-plus years since 1945, however, not much has changed, at least not compared to the vast and sweeping changes in British society.

An open debate is needed. My own answers would be to effect a change in the electoral system to make politics more competitive, and to make parties more open to influence from the voters. Add a House of Lords chosen by lottery, as juries are. This would require some thought and reorganisation. But it would give us two houses, each representing the people in a different sense of the term “represent”.

Churchill’s remark that the best argument against democracy is a three-minute conversation with the average voter is apposite as an argument against referendums: three minutes of conversation or consideration is no way to make sense of anything. But a representative sample of the electorate, free from party whips and debating issues that matter to ordinary people, would breathe new life into parliament.

While we are at it we should do something about the funding of political parties. The present non-system brings unhealthy relations with the few, and the distrust of the many. How about a system in which all taxpayers could allocate a tiny part of their taxes, either to the government budget of their choice – health, education, development, defence – or to a political party? Plus strict limits on donations.

The chances of such a radical programme are not great. Those in power often think that the arrangements which got them there must, ipso facto, be a good thing. Yet the renewal of states very often begins with renewal of institutions.

Such changes would be experiments. Put them in place for ten years, with a sunset clause; then debate them again. In the end, democracy is one long experiment.

But this is straying from the main subject.

Wouldn’t a referendum settle the question of the EU once and for all?

No. If that were the case, it would have been settled by the 1975 referendum – when two-thirds of the British voters elected to remain in the EU.

Those who want to leave now argue that we were tricked, or that Britain has changed since then, or that the EU has changed. These arguments will be available again whenever anyone wants to use them.

Is that all? No. We should also fear the referendum because it might end in Britain leaving the European Union.

Probably some of those who tell opinion pollsters that they would vote to leave would think again if the question became real. But the conditions are different from those of 1975. The leading figures opposing membership then were from the fringes (Peter Shore, Enoch Powell, Tony Benn) and the media were almost unanimously in favour. Now we have had ten years of the drumbeat of media opposition.

Referendums are unpredictable – never a good way to govern a country – and we might end up out. That would be stupid.

Why?

In broad terms there are three ways of looking at the EU. On a practical level, the main product of the EU is regulation. There is good regulation and bad regulation; but there is no escape. No one is going to buy British products that do not meet inter­national standards. Those standards are set mostly by the EU or the US. If the UK wants to be at the table when the standards are set it has to belong to the EU; otherwise it will have to follow regulations that someone else has made.

From the point of view of realpolitik, which is the usual British way of thinking about foreign policy, a permanent coalition of European states to which we did not belong is the nightmare of British policymakers through all the ages, as I think Douglas Hurd once said. Happily, today this would not be a coalition that would threaten British security, but it might be tempted from time to time to take economic advantage of the UK’s absence to organise things in ways that suited their interests and not ours. In fact, it would be a surprise if it didn’t. Ask Norway; or look at how the EU developed in Britain’s absence from 1956 to 1973.

Or, if you believe (as I do) that international politics does not always have to be about the balance of power, the EU (with its twin, Nato) is, for all its faults, a kind of political miracle: the most successful collaboration among sovereign states ever achieved. In spite of the mess of the euro, it is still admired and imitated on other continents. This is the best Europe we have ever had; and Britain, as an influential member, has been a force for good in it. Both altruism and self-interest tell us to remain.

These three perspectives – which are not contradictory – all point to one conclusion. Much in the EU needs to be fixed. With 28 sovereign states around the table, that will be a slow and clumsy process. But the euro crisis has brought a more sober mood and the advocates of unending integration in every area are a dying breed.

There could not be a better moment to work with others for a programme of reform. That would make sense. A referendum makes none. l

Robert Cooper worked for Javier Solana and Catherine Ashton at the EU until last year. He is a visiting professor at the London School of Economics and a member of the European Council on Foreign Relations