On Friday 24 June 1983, a week after he had been appointed junior employment minister in Margaret Thatcher's government, Alan Clark was told by his personal private secretary that Sir Robert Armstrong wished to see him.

Clark, a famous lothario, was more interested in becoming better acquainted with his new secretary, Jenny, but she had other ideas. A couple of hours later, he was in the cabinet secretary's anteroom, where Sir Robert produced two files. A red file contained information about political matters, while an orange one contained what the cabinet secretary described as "certain matters of personal conduct which could quite possibly leave you open to blackmail".

Having given an assurance that all the women in question had since married into grand Scottish families, Clark retreated to his office and stole a glance at Jenny: "Her eyes seemed slightly slit, malevolently gleeful."

But how did they know? "They must have been bugging my phone," he wrote. "There was no other explanation. And for ages." A decade later, when Clark sold the serialisation rights to his diaries to a Sunday tabloid, the newspaper predicted – correctly – that the revelation that politicians' phones were being tapped "will cause a storm of protest in the Commons among MPs who jealously guard their privacy".

Alan Clark when he was a government minister. Photograph: Richard Gardner/REX

The storm was driven not only by the politicians' desire for privacy, however, but by a little-known convention of Westminster politics called the Wilson doctrine, which is said to guarantee that the phones of MPs and members of the Lords will never be bugged by the UK's intelligence agencies, that their letters will not be opened, and that their emails will be completely off limits. A few days later, when MPs demanded to know whether Clark's phone had been bugged, the then prime minister, John Major, gave a cast-iron assurance that the Wilson doctrine remained in full force.

Today, however, there are fresh concerns at Westminster, as politicians question whether the doctrine is routinely being undermined by the modern mass interception practices that were revealed by the NSA whistleblower Edward Snowden.

Two Green party parliamentarians – Caroline Lucas, MP for Brighton Pavilion, and Lady Jones of Moulsecoomb – have embarked upon court proceedings asserting that the doctrine has force in law, and alleging that their legal rights have been breached.

Earlier this month there was an unexpected twist when the case reached the Investigatory Powers Tribunal, the extraordinarily secretive court that hears complaints about the intelligence agencies. There, counsel for GCHQ, MI5 and MI6 explained that the agencies wished to reserve the right to make submissions on the issue in "closed" – or secret – session. "It may well be that we would want to say something in closed about past policy and procedures in relation to the Wilson doctrine," she said.

What on earth, many in court asked themselves, might they be attempting to hide?

The Wilson doctrine had a curious birth. It is named after the former Labour prime minister Harold Wilson, who surprised the Commons in November 1966 with an announcement that he had banned the bugging of MPs' telephones.

Wilson had been in a tight spot: an inquiry into the practice of phone-tapping had concluded some years earlier that MPs were "not to be distinguished from an ordinary member of the public", so far as the interception of communications that were not part of parliamentary proceedings was concerned, and Labour MPs were persistently demanding to know whether they were being bugged. When his own backbenchers were joined by a much-lampooned Tory, Sir Tufton Beamish, Wilson decided to outflank them all by making his announcement. Mischievously, he added that he had introduced the ban when he first entered Downing Street – two years earlier.

The implication was clear: MPs had been bugged when the Conservatives were in office. The next day's Times described the "uncomfortable silence" that descended on the Tory front bench when Wilson sat down. Five days later the government announced that the ban on bugging applied also to members of the House of Lords.

Since then, every incoming prime minister has been asked to provide the Commons with fresh assurances that the Wilson doctrine remains unviolated. Over the years it has become something of a Westminster ritual. It would be a brave PM who refused to oblige.

In 1997, for example, with email becoming a popular means of communication, Tony Blair confirmed that the convention covered other forms of electronic surveillance, and not just telephones. A few years later Blair appeared to extend the doctrine still further when he said it "extends to all forms of warranted interception of communications".

By now, what began life as a nimble piece of parliamentary footwork was being seen by many at Westminster as a rule of political life that was written in stone. And in the absence of a written constitution, the Wilson doctrine appeared to be on its way to joining those assorted pieces of case law, common law, treaties, statutes and conventions that accumulate to form the British constitution. Until, that is, one looks a little more closely at the fine print.

When Wilson made his statement, he had added: "If there was any development of a kind which required a change in the general policy, I would, at such moment as seemed compatible with the security of the country, on my own initiative make a statement in the House about it."

As a consequence of that rider, MPs could at any time find themselves in a position where the no-bugging policy had been suspended, but the prime minister of the day had decided, for reasons of national security, that it was not the right moment to make an announcement. The record of the debate that followed suggests that those MPs who were present failed to pick up on this. From the moment of its birth, the Wilson doctrine was misinterpreted.

Sinn Fein President Gerry Adams with part of the mysterious device found in his car. Photograph: PA Archive/Press Association Ima

So have MPs been bugged by MI5?

Well, the agency acknowledges that it keeps files on members of parliament so that it can alert prime ministers and leaders of the opposition to any concerns it has about proposed new members of the cabinet or shadow cabinet.

And Gerry Adams of Sinn Fein has been bugged, of course. At a press conference in December 1999, Adams brandished a 2ft-long device that had been installed in a car that he regularly used. The following year the former Northern Ireland Secretary Mo Mowlam admitted, during a TV chat show, that she had authorised the bugging. But that operation was said not to have breached the convention, on the grounds that the target was a senior IRA man who also used the car.

Four years later Adams's colleague Martin McGuinness was also shown to have been bugged when transcripts of his telephone conversations appeared in a new biography. During one conversation, Mowlam – who would have known the call was being recorded – addressed McGuinness as "babe", while Jonathan Powell, Blair's chief of staff, was recorded saying he thought some Unionist MPs were asses.

The biographers were arrested, along with a policeman accused of leaking the transcripts. This time, McGuinness was said not to be protected by the Wilson doctrine because he had not taken up his seat at Westminster and uttered his oath of allegiance to the Queen. When Sinn Fein found a bug in their west Belfast offices the following year, party officials tried to sell it on eBay.

Then in 2008, Sadiq Khan, now the shadow justice secretary, was discovered to have been bugged while meeting a constituent, Babar Ahmad, who was being held at Woodhill high-security prison north of London, while fighting extradition to the US on terrorism charges. The listening device was hidden in a table in the visiting area, and was operated by police on behalf of MI5. When the details emerged in the Sunday Times, Sir Christopher Rose, the government's chief surveillance commissioner, was asked to conduct an inquiry. He concluded that the Wilson doctrine did not appear to have been broken, because Khan had visited Ahmad as an old childhood friend, and that while the junior officers involved in the operation knew he was an MP, the senior officers who authorised the bugging did not.

Sadiq Khan … bugged visiting a constituent in prison. Photograph: Sarah Lee for the Guardian

Critically, Rose noted that the law allowed the bugging to be authorised in this way, rather than by a warrant from the home secretary. And as Tony Blair had said 11 years earlier – in the statement that appeared, at the time, to extend the Wilson doctrine – it applied only to "all forms of warranted interception of communications". With some understatement, Rose concluded that "there is manifest scope for confusion in the minds of officers of public authorities and MPs as to the correct interrelationship between the Wilson doctrine and the legislation".

Few at Westminster believe that Sinn Fein politicians and Sadiq Khan are the only MPs to have been bugged in recent years. Mike Hancock, the MP for Portsmouth South, is convinced that his phones were tapped in 2010, when his Russian assistant Ekaterina Zatuliveter was arrested on suspicion of being a spy. Zatuliveter eventually won her appeal against deportation after an immigration commission concluded that it was highly unlikely that she had ever been an intelligence agent. Hancock told other members of his staff that he is certain he was under surveillance during that case. "He said that the only way they could have known some of the things they did was by tapping his phone," said one.

Mike Hancock MP … convinced his phones were tapped. Photograph: Solent News/REX

It seems unlikely that the Wilson doctrine has been completely suspended. It is more likely that it is suspended occasionally to give MI5 the chance to spy on a particular MP or peer, without any announcement being made.

If that were not the case, it would be difficult to explain the attempt that the Blair government made in late 2005 to persuade prominent MPs that the doctrine should be scrapped. Michael Martin, then the speaker of the House of Commons, recalled when giving evidence to a parliamentary committee several years later how he was approached by a high court judge who said that he believed the doctrine should be killed off on the grounds that there were MPs who "can get into difficulties and can be involved in illegal organisations". Martin refused to assist, but says the judge persisted. "In fact, so insistent was he about the Wilson doctrine ... I had to involve the deputy prime minister, John Prescott, and say: 'John, get this man off my back. He is not going to do this.' "

There was a row in the cabinet about the plans, with Blair insisting MPs should enjoy no greater safeguards against surveillance than any other citizen. John Reid, then defence secretary, was said to have been furious. One commentator noted that this surprised cabinet colleagues, as Reid was not known for his support for civil liberties. The prime minister backed down.

The argument for scrapping the Wilson doctrine was set out clearly by Sir Swinton Thomas, who was serving as the surveillance regulator, the interception of communications commissioner, at that time. He complained in 2006 that while it was fundamental to the constitution that nobody is above the law, "MPs and peers are anything but equal with the rest of the citizens of this country and are above the law." There was a comprehensive statutory and regulatory framework governing surveillance, something that had not existed in 1966. Moreover, he said, Westminster opposition to scrapping the doctrine appeared to be based on little more than self-interest.

Defenders of the doctrine say this is simply not true. David Davis, the former shadow home secretary, says: "It's not just there to protect MPs. The bigger aim is to protect people who come to MPs – whistleblowers, people who have been badly treated by the government – for whom MPs are often a last resort. If we remove the right of these people to have confidential conversations with members of Parliament, we will undermine our democracy."

But is the IPT about to hear evidence in secret that would suggest that there is already a degree of interception of such conversations? Is that why GCHQ may want the court to sit in secret? Earlier this year, in a little-noticed Commons exchange, Davis asked Francis Maude, the cabinet office minister, about a letter that Maude had sent some time earlier to a third Tory MP, Nick de Bois. In this letter, Maude had informed De Bois that while the Wilson doctrine applied to the contents of communications, it did not apply to metadata, the data created by those communications, such as numbers, email addresses, times and locations. Very often, metadata may be all that is neeeded in order to understand the nature of a communication, and its interception does not require a warrant.

Davis asked Maude: "Will he review this policy, discuss it with the prime minister and report back to the House?" Maude replied that he "absolutely understood the point", and said he would take it up with David Cameron and the home secretary, Theresa May.

That was in March, and the House is still waiting.