Imagine this scenario. German police investigating a press leak descend on Deutsche Bank headquarters in Frankfurt. They serve a warrant to seize a bundle of private letters a US reporter is storing in a safe deposit box at a bank branch in Manhattan. The bank complies and orders the branch manager to open the reporter’s box and fax the private letters to the Stadtpolizei.

Uproar! The US would be outraged at the bypassing of bilateral agreements and flouting of its citizen’s rights. And yet this is exactly what the US government is ordering Microsoft to do, according to the software giant’s general counsel, Brad Smith.

After bruising revelations regarding the company’s working relations with the National Security Agency (NSA), Smith has emerged as one of the technology industry’s loudest voices for reform and greater openness.

Now he is spearheading Microsoft’s fight against US government demands for access to emails from a Microsoft customer which are currently sitting on a server in Dublin, Ireland, as part of a narcotics investigation. Earlier this year, a US court ruled that Microsoft should hand the data over. Microsoft declined to comply, voluntarily entering into contempt.

Last week Microsoft filed its appeal: “The power to embark on unilateral law enforcement incursions into a foreign sovereign country – directly or indirectly – has profound foreign policy consequences. Worse still, it threatens the privacy of US citizens,” the company said in court documents.

If Microsoft loses, Smith argues it could put all of our private digital information at risk as well as further damaging the standing and reputation of US tech firms still reeling from the NSA whistleblower Edward Snowden’s leaks.

It is believed to be the first time a US company has fought the government against a domestic warrant for data held overseas and it is likely to prove one of the most important test cases to emerge since the NSA leaks. It is a case almost tailor-made for Smith. Microsoft’s top lawyer since 2002, he has a deep interest in the long history of government challenges to privacy – challenges he says have often been triggered by wars and by changes in technology.

Smith says the current debate, and Microsoft’s upcoming court case, were “historically inevitable”. “The pendulum swung for lots of reasons we can understand following 9/11. The pendulum always swings back, that’s one of the lessons of history,” he says.

Washington has often clamped down on personal liberty during times of war. Smith cites President John Adams’s introduction of the alien and sedition acts in the name of national security in 1798, during a “quasi-war” with France. The bills effectively disenfranchised voters who disagreed with Adams. One of the acts, the alien enemies act, was used by Franklin D Roosevelt to intern Japanese Americans during the second world war, another egregious example of government overreaction in the name of security.

Today’s debate differs from previous ones on two major points, Smith says. First, previous wars had defined endings. The “war on terror” appears without end.

President Franklin Roosevelt used 18th century emergency legislation to intern Japanese Americans during the second world war, in an overreaction in the name of security. Photograph: Bettmann/Corbis

“The current challenges to the security and safety of people don’t have that defined ending,” he says. “By definition we are having to think about how we navigate through a more dangerous world. At the same time one really needs to think hard about how the balance should be struck between public safety and personal privacy.

“Both are not just important goals but important values for society. It’s not as if we can throw one aside for the other.”

The second major difference is technology. Before the internet, issues of privacy were largely national affairs. Now those borders are gone. There are about 3 billion connected devices today – including a billion PCs and 2 billion smartphones. By the end of this decade, Smith says there will be 50 billion devices connected to data centres, everything from smart watches and health bands to thermostats and fridges. All the information they collect will be very telling about us and will all be stored – like those emails in Dublin – in the cloud. How it is protected will be one of the big debates of the coming decade, he says.

History has led to starkly different attitudes in different countries. “You can’t help but be in Berlin without feeling the full force of German history and what that means for the German people,” says Smith. “In the same vein it has impacted the views of people in the United Kingdom and the United States.”

While the Germans reacted with fury to the NSA revelations, especially the news that the NSA spied on Chancellor Angela Merkel, the UK government has fought back. Robert Hannigan, newly installed GCHQ boss, recently castigated US tech firms for failing to cooperate enough with spy agencies, calling them “the command and control networks of choice” for terrorists.

“[There are] few countries where people have lived for more decades with the threat of domestic terrorism,” says Smith. “The UK hasn’t even had the same history as the US with the government overreaching and abusing its own power. The UK doesn’t have Watergate but it does have Bletchley Park.”

Hannigan’s comments followed similar criticisms from the FBI director, James Comey, who has charged that the increasing use of encryption aids criminals and terrorists and is leading us to a “a very, very dark place”.

“I understand the point but the focus falls short of what is going to lead any of us to a better path,” says Smith. “Fundamentally, technology companies are reacting to not just a business but a societal imperative. People will only use technology that they trust and given that that trust has been under pressure companies have to find ways to restore trust.”

The only ways to do that are through stronger technology or better laws, he says.

“If there are those in government concerned about stronger technology then we need to have a dialogue about better laws. The only thing I am certain will not get us anywhere is complaints about better technology and a resistance to better laws.”

In the absence of better laws, Smith says technology companies will continue to innovate and continue to encrypt data – unless they are stopped.

So far Smith has been disappointed by the lack of progress made by legislators. In November the US failed to pass the USA Freedom Act, a bill that would have curbed some aspects of surveillance but which Smith said would still have left major issues unaddressed. For example the US has yet to rule out hacking into the data centres of firms outside the US. “We believe we are entitled to constitutional protection from our own government wherever we happen to be,” Smith says.

Regarding the UK, he is concerned about the expansion of surveillance under the Data Retention and Investigatory Powers Act. “I understand how that enhances public safety in Britain,” he says. “And I think you have to appreciate that. But on balance it creates more difficulties for creating an international regime where countries each have laws that they can follow and respect.”

If other governments follow the UK and US, Smith says, tech companies will be placed in the middle of an irresolvable issue. “One government will say you must go get this and another will say you must not hand it over,” he says. And tech companies are not in the position to solve this issue themselves, he says.

“When people suggest technology companies should go beyond what they are legally required to do, I feel what they are really asking technology companies to do is make decisions that the governments themselves haven’t been prepared to make. The better course is for governments to decide in the form of law what technology companies must do.”

The best news so far for the tech sector and privacy has come from the supreme court. In a sweeping June decision, the US’s highest court unanimously ruled that police must obtain a warrant to search the contents of cellphones seized from people they have arrested.

The so-called Riley decision shows that the supreme court understands how important technology has become in terms of storing people’s private information and reasserts the concept that “protection of privacy is a timeless value even amidst changing technology”, Smith says.

Despite some high-profile opinions, notably that of the Facebook founder Mark Zuckerberg, who infamously said privacy was dead, privacy is very much alive and well, says Smith. “The meaning of the term has evolved. It’s not about keeping something secret. It’s about continuing to control who you share information with and what they can do with it. That notion of privacy deserves legal protection.”

Microsoft’s own polling in various states and countries shows consumers agree with the supreme court that information stored in the cloud should have as much protection as information stored on paper. “So far, repeatedly 80% or more of the public embraces that principle,” he says. “The public gets it and if the public gets it then I think the courts will get it as well.”

Microsoft’s appeal is backed by Apple, AT&T, Cisco, Verizon and others, all of whom argue that a final decision in favour of the US government would create a “dramatic conflict with foreign data protection laws”. Germany has already stated that if the decision is upheld, it will not store data with US cloud service providers.

The film The Lives of Others concerned East German domestic surveillance. ‘You can’t help but be in Berlin without feeling the full force of German history and what that means for the German people,’ says Brad Smith. Photograph: PR

The Guardian and other news organisations are filing supporting briefs, arguing that if the company loses, the US government could come after news organisations by targeting emails and other information stored overseas.

Oral arguments are expected to begin in the spring. If Microsoft loses, Smith says companies will be forced to do more of what the government dislikes the most, in order to reassure customers. “It will force companies to look for more ways to encrypt data and not retain the keys. Partner with non-US companies so that non-US companies have the servers. None of which will be helpful to the US,” he says.

A win for the US will also encourage other governments to follow suit. “The US government cannot expect to have one model that it follows without anticipating that the rest of the world will follow that model,” Smith says. “And this is a model that encourages governments to reach into other territories. That does not seem like a sound approach to international stability or mutual respect in the 21st century.”

Smith sees more battles ahead as the world’s major governments are forced to rewrite rules written for another era. The Electronic Communications Privacy Act, the act over which Microsoft and the US are clashing, was introduced in 1986 by Ronald Reagan. That same year, Smith brought his first computer to work. There was no Facebook, no Fitbit; Microsoft Word was three years old. Most people still used typewriters.

The intersection of technology and law was peripheral. Now it is central, and so important that Smith believes some form of international framework will have to be worked out. He is optimistic that it will come, despite international differences.

“It’s become so important to society that it has become a mainstream topic. It makes it even more important to think now about the values that are at stake because it affects us all in profound ways,” he says.

“Ultimately it’s the governments that must make the decisions but we in technology have a role to play in trying to ensure that the debate is well informed and that we think broadly about our responsibilities as well.”