For more than a year, the intelligence services of various authoritarian regimes have shown an intense desire to know more about what goes on in an office building on L Street in Washington DC, six blocks away from the White House.

The office is the HQ of a US government-funded technology project aimed at undermining internet censorship in countries such as Iran and Syria. And so every week – sometimes every day – email inquiries arrive there that purport to be from pro-democracy activists in those places, but which, the recipients are confident, actually come from spies.

This is international espionage at its most elementary: pretend to be a sympathiser, ask for more details, and just maybe, if you're lucky, some unsuspecting intern or temp will forward you the secret plans. It's a long shot, but it's much easier than intercepting phone calls, cultivating double agents or infiltrating opposition groups.

This kind of dissimulation has been going on forever. But if you wanted a sign of the strangeness of this moment in history, you could not hope for one more telling than this: the spies who email the L Street office always get their questions answered, promptly and in detail.

"Everything is completely open – all the code, everything," says Sascha Meinrath, who runs the project, codenamed Commotion Wireless, at the New America Foundation thinktank.

"If it's a really good question, we'll put it on our frequently asked questions page. Hard questions help us get better."

The fact that Meinrath and his fellow geeks are committed to radical transparency is not surprising: that has been the ethos of grassroots internet culture since the beginning. (Meinrath used to work for Indymedia, the citizen-publishing site that grew out of the Seattle World Trade Organisation protests of 1999.) But what certainly is a surprise is the fact that the US state department is providing such people with millions of dollars.

Such are the strange bedfellows of the era: on one hand, the US government is funding Commotion Wireless, which will enable anyone with a smartphone to connect with other smartphones, forming a "mesh network" – an impromptu internet – to communicate by stealth, regardless of efforts to monitor or shut down traffic.

On the other hand, the FBI recently distributed flyers warning that the use of "anonymisers", encryption or "multiple cell phones" might be "indicators of terrorist activities".

In January, Meinrath's team gave Commotion a trial run at the Occupy DC encampment – meaning the state department was effectively facilitating the protest.

Multiple fractious arguments about the internet dominate headlines these days, but ultimately they are all battles in a single war.

Debates about China's "Great Firewall", about WikiLeaks, about intellectual property and piracy, about the UK government's plans to expand email surveillance powers, about Facebook and privacy, pornography and terrorism, Anonymous and LulzSec, all boil down to the fundamental question of whether the internet should be "open" – a place of unfettered expression, self-organising order and plenty of chaos – or "closed" and controlled, subjugated to existing frameworks of politics, policing and law.

Common sense might dictate some middle way, which is the official line of the Foreign Office: cracking down on criminals but not democracy activists; letting respectable businesses harness the web's potential, without letting copyright anarchists such as the Pirate Bay do the same.

But it is by no means clear that this is possible. Can you facilitate "Twitter revolutions" without also facilitating organised crime? Can you weaken dictators without weakening democratic governments? Can you support an open internet abroad but not net neutrality – which stops internet providers making it easier or harder to access certain sites – at home? Can you be open and closed as it suits you? Or must you make a choice?

For a certain breed of digital utopian, these were always supposed to be irrelevant arguments: eventually, they argued, the inherent logic of the internet would defy all efforts to control it.

Instead, China has not only developed a vast and highly effective system of censorship over its more than 500 million internet users – combining human and digital surveillance, and permitting closely watched clones of non-Chinese social networks to thrive – but is exporting its thinking and technology to other countries. UK and US firms, meanwhile, have been among those implicated in marketing surveillance technologies used to censor the web in China and Egypt. None of the easy assumptions of the technolibertarians – that the democratic world would champion openness, or that openness would win – any longer seem to apply.

Nowhere are the dilemmas and the possibilities better illustrated than in the case of the US state department, which under Hillary Clinton has become a full-throated supporter of "internet freedom" as a human right – while simultaneously trying to avoid the impression that it is fomenting regime change in countries such as Saudi Arabia and Bahrain, American allies that heavily censor the web. (And also while opposing that other pioneer of internet freedom, WikiLeaks, with its penchant for leaking secret state department cables by the truckload.)

"We want to keep the internet open for the protester using social media to organise a march in Egypt," Clinton said in a major policy speech last year. "[For] the college student emailing her family photos of her semester abroad; the lawyer in Vietnam blogging to expose corruption; the teenager in the US who is bullied and finds words of support online; for the small business owner in Kenya using mobile banking to manage her profits; the philosopher in China reading academic journals for her dissertation … internet freedom is about defending the space in which all these things occur."

Between 2008 and 2011, Congress authorised $76m (£48m) of spending on such efforts – a sign of serious intent, though admittedly very little compared with the forces it is intended to fight.

The ethos of openness was central to the vision of the internet's founders – academics, mainly, who trusted each other and were not focused on the network's global growth. "The primary design principle underlying the web's usefulness and growth is universality. When you make a link, you can link to anything," its founder, Sir Tim Berners-Lee, wrote in 2010, in a polemic targeted just as much at the "closed worlds" of Facebook and the Apple App Store as at Iran's "Halal internet".

"Totalitarian governments aren't the only ones violating the network rights of their citizens," Berners-Lee said, citing a 2009 French law, which in its original form would have allowed the authorities to cut off a household's internet access for a year based on nothing but a media company's claim that its copyright had been violated. (Now, a court must agree as well.) "Given the many ways the web is crucial to our lives and our work, disconnection is a form of deprivation of liberty," Berners-Lee added.

In practice, though, even well-meaning efforts at openness can aid those who seek to exert tight controls over the internet. That seems to have been the case with the fiasco of Haystack, a project launched amid the 2009 Iranian election protests by a US programmer, Austin Heap, which promised to allow dissidents in Iran to circumvent state censorship.

The state department endorsed the project (and Heap won a Guardian innovation award for his efforts) but Haystack was abruptly withdrawn amid allegations that its poor design exposed activists to possibly fatal harm, revealing information about them to Iran's state monitors. "Given that history, it's not that [activists] are paranoid," says Meinrath. "It's just good sense – the history is that when we've done these things, it's been very bad!"

The radical openness of Commotion is an effort to prevent a rerun of those problems. Anyone can inspect the code, testing it for flaws. Being exposed to the prying eyes of foreign governments means it must adapt to withstand such scrutiny; those using it can accurately grasp the risks, and need not rely on faith in America's goodwill, or on software developers keeping certain things secret. "Instead of trying to hide security problems, we're trying to actively expose them," says Meinrath. "It makes for better relations with people who are going to be putting their lives on the line."

Last year, when the New York Times mentioned Commotion, Iran's intelligence minister, Heidar Moslehi, declared that his country had devised ways to sabotage the project. But the logic of the internet has a strange way of frustrating such strategies. "We got flooded with messages from people in Iran wanting to know how to download it," Meinrath said. "He did more to spread word about our technology than we ever could have done."

And once a handful of people have downloaded it in any given location, its spread – should they find it useful – is virtually guaranteed. When completed, Commotion could allow demonstrators to plan their real-world gatherings free of state interference.

Used in conjunction with a "delay-tolerant" Twitter application, it could let people continue to use the social network despite censorship. Then, as soon as any node on the network managed to connect to the wider internet, it could push out all backlogged tweets for the world to see.

Another project receiving state department backing is a "panic button" app for smartphones, called InTheClear, which allows the contents of a phone to be speedily erased, and prewritten texts sent to specific contacts, when the owner is taken into custody.

But many of the "internet freedom" dollars go towards digital workshops held globally – some publicly acknowledged, but most secret – at which more than 5,000 activists have so far been trained.

"One blogger was a Tunisian and he'd been active before the revolution, before Ben Ali left," a senior state department official said at a briefing, explaining one benefit of the workshops. "And when he got to the training programme, the trainer looked at his laptop and found that he had scores of viruses on his laptop, including … keylogger software [that] was sending back every single stroke he typed."

Scepticism about such efforts is widespread. "The impression I have, to be honest, is that [the internet freedom agenda] is mainly a public relations thing," says the Cairo-based Middle East analyst Issandr El Amrani, founder of the news and commentary site Arabist.net. "It's a big idea that comes at relatively low cost. And while the administration has been helpful in Syria" – condemning Syrian web crackdowns, and energetically using Facebook and other networks there itself – "we haven't seen Clinton or Obama push aggressively for internet freedom in China. We haven't seen any real efforts to ban the export of software to let governments spy on citizens."

In its public pronouncements about its activities, Washington toes a careful line, insisting that "internet freedom" is a politically neutral human-rights effort, not a way of undermining authoritarian regimes – some of which, after all, are officially allies.

"The biggest negative critique of the state department in this area is built entirely around a straw man," insists Alec Ross, Clinton's senior adviser on innovation. "It's rooted in this false premise that we view this principally as an engine for regime change in authoritarian societies. It's just a false critique."

To the internet commentator Clay Shirky, by contrast, America's interest in using internet freedom to undermine autocracies is "acknowledged internally and understood externally. It's just that nobody can come out and say it."

In the end, though, it may not matter who acknowledges what, or how the state department views things: the forces they are helping to unleash are inexorably exposing the contradictions in how democratic governments approach the question of online openness.

"It's very difficult to weaponise social media," says Shirky – to choose the battles you want to wage, so as to support openness in Iran, say, while overlooking abuses in South Korea or Thailand, and not aggravating the Motion Picture Association of America (MPAA), for whom a free and unmonitorable internet is less a human right and more a grave business threat.

Shirky says: "The real threat to internet freedom isn't from Iran saying we're going to disconnect and build an alternate internet – that's a desperate act. The real threat comes from the MPAA [Motion Picture Association of America] and our allies. We should be much more worried about what's going on in South Korea than in North Korea."

In the utopian version of this storyline, by collapsing governments' abilities to promote freedom in some countries but not others, or in the political realm but not the commercial one, openness may force governments to pursue a more principled kind of politics.

"We've always been able to say 'oh no, we don't mean you, Saudi Arabia,'" said Shirky. "And now our bluff has been called."

Sitting in the wood-lined rare books room of the state department library, Alec Ross, a 40-year-old West Virginian who was closely involved in Obama's presidential campaign, before being hired by Clinton, dismisses many of these criticisms, but paints internet freedom in similarly grand historic terms.

"I think the fight over an open internet is today's version of a battle between open and closed that has been raging now for 2,300 years," he says, sketching a sweeping tale of clashing forces dating back to the third century BC, taking in the burning of the library at Alexandria, the invention of the number zero, the dark ages and the impact of the printing press.

In this view, the crusade for openness rises above mere politics.

"What distinguishes the Bush 'freedom agenda' from Clinton's internet freedom agenda is that [Clinton's] is less politically deterministic. It's rooted in individual rights, not a presumed political outcome. There are lots of grantees of ours whose members have political views that would be well outside the mainstream in this room. But that's a good thing!"

Clinton says she will step down this year, but America's role in the open internet's future may just be starting to get interesting.

If the promise of neutrality is not kept – if the policy is used instead only to further narrow US interests – then it could alienate pro-democracy activists and harm US credibility abroad.

On the other hand, if the promise is kept, it could unleash forces of openness too great for any nation to control, and with no guarantee that movements thus empowered will be friendly to America. Commotion Wireless may prove to have been presciently named.

• Explore the seven-day special series on the Battle for the internet