18F has quietly become the bleeding edge of the US federal government's adoption of open source software. Read about the benefits and challenges of open source going mainstream.

Image: 18F

Earlier this week, I saw what the future of building government services may look like when I stumbled upon a simple dashboard of projects-in-progress. The dashboard is hosted by 18F, the new development unit within the US General Services Administration.

18F, which explicitly seeks to tap into the success of the UK's Government Digital Services unit, is pursuing a similar strategy, trying to lure developers from Silicon Valley and the ranks of civic developers all over the country with a daunting mission: change how federal technology gets done, at a time when bad government websites now damage public faith in government. Behind the dashboard is 18F's GitHub account, which exemplifies a quietly revolutionary idea that the UK has been pursuing with great success: build beautiful digital services for the public, in public.

Love this approach to building out in the open. Major props to @18F. Lots here for other govs to copy https://t.co/rgKUZvYMGh h/t @digiphile -- Mark Headd (@mheadd) November 8, 2014

The use of GitHub by 18F and many other governments is why, incidentally, I thought the social coding startup was the most interesting company in government IT last year, with increasing activity week over week.

"Simply put, every metric we can possibly generate is a hockey stick going straight up at this point," said Ben Balter, government evangelist for GitHub, in an interview. Balter says that over 700 government organizations in 30 countries are now there, including 120 US federal agencies, 40 states, 30 cities, and 20 counties.

WordPress, the open source content management system and blogging software, now powers about 60 million websites, including 22% of the top 10 million sites on the planet and a number of .govs. Earlier this year, I talked about open source in government with Matt Mullenweg, the co-creator of WordPress and CEO of Automattic at the inaugural WordPress and government meetup in DC. Video of our interview and the question and answer session that followed is embedded below.

What's driving the adoption of WordPress, Drupal, and many other open source applications is a need to "do more with less," adopt modern technologies and development practices, and avoid more costly failures. The UK's executive director of digital, Mike Bracken, told me in February 2014 that this approach to building digital services is "vastly cheaper," a claim that 18F will be proving out in the years to come.

"Healthcare.gov was a watershed moment in the conversion," said Balter. "An inflection point. Geeks in government had been saying for a long time that traditional, heavyweight large dollar, long-term enterprise projects were far less resilient than more modern, open systems. The traditional management playbook has always been to throw more money and bodies at the problem. Healthcare.gov (among other examples) show that that's not the answer. No software is perfect, but today agencies realize that communicating more, not less, working openly, with shared tools, and shipping 0.1."

It's no accident that open source projects are driving innovation in multiple sectors and parts of government. The defense community has been a notable adopter of open source in many contexts, most recently in its battleships. Open source software and hardware will be core to adaptable platforms deployed to battlefields around the world. "If you can't hack it, don't pack it" is a mantra I've heard from more than one military veteran over the years, and one that's been heard at DARPA more than once.

This shouldn't be so exciting, but it is. After all, open source is widely adopted and mainstream in the modern technology sector, as my former colleague Roger Magoulas noted in 2012, changing not just code but models for work.

"A new class of innovative, widely adopted technologies has emerged from the open source culture of collaboration and sharing -- turning the old model of replicating proprietary software as open source projects on its head," he wrote. "Think Git, D3, Storm, Node.js, Rails, Mongo, Mesos, or Spark."

While open source's impact on the development community may be old hat to TechRepublic readers, it's still reverberating through government and media, both of which have legacy institutions and processes that have been slower to adapt to changing contexts. If 18F makes real progress, you'll know it by the strenuous objections you'll start reading in the trade press from US government contractors and growth in lobbying activity applied in Congress, just like the reactions that have been catalyzed by the growth of the Government Digital Service in the UK.

"There are still rearguard objections, about whether an operating system is secure or not," said Gunnar Hellekson, the Chief Strategist for Red Hat's US Public Sector group, in an interview. "The big challenge now is not legitimacy but where to apply open source. It's not monolithic."

That's true in multiple contexts. For instance, when I thought about how the "computer-assisted reporting" of decades past and data journalism differ today, my research concluded that open source was one of the core differences. I was delighted to find in Brazil that the thesis of professor Marcelo Trasel came to the same conclusion, as does the research of professor Mark Coddington.

In DC, 18F has quietly become the bleeding edge of the federal government's adoption of open source software because of the way they work, not just the code they use. Not only do they use the code and operating systems built upon open code, like millions of other developers and tech companies around the world, they've committed to developing free and open source software by default online, together. That means that the code created with taxpayer dollars is released back to the public, much in the same way open access to scientific research by government works.

"There's a clear progression that nearly every government agency goes through, from consuming open source, to publishing open source (as a one-way broadcast), to collaborating on open source (either their own projects, or others)," said Balter. "A similar progression is also seen from open source (geeks), to open data (geek consumed, but produced by wonks), and open government policy (fully wonks). Policymakers see the geek's tooling, realize the value of collaboration (and version control), and want to bring it into their own workflow. If your doctor takes a multivitamin every day, wouldn't you? To me, the idea of working more openly, regardless of format or form, within an organization, or with the public is the idea that we're seeing catch on. It's starting with open source, but that's just the start."

At 18F, starting with open source means that the public can go look at their work on Federal Election Commission data or read the unit's open source policy or track how a project on Freedom of Information Act requests is going. I use GitHub to follow what's happening with Project Open Data, one of the explicit outcomes of the open data executive order the president issued last year. It's messy, sometimes uncomfortable, and puts mistakes out the open, but it's absolutely fascinating. To say that this is all a bit different than the development process for other government websites, policies, or software applications would be to engage in a gross understatement.

"It used to be that mentioning open source to an agency CIO might conjure images of hippies out in California with tie-dyed laptops passing around code like an illicit substance," said Balter. "Today, that perception has changed with proprietary and purpose-built software being the face of legacy, Beltway IT, the way things have always been done, and open source representing modern development practices that's more often seen in the private sector. Would 18F's younger, faster, smarter aura work to the same degree if they primarily recommended mainframes?"

To be fair, 18F is building upon the success of other government agencies and decades of pioneers who have contributed to government IT's quiet open source evolution. Most recently, the Consumer Financial Protection Bureau explicitly aligned open source with its mission and published an open source policy that set a high bar for 18F to reach. In recent years, the State Department, NASA, the FCC, and the White House embraced open source software and platforms, but the history of open source in government goes at least back to the 1990s, as a review of a timeline highlights.

"Government has been using open source since before the World Wide Web was born," said Balter, "but it's been stuck in the sub-basement in a room strewn with Mountain Dew cans and Doritos crumbs. Open source in government is nothing new, it's just that the tooling is finally there to lower the friction cost to such a point that it's becoming easier to work together, than to work alone, and no longer limited to those who are comfortable with command lines laden with green on black text. It's becoming accessible. We're seeing that trend -- adopting open source philosophies and workflows and bringing them inside your firewall -- happening in large enterprises, both public and private sector. Heck, this week Microsoft of all people open sourced its foundational framework. We've come a long way."

In the near future, all of the US government should be able to refer to an official open source policy for the nation, one of the commitments made in the second U.S. National Action Plan on Open Government. (The White House declined to comment on progress made on this policy to date.) There are several challenges that this policy will have to address, and a couple that will depend on purely human drivers.

One challenge the policy will need to address is one of perception, as is continued uncertainty around security, despite the use of open source in the scientific, defense, and intelligence communities.

"Yes, there's still some bad open source, but on balance, it's as good, if not often better than proprietary alternatives, but perceptions haven't caught up, especially around the C suite," said Balter. "If you're using a mental schema from the 90s, in which open source is a fringe effort, you're going to develop IT like it's the 90s, mainframes and all."

Another set of challenges exist around how to handle not just maintenance but disclosures.

"There are lots of programs trying to figure out how to maintain projects over time," said Hellekson. "There's a relevant old joke: 'open source is free like a puppy is free.' Governments are figuring that out. If no community adopts the code, you have to maintain it, and when something like Heartbleed comes along, you have to deal with it. Government is not necessarily set up for that. When 18F publishes software, how do they handle vulnerabilities and disclosure? Do they report it to NIST? Is GSA set up to embargo disclosures, which is common practice in the private sector? These are boring topics unless you're a developer, but they're important."

There are many reasons that all government software is not open source yet, including some practical limits.

"I don't think anyone, except maybe the most radical proponents, thinks governments should open source all of the code they're using," said Hellekson. "There's a certain amount of value in releasing on principle, but operationally it's not practical. To release and get the most value, you need to have someone build community, maintain it, manage suggestions, patches, and teach people."

Another challenge is even more complex: a collective action problem. If a core part of online security infrastructure, like OpenSSL, is open source but the resources put into maintaining it are insufficient to keep it secure, who's responsible? After the Heartbleed bug headaches this past year, the OpenSSL Foundation received more support from members of the tech industry that use it. That's good news, but there is almost certainly a role for government institutions to play here elsewhere.

"The Department of Homeland Security has a partnership with Coverity to do regular audits of popular open source projects," said Hellekson. "That's meant to solve this collective action problem. No one is actively incentivized to look for security flaws. Do you want white hats finding and disclosing flaws rather than the black hats finding and selling them? We have vendors who will do their best, but for a lot of people, given the choice between fixing a bug, adding a feature, or looking for flaw, will they look for more work? Coverity runs through hundreds of projects and, if they find flaws, logs them with the project. That's paid for by the DHS and meant to improve the industrial base. This is a function of government: create incentives where the market can't summon one naturally."

A final challenge around open source adoption is in some ways, the thorniest, as it involves community and sustainability, something that every government entity will have to think through and nurture through human connections, not just policy.

The open source projects that are the most successful are the ones that are self-sustaining and move outside of government, said Hellekson. For example, he pointed to the Tor project, which originated in a U.S. Navy research lab, Apache Accumulo, OSSIM (Open Source Security Information Management), and Suricata, and expressed measured optimism that the Department of Energy would find a steward for the National Training and Education Resource (NTER).

The most important recent example of this dynamic I know of is OpenStack, one of the fastest growing open source communities in history. I first started tracking OpenStack in 2010, when I learned that it had grown out of NASA's Nebula project. The fact that an open government innovation from NASA fueled the launch of OpenStack remains underappreciated to this day (it's classified under "technology transfer"), but the venture-backed Nebula One hardware that combinesFacebook's Open Compute Platform remains a startup to watch.

"The real successes are when government gets to use the software, but doesn't have to keep it together with both hands," said Hellekson.

Also see