It's tough being a government these days; who has the energy to clean up the Internet after a hard day's work bailing out the financial sector? Not the Australian government, it seems. Rather than actually doing something about illegal content, they just make a list of it and tell ISPs to filter everything that's on the list. Sidestepping the murky political details and—for the moment—the civil liberties problems inherent in this approach, let's take a closer look at the technical aspects of such a plan.

In the Internet Service Provider Content Filtering Pilot Technical Testing Framework document, the Australian Government Department of Broadband Communications and the Digital Economy provides some details about what it wants ISPs to do in a pilot project. The main part is that ISPs who are interested in participating in the pilot will test solutions for filtering a list of at most 10,000 URLs on a blacklist maintained by the Australian Communications and Media Authority, a regulator not unlike the FCC. "Prohibited online content" includes what you would imagine, but also your garden variety porn (yes, the stuff they broadcast over the air on public TV in the Netherlands), and under special circumstances even R-rated movies. Filtering URLs on the ACMA blacklist is a mandatory part of the pilot, though additional filters that aren't clearly specified are optional.

So how would an ISP go about blocking certain URLs from being accessed by its customers?

Filtering: harder than it sounds

First, a little refresher on how the Internet works. It all starts when a user types or clicks a URL. A browser or other application then looks up the domain name in the URL through the Domain Name Service. This usually happens through a DNS server operated by the ISP, but that's not necessarily the case. URLs can also contain IP addresses, avoiding the need for a DNS lookup altogether. Then, the browser starts sending packets to the IP address returned by the DNS. It is of course the ISP's job to make the packets flow in the right direction using the global routing system.

The first place where blocking can happen is on the user's computer. However, unless the Australian government is prepared outlaw open source software and administer all of its resident's computers, this isn't going to work.

The next option is the DNS. Filtering in the DNS is doable, and has been done in the past. This should work well for most users, but it doesn't take too much tech savvy to configure an unfiltered DNS server, bypassing the ISP's filters. Another problem with DNS filters is that a single domain name may host both blocked and unblocked URLs. For truly illegal content this usually isn't much of a problem, but a popular technique for dodgy content is to publish content that is prohibited in the target jurisdiction on a big server elsewhere, where the content in question is legal. Then the blockers are faced with the dilemma of whether to block a popular domain or let the offending content through.

It's also possible to filter packets. In this case, the workaround is installing a proxy. This is not as easy as configuring different DNS addresses, but it's certainly doable. (The obvious counter action by the government would then be to block the proxies.)

However, filtering packets, or making them disappear by manipulating the routing system, has the same problem as DNS-based filters: a single address may host both legal and illegal content. And it's worse in the sense that many different DNS names may resolve to a single IP address. A common technique for hosting questionable content is to use a large number of servers with very different IP addresses and let the DNS cycle through these addresses in quick succession, making it hard to determine which addresses host the content in question. (And maybe throw in Google's addresses once in a while so those get blocked as well?)

Another issue is that the government set an upper limit of 10,000 URLs. This gives bad actors an obvious way to defeat the system: simply host prohibited content on more than 10,000 URLs. A DNS-based filter can probably be made to work with arbitrarily big blacklists, but any system that requires firewall rules or routing table entries to block addresses will be limited to something in the order of 10k blocked addresses or address ranges—simply upping the limit won't work because the hardware would get too expensive.

A different approach to filtering is to intercept all packets and use deep packet inspection to determine if they're going to or from a blacklisted URL. This has two downsides: it obviously doesn't work for encrypted sessions, and it doesn't scale.

Even medium-sized ISPs have many 1Gbps links, and the larger ones have 10Gbps links. At 10Gbps, a router, switch, or firewall has about 400 nanoseconds to decide what to do with a packet—not enough time to run through a list of 10,000 URLs. And that's assuming that the target URL is conveniently present in a single packet, rather than having one half of the URL in one packet and one in another packet—and what happens when the second half is actually transmitted first?

So the DPI equipment must do full TCP/IP processing and reconstruct TCP sessions from the packets flowing by. This can (maybe) work at 1Gbps speeds, but even then it requires hefty boxes, of which a big ISP would have to deploy a good number. And did I mention that simply using HTTPS defeats this type of filtering completely?

Opting-out

An additional complication here is that the Aussie government is interested in letting users opt out of (part of) the filtering. Users can turn off the porn blacklist, but all Australians will still be subject to a filter on "illegal content."

This makes certain types of filtering a lot harder. As long as you have the routing table slots, it's easy to instruct routers to send packets for a certain IP address to the "null" interface so they are filtered. However, this is a binary thing: packets from all users are filtered, or packets from all users are allowed through. Setting up two different filter levels makes everything more difficult.

My conclusion: this isn't going to work. There's no way to build a filter box that can filter all the URLs where porn is hosted throughout the Internet. A DNS-based filter that helps naive users avoid being confronted with explicit content would probably work to a certain degree. An IP-based filter for a small amount of very illegal content—that would be the stuff that even the spam hosters in China don't want on their servers—may also work. But anything more ambitious than that is certain to fail; either it won't work very well, or it will bankrupt the ISPs.

As for the ISPs, they tend to agree.

Here's an idea: if the Australian government actually finds child porn, nuclear bomb making manuals, and the like on the Internet, why not do their best to find the perpetrators and put them behind bars? That way we get to keep our free speech and have less crime and terrorism, rather than less of the former without actually reducing the latter. Then again, imposing restrictions on what local taxpayers can do is a lot easier than tracking down and rounding up international criminals and terrorists, and the filtering plan is moving forward despite the massive and fairly obvious drawbacks.