Duckduckgo Quietly Banishes a Bunch of Bangs

04 Dec 2018

DuckDuckGo has been steadily growing in popularity as a search alternative to Google. It’s now included in Firefox’s and Safari’s list of search engines. DuckDuckGo has emphasized user privacy by promising to not build profiles on its users and as a consequence to show the same search results to all its users for a given query. Compare their approach to Google’s, which tailors results to its users, and tracks users across the internet with its advertising and web console platforms. So DuckDuckGo’s marketing department throws around the term “filter bubble” a lot. Actually, they recently released their own study on spreadprivacy.org: Measuring the “Filter Bubble”: How Google is influencing what you click

I myself use DuckDuckGo, and applaud their approach to search.

I also use their bangs quite a lot. What are bangs?

duckduckgo.com/bang:

Bangs are shortcuts that quickly take you to search results on other sites. For example, when you know you want to search on another site like Wikipedia or Amazon, our bangs get you there fastest. A search for !w filter bubble will take you directly to Wikipedia.

Basically, they’re a built-in way to quickly search your favorite sites right from your search bar. They are a little-known but powerful feature of DuckDuckGo. For example, !wiki to search Wikipedia, !wayback to look for a page that has gone down on the Internet Archive, !yt for Youtube, or even !g if I want to get results from Google. If there’s a site you want to see on there that isn’t, you can ask them to put it up for you.

The silent purge

On November 29th, 2018, the search engine DuckDuckGo executed a massive cleanup of its “bangs.” I noticed when the bang !lg for Library Genesis went missing. Library Genesis is a host for a massive library of books. It’s where I get a lot of my reading material.

And I’m not the first to notice. It turns out a lot of sites have been removed, including Sci-Hub (a site facilitating download of academic articles), scores of torrenting sites, TV and movie streaming sites, subtitle hosts, and more.

To be clear, nothing is being censored from search results. It’s just that a lot of handy search shortcuts to specific sites have been removed.

DuckDuckGo didn’t announce any changes to their bang list. However, user tagawa on Reddit, a DuckDuckGo “Staff” according to the subreddit flair, has made some comments about the change in response to user upset.

Comment on /r/duckduckgo:

My apologies for the poor communication regarding this. Bangs had been neglected for some time, and there were tons of broken ones. As part of the bang clean-up, we also removed some that were pointing to primarily illegal content. They’re still available in our search results, and some browsers have custom search shortcut features that can be set up to replicate bang functionality locally.

It may not seem like so at first blush, but it is very different legally if it is a bang vs. in the search results because the bangs are added to the product by us explicitly, and can be interpreted legally as an editorial decision that is actively facilitating that site and its content. We operate globally, as do bangs, and products that actively facilitate interaction with illegal content can have us and our employees face significant legal liability, and jeopardizing the entire service. Again, sorry for not making this clear.

Removing dead sites is a no brainer, no debate there. And clearly, the bulk of the pruning was with the intent of minimizing legal liability for DuckDuckGo. However, the timing on the DuckDuckGo study (December 4th) is likely to be related to the changes to DuckDuckGo that preceded it by only days. The news cycle on this study is still churning (see, for example, Google’s response to the study).

How many bangs are gone?

Curious, I went looking for a list of the damned. It’s not out there, as far as I can tell. There is a directory of current bangs, but with no changelog. What I did find was a helpful series of data files in JSON format, containing every single bang, versioned 232 to 242, dated October 9th 2017 to December 4th 2018. For example, this is the most recent file, updated two days ago.

I compared them with a little python script (repository with files here). Here is the shortened output for changes made in the past month:

compare v239 to v240: v239: 2018-10-15 11:31:35 v240: 2018-11-29 13:34:47 number in v239: 11414 number in v240: 9886 removed: 1507 added: 59 compare v240 to v242: v240: 2018-11-29 13:34:47 v242: 2018-11-29 13:34:48 number in v240: 9886 number in v242: 9886 removed: 1 added: 3 compare v242 to v243: v242: 2018-11-29 13:34:48 v243: 2018-12-04 10:25:56 number in v242: 9886 number in v243: 9886 removed: 1 added: 1

That adds up to 1509 bangs banished. Here’s a full list of added and removed bangs.

If you just can’t do without the bangs

There’s Jive Search, which has many if not all of the old bangs from DuckDuckGo. Thanks to Reddit user yufsdfsdfsdffds (that’s a mouthful!)