Hi, I’m Mark Nottingham. I currently co-chair the IETF HTTP and QUIC Working Groups, and am a member of the Internet Architecture Board . I usually write here about the Web, protocol design, HTTP, and caching. Find out more .

If You Can Read This, You're SNIing

When TLS was defined, it didn’t allow more than one hostname to be available on a single IP address / port pair, leading to “virtual hosting” issues; each Web site (for example) now requires a dedicated IP address.

HTTP solved this problem with the Host request header; TLS did it with the Server Name Indication (SNI) extension. You can read more about it at the Wikipedia page.

The problem is that some clients – most notably Windows XP of any flavour, Android 2.x, and Java up until 1.7 – don’t send SNI information. Since requiring SNI effectively orphans these clients, people have been reluctant to require it on their Web sites, even though it saves precious, precious IPv4 addresses.

An Experiment

However, I believe in living on the bleeding edge, want to get more deployment experience with the protocols we’re developing, and most importantly, don’t want to give Rackspace the extra $2 a month for another IP address.

So, both this site and redbot.org have switched to HTTPS URIs, and require the use of SNI to access them. My audience tends not to run Windows XP, so I think that’s a reasonable thing to do, and somebody has to take the first step.

(BTW - for those who think this site may have got a bit slower, it’s not because of the TLS – it’s because I moved the server from Texas to Sydney.)

Along the way, I tried to get reasonably close to best practice for both security (as outlined by bettercrypto.org) and performance (see Is TLS Fast Yet?). Once things settle down a bit more, I’ll turn on Strict Transport Security too.

The SNI configuration for Apache 2.4 was a bit fiddly, mostly because I didn’t want to set a “default” site for clients that don’t send SNI; I wanted a hard fail. The relevant config ended up looking like this:

SSLStrictSNIVHostCheck on ErrorDocument 403 "TLS SNI Required." Listen 443 <VirtualHost *:443> ... SSLStrictSNIVHostCheck on <Directory ...> ErrorDocument 403 default SSLRequireSSL SSLOptions +StrictRequire </Directory> </VirtualHost>

In my testing, this generates a 403 with the message “TLS SNI Required.” for clients that don’t send SNI information.

403 really isn’t the greatest status code for this semantic, so I filed a bug with Apache to make it more appropriate, and to give more flexibility in the error document (since most people’s reaction to that message will be, deservedly, “huh?”).

Stories from the Logs

Interested to see what the effect of requiring SNI would be, I set my server up to log the server name presented:

LogFormat "%h %t %D %u %{SSL_TLS_SNI}e \"%r\" %>s %R %b %X %I %O \"%{Referer}i\" \"%{User-agent}i\"" site

Clients that don’t present SNI get logged as “-“, and of course get a 403.

After a bit of command-line grep / sort / uniq foo, here is a list of UAs that didn’t present SNI information. In some cases, I’ve trimmed the “helpful” compatibility information out (e.g., Mozilla/blah…). I’ve also taken out the browsers that don’t support SNI (almost all on XP or older versions of Windows).

Sites, Services

These are online sites and services that made fetches without SNI, where I can’t determine what their implementation language is.

CareerBot/1.1; +http://www.career-x.de/bot.html

FeedBooster; +http://feeds.qsensei.com

FlipboardProxy/1.1; +http://flipboard.com/browserproxy

FlipboardRSS/1.1; +http://flipboard.com/browserproxy

Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)

Hatena::Bookmark/2.00

James BOT - WebCrawler http://cognitiveseo.com/bot.html

kouio.com RSS reader - 5 subscribers

Kraken/0.1; http://linkfluence.net/; bot@linkfluence.net

livedoor FeedFetcher/0.01 (http://reader.livedoor.com/; 26 subscribers)

LiveJournal.com (webmaster@livejournal.com; for http://www.livejournal.com/users/mnot_main/; 2 readers)

NewsBlur Feed Fetcher - 263 subscribers - http://www.newsblur.com/site/1139/mnots-blog

PercolateCrawler/4 (ops@percolate.com)

pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm

PushBot/1.1; +support@appnotifications.com

TBRSS/1 (https://tbrss.com/; 1 subscriber)

woriobot +http://worio.com

Seach Engines

Really just a sub-category of Sites and Services, search engines that didn’t send SNI include:

Baiduspider/2.0; +http://www.baidu.com/search/spider.html

BazQux/2.4; +https://bazqux.com/fetcher; 9 subscribers

BlogSearch/2 +http://www.icerocket.com/

bingbot/2.0;+http://www.bing.com/bingbot.htm

Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot

EasouSpider; +http://www.easou.com/search/spider.html

Exabot/3.0; +http://www.exabot.com/go/robot

MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0

MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+

msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)

msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)

Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp

YandexBlogs/0.99; robot; B; +http://yandex.com/bots)0 readers

YandexBot/3.0; +http://yandex.com/bots

YandexImages/3.0; +http://yandex.com/bots

These ones look like desktop or mobile tools, but I can’t tell the implementation language.

Feedreader 3.14 (Powered by Newsbrain)

HTTrack Website Copier/3.0x (offline browser; web mirror utility)

Instapaper for Android 3.0

Pogo Reader (feed reader)

RSSOwl/2.2.1.201312301258 (X11; U; en)

Seznam screenshot-generator 2.0; +http://fulltext.sblog.cz/screenshot/

Unknown

Not sure about these. shrug

Gregarius/0.6.0 (+http://devlog.gregarius.net/docs/ua)

http_get

kjb-RSS/5.13

MetaURI API/2.0 +metauri.com

NerdyBot

newspaper/0.0.6

news_publisher_rss_spider/1.0

rawdog/2.19

RTWT http://rtwt.aloodo.com/

RubeGoldbergPostalService/1.0

seo/1.0

u12Bot/5.0; +http://u12files.com/

Language-Specific

Finally, agents that can be tracked to a specific implementation language.

Java

Java supports SNI since 1.7, but of course there are still some old clients out there:

CommaFeed/1.0 (http://www.commafeed.com)

JDK/1.0.2 Java/1.6.0

Perl

Apparently, Perl IO::Socket::SSL has supported SNI for a while now, provided that the underlying OpenSSL does. These people should just need to upgrade OpenSSL and rebuild their Perl.

libwww-perl/5.834

W3C_Validator/1.3 http://validator.w3.org/services

PHP

PHP gave control over SNI in 5.3.2, but if they’re using Curl, it’s just a matter of getting the right versions.

MagpieRSS/0.72 (+http://magpierss.sf.net) (Tiny Tiny RSS/1.4.1)

PECL::HTTP/1.7.1 (PHP/5.3.10)

Ruby

It looks like Ruby has supported SNI for a while now. Not sure what’s going on here.

EventMachine HttpClient

Feed2Imap v1.0 http://home.gna.org/feed2imap/

Python

Later releases of Python 3 have SNI, and Python 2 should be getting it in the next few weeks, when 2.7.7 is released. Hopefully these folks will upgrade and this list will thin down soon:

bookie / (https://github.com/bookieio/bookie)

Feedjack 13.08.16 - https://github.com/mk-fg/feedjack

FeedValidator/1.3

LinkChecker/8.4; +http://wummel.github.com/linkchecker/

plagg/3.0 (+http://drbeat.li/py/plagg/)

pnntprss/0.01 +http://david.wragg.org/pnntprss/

Python-httplib2/$Rev$

python-requests/2.2.1 CPython/2.7.6 Darwin/13.1.0

Python-urllib/2.7

RED/1 (http://redbot.org/)

rss2email/2.71 +http://www.allthingsrss.com/rss2email/

UniversalFeedParser/5.1.3 +https://code.google.com/p/feedparser/

Conclusions

It’s early days, of course, but this makes me pretty hopeful; if Python gets fixed soon, it knocks out a fair bit of the list, and there’s a small-ish list of sites and services that needs to upgrade (which is dwarfed by the list of those that do support SNI, btw).