Jun 2 2015

Auditing GitHub users’ SSH key quality

Translations are available in: русский (thanks to vectorified)

If you have just/as of late gotten an email about your keys being revoked, this is because of me, and if you have, you should really go through and make sure that no one has done anything terrible to you, since you have opened yourself to people doing very mean things to you for what is most likely a very long time.

A little known feature of GitHub is the ability to look at the public SSH keys that other users have set to be authorised on their account (for example https://github.com/torvalds.keys)

This is a great debugging feature and in addition a great way to share SSH public keys (for example when you are giving someone an user account on a server of yours, and you trust GitHub not to tamper with the contents.)

However one of the other side effects of this is that it means that everyone can see your public keys, and if someone cares enough, collect a massive database of everyone’s SSH keys.

I took a stab at this in 2013 but found that too many people didn’t use GitHub in SSH mode and thus had no keys set. This time however (with a new program that used the events api) I found that the majority of active users had some SSH keys in there.

I started running the streamer on 2014-12-27 and now 2015-01-09 I have quite a few keys:

mysql> SELECT COUNT(*) FROM `keys`; +----------+ | COUNT(*) | +----------+ | 1376262 | +----------+ 1 row in set (0.00 sec)

Now the first thing to test is how many people don’t have keys:

mysql> SELECT COUNT(*) FROM `keys` WHERE `key` = ''; +----------+ | COUNT(*) | +----------+ | 142180 | +----------+ 1 row in set (0.38 sec)

That’s a surprisingly low number. This means that only around 30% of GitHub users who have committed haven’t got SSH public keys in their account.

If we break it down into key types we see that RSA is by far the most used (not a suprise since has been the standard for a long time now):

mysql> SELECT COUNT(*) as cnt,LEFT(`key`,17) as keytype FROM `keys` GROUP BY keytype ORDER BY cnt; +---------+-------------------+ | cnt | keytype | +---------+-------------------+ | 21 | ecdsa-sha2-nistp3 | | 210 | ssh-ed25519 AAAAC | | 336 | ecdsa-sha2-nistp2 | | 502 | ecdsa-sha2-nistp5 | | 27683 | ssh-dss AAAAB3Nza | | 142180 | | | 1205330 | ssh-rsa AAAAB3Nza | +---------+-------------------+ 7 rows in set (0.00 sec)

Still a large amount of users who have DSA keys, and a pretty low count of users who have ECDSA keys.

As for everything else, here is the (observed) key bit-age share used by GitHub users:

A trip down scary lane

After this I wanted to figure out how many keys were dangerously weak to the point where cracking was trivial.

I wrote a program to parse all the keys in the database and populate the “Bits” column on the database.

What I found was not as bad as it could have been, but it was still pretty bad. (I’ve redacted full public keys and full user names.)

mysql> SELECT LEFT(Username,3),LEFT(`key`,30),Bits FROM `keys` WHERE Bits < 1023 AND Bits != -1 ORDER BY Bits ASC; +------------------+--------------------------------+------+ | LEFT(Username,3) | LEFT(`key`,30) | Bits | +------------------+--------------------------------+------+ | Mar | ssh-rsa AAAAB3NzaC1yc2EAAAABJQ | 256 | | Gl2 | ssh-rsa AAAAB3NzaC1yc2EAAAABJQ | 256 | | rdp | ssh-rsa AAAAB3NzaC1yc2EAAAABIw | 512 | | MrM | ssh-rsa AAAAB3NzaC1yc2EAAAABJQ | 512 | | pun | ssh-dss AAAAB3NzaC1kc3MAAABBAM | 512 | | ryo | ssh-rsa AAAAB3NzaC1yc2EAAAABIw | 512 | | ilj | ssh-dss AAAAB3NzaC1kc3MAAABBAO | 512 | | del | ssh-dss AAAAB3NzaC1kc3MAAABBAI | 512 | | meg | ssh-dss AAAAB3NzaC1kc3MAAABBAK | 512 | | gal | ssh-rsa AAAAB3NzaC1yc2EAAAABJQ | 767 | | gol | ssh-rsa AAAAB3NzaC1yc2EAAAABJQ | 767 | | jnw | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | ken | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | txc | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | Pha | ssh-rsa AAAAB3NzaC1yc2EAAAABIw | 768 | | too | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | hir | ssh-dss AAAAB3NzaC1kc3MAAABhAO | 768 | | lur | ssh-rsa AAAAB3NzaC1yc2EAAAABIw | 768 | | lik | ssh-dss AAAAB3NzaC1kc3MAAABhAN | 768 | | mku | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | tra | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | mas | ssh-dss AAAAB3NzaC1kc3MAAABhAO | 768 | | fox | ssh-rsa AAAAB3NzaC1yc2EAAAABJQ | 768 | | and | ssh-rsa AAAAB3NzaC1yc2EAAAABIw | 768 | | aus | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | Sky | ssh-rsa AAAAB3NzaC1yc2EAAAABIw | 768 | | coc | ssh-dss AAAAB3NzaC1kc3MAAABhAI | 768 | | coc | ssh-dss AAAAB3NzaC1kc3MAAABhAI | 768 | | coo | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | Vis | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | Vis | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | sai | ssh-dss AAAAB3NzaC1kc3MAAABhAJ | 768 | | and | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | Mat | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | x89 | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | nao | ssh-dss AAAAB3NzaC1kc3MAAABhAO | 768 | | exp | ssh-dss AAAAB3NzaC1kc3MAAABhAJ | 768 | | Ric | ssh-rsa AAAAB3NzaC1yc2EAAAABIw | 768 | | bil | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | kat | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | Tyr | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | jac | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | asu | ssh-rsa AAAAB3NzaC1yc2EAAAABIw | 768 | | bou | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | bou | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 768 | | bey | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 770 | | ser | ssh-rsa AAAAB3NzaC1yc2EAAAABIw | 799 | | akD | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 800 | | ccw | ssh-rsa AAAAB3NzaC1yc2EAAAABJQ | 919 | | sky | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 1000 | | dtg | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 1000 | | phi | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 1000 | | phi | ssh-rsa AAAAB3NzaC1yc2EAAAADAQ | 1000 | +------------------+--------------------------------+------+ 53 rows in set (0.34 sec)

There are 9 keys that stick out to me here. There are 2 keys with only 256 bits to them and another 7 that only have 512.

512 bit keys have been known to be factorable in less than 3 days. The main example of this is the Texas Instruments calculator firmware signing key that was broken, allowing the modding community to upload any firmware that they wanted.

I tried on my own to make a 256 bit key and factor it, and the process took less than 25 minutes from having the public SSH key to the factoring of primes (on a subpar processer by today’s standards, and then a few more minutes to transform those back into a SSH key that I could log into systems with.

This risk isn’t only real if someone had gathered together top of the line mathematicians or supercomputers worth of power, the 256 bit key I factored was factored on a i5-2400 in 25 mins.

The Debian bug bites again

After realising I have a public key database of most users on GitHub, I remembered back to the May 2008 Debian OpenSSH bug, where the randomness source was compromised to the point where the system could only generate one of 32k keys in a set.

I used g0tmi1k’s set of keys to compare against what I had in my database, and found a very large amount of users who are still using vulnerable keys, and even worse, have commit access to some really large and wide projects including:

Spotify’s public repos (and any private repos those employees had access to)

Yandex’s public repos (and any private repos the person had access to)

Crypto libraries to Python

Django

Python’s core

gov.uk public repos (and any private repos the person had access to)

Couchbase (and any private repos the person had access to)

A ruby gem that is used on a large amount of CI systems (compromise of that, means compromise of your build server, and possibly your internal network)

I have not tested if I could pull any private repos on these.

The most scary part of this is that anyone could have just looped through all of these keys just trying to SSH into GitHub to see the banner it gives you.

It would be safe to assume that due to the low barrier of entry for this, that the users that have bad keys in their accounts should be assumed to be compromised and anything that allowed that key entry may have been hit by an attacker.

GitHub response

While I would have liked to get another listing on the bounty hunters page, it seems that GitHub don’t consider this as something that fits in there. I can see where they are coming from, as this is primarily a user based mistake, however we generally protect customers from these kind of mistakes, like for example Amazon AWS scanning GitHub for API keys

I understand the response (or in this case, a lack of a formal one), but I think at some stage companies should have some protection for their users doing silly things, and discovering things that should be expected of the service (since OpenSSH does this for you itself) not happening should be flagged as a security issue.

Timeline

[27th December 2014] Key Crawl Started

[28th Feb 2015] Disclosed low bits issue to a peer at GitHub

[1st March 2015] Discovered the weak Debian Keys (and disclosed)

[5th May 2015] Debian keys revoked, emails sent out

[1st June 2015] Weak and low quality keys revoked

I will be detailing more of my findings in the one of my upcoming posts in the next few weeks/months. You may want to consider using the site’s RSS feed