How to Harden SSH with Identities and Certificates

Summary It's 2014 and remote unix shells are as popular as ever. Which is great, except that people don't seem to be using the more advanced security features nearly enough. So, what are they and how can you use them?

Introduction

Whether you just need to feel in power or you actually use shells for day-to-day tasks, the Secure Shell [SSH] is probably the most important administrative access tool to your servers. It's also one of the least secured mission-critical services on most UNIX servers. Why? Because for some reason people are still using mere passwords to protect their root accounts. That's not quite as bad as using telnet, but not by too much. You might as well be using plain FTP to transfer data to your server... oh, wait, that's another article.

Using passwords for your remote servers exposes you to a whole class of unnecessary security risks, which are easily avoided by either switching to SSH identities or SSH certificates. This article will cover both, since they're conceptually very similar. We'll be working very closely with OpenSSH's configuration files, hopefully explaining some of the more intimidating options you might encounter.

Why Passwords are Bad for You

I'll have a hard time explaining the entropy part any better than this xkcd, but I would like to hook into the part where you have people guessing your passwords. If you've recently set up a new server with SSH, you've probably found a whole lot of noise in your syslog about authentication failures. The majority of these will be caused by bots, which are trying to guess common user name and password combinations - and oftentimes will also try to exploit known vulnerabilities in certain versions of OpenSSH. You can mitigate the former of these issues with programmes like Fail2Ban, however you should remember that hope is not a strategy, and bandaids like these only delay the inevitable: the bot getting enough guesses in to guess your password.

(Re-)Introducing SSH Identities

Fortunately, SSH includes a feature to vastly increase the security aspect - and potentially make things more convenient for you, as well. This little bit of magic is called an SSH identity - or, more commonly, SSH keys. The keys work like this: instead of authenticating to a server by supplying a password, you generate a pair of public/private keys, much like in PGP/GPG. You keep your private key safe while uploading the public key to your server. The advantage? You increase the entropy from something under 50 bits to your key's length: typically 4096 bits or more. That's about two orders of magnitude.

Preparing the Client

Using SSH keys is very, very easy. Here's how: you generate a new key on the client machine using a command like this:

$ ssh-keygen -b 4096 -t rsa -f ~/.ssh/id_rsa

This will create two files: Your private key ~/.ssh/id_rsa, and your public key ~/.ssh/id_rsa.pub. The command will ask you for a password to secure your private key with - it's up to you whether you feel comfortable that this file won't get into the wrong hands, so choose carefully if you set a password or not. The options we used so far are the following:

-b 4096 This instructs ssh-keygen to generate a 4096-bit key. Feel free to increase this to your desired key length - remember to use powers of two. -t rsa Makes ssh-keygen generate RSA keys. According to the man page, valid algorithms are rsa, dsa, ecdsa and ed25519. ed25519 is a new, elliptic-curve based algorithm that was introduced in OpenSSH 6.5, whereas ecdsa is the old elliptic-curve DSA implementation that is known to have severe vulnerabilites. Elliptic-curve cryptography relies on the infeasibility of finding the discrete logarithm to a random elliptic curve element and is thought to be mathematically harder than the prime factorisation that RSA relies on, so in theory it should be more secure even at significantly lower key strengths. However the only implementation of this available in SSH - until very recently - was flawed the same way DSA was, and ed25519 may not be available on a lot of the machines you might want to use the key on. Finally, dsa - the standard, non-elliptic-curve variant of DSA - has many known attack vectors, so you should avoid that. -f ~/.ssh/id_rsa The -f option sets the output file name for your new private key. .ssh/id_rsa in your home directory is the default for SSH RSA identities, so it will be used automatically. You're free to use any location you choose, however you must make sure that your private key file is only readable by your own user account or OpenSSH will refuse to use it. Also, the public key is placed alongside this file with a .pub extension.

After your key is generated you will be presented with a small ASCII randomart image. Feel free to remember this image, as you can use it at a later point to visually identify a suspect key. To retrieve this image later, use this command:

$ ssh-keygen -lv -f .ssh/id_rsa.pub 2048 fa:86:25:1d:9c:c9:89:58:b2:dd:a5:5c:17:5b:f1:5f user@some-host.example.com (RSA) +--[ RSA 2048]----+ | ..o. | | . . o .o . | | * * B .. E| | o o @ o| | .S. .| | ..o | | .+ | | ... | | .. | +-----------------+

The generated randomart is fairly distinct - this one in particular kinda of looks like a palm tree with a coconut falling off on the right side there, doesn't it? You will probably be able to pick yours out after seeing it a few times. You'll rarely be identifying public keys like this, however. The option is rather intended for identifying host keys, by virtue of the VisualHostKey setting in SSH's config file. You should enable this setting in your ~/.ssh/config file - or the system-wide /etc/ssh/ssh_config if you have write access to it - by adding the following line at the beginning:

VisualHostKey yes

This will present the remote system's randomart picture every time you log in - and if you were using a password, it would do so before you would enter it, which would allow you to visually compare the remote host's key to what it usually looks like and see if something is wrong before giving away any sensitive passwords. This works by SSH host keys pretty much just being a private/public key pair like the one you just generated, unique to each machine.

Preparing the Server

Now that you have your identity file, you will need to log in on the server and add the contents of the ~/.ssh/id_rsa.pub file to your server's ~/.ssh/authorized_keys list. If there is no ~/.ssh directory on the remote host, create it. Your public key will just be a single line - albeit a long one - and this list of authorised keys will take as many public keys as you want. You're supposed to create a new private/public key pair on each host, by the way - not too many people seem to actually do that, mind you, but if you were to do so you could very easily revoke access from specific, compromised machines by removing the corresponding entry in the list of authorised keys. Note also that the list of keys is user-specific, meaning that any user where your public key is in the user's ~/.ssh/authorized_keys file is a user you can log in as, also implying that you can easily give someone else access to your account on a foreign machine without ever giving out any passwords or being root.

Once the key has been appended you can simply log into the machine using the ssh command line tool as normal. If your key requires a password you will be prompted to enter one - otherwise simply having the private key file will be enough to log in. To make sure that the key was actually used and to verify which key was used, you can use OpenSSH's -v flag, like so:

$ ssh -v magnus@stinger.becquerel.org [...] debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Trying private key: /Users/magnusdeininger/.ssh/id_rsa debug1: read PEM private key done: type RSA debug1: Authentication succeeded (publickey). Authenticated to stinger.becquerel.org ([46.226.106.4]:22). [...] magnus@stinger:~$

In this example I logged onto my web server. I cut away a few lines of output as they're not relevant. The highlighted debug1 messages indicate which private key file OpenSSH tried to use and more importantly that it tried - and succeeded - to log on using the publickey authentication method. Scan your debug output for lines like these to make sure it's working before heading to the next section.

Disabling Password-based Authentication

Now that you can log on with an SSH identity as opposed to a plain password, you should disable password-based authentication altogether. Make sure to only do this once you have verified that you can log in with an account that can escalate to root - or that you have an alternate way of getting back onto your machine. To lock down your server, edit your /etc/sshd/sshd_config and set the following options:

Protocol 2 PermitRootLogin without-password PubkeyAuthentication yes ChallengeResponseAuthentication no PasswordAuthentication no UsePAM yes

This sets a fairly strict set of defaults that should make most bots give up right after connecting. The options we used were:

Protocol Verify that only protocol version 2 is allowed. There's no point in supporting the rather dated version 1 and you're only opening yourself up to ye olde bugs of old. If this reads 1 or 1,2, change it to just 2. PermitRootLogin The setting without-password is a bit of a misnomer. What that does is it enables root logins, but only if the mechanism to authenticate was not a password - i.e. it enables root logins, but only for public key authentication. This is good. Never set this to yes. PubkeyAuthentication Make sure this is set to yes, otherwise you won't be able to log in once you disable passwords. ChallengeResponseAuthentication Set this to no to disable non-pubkey logins that could otherwise be handled through PAM. PasswordAuthentication This is what we were here for: set this to no to disable tunneled clear text passwords. UsePAM If your system has PAM set up, it'll still be a good idea to keep this enabled even if you disabled password-based authentication. This is because PAM also provides session and account management, so set this to yes.

All you need to do now is restart the SSH server, like so:

# /etc/init.d/ssh restart

Note that restarting SSH will not kill your active session, so you should verify that the new settings will actually let you log in before closing your current session - and revert and restart SSH again if they don't.

For maximum effect, make sure that SSH is the only way to log on to your server. At the very least make really certain that you don't have telnet enabled! There, now you're all set and can be very confident that bots won't be able to access your machines through SSH. But wait, there's more!

SSH certificates

SSH certificates are the latest and greatest enhancement to the public and private key authentication SSH has to offer. They work by introducing a new certificate authority that signs your host or user keys, which adds a few significant improvements to the concept, such as:

Central Authority You probably have more than one server and more than one user account - like, root and your favourite non-root user account for normal, day-to-day work. If you generate a new key or try to give a new user access to your machines, you would have to add that key to a lot of authorized_keys files, which is a slow and error-prone, annoying process. By using SSH certificates, you can cut down on this by simply signing allowed keys once and then you're done with it. Key Expiration Ever created a PGP/GPG key and lost the private key afterwards while people are still sending you messages using your old public key? Yeah, that's why you're supposed to set an expiration date for your keys. Unfortunately you can't do that with ordinary public/private key pairs in SSH - but you can with signed keys. Signed Host Keys Notice how whenever you set up a new machine and connect to it for the first time, SSH asks you to accept or reject its host key? If you were paranoid enough, you were actually supposed to distribute the host key through some kind of physical medium and be very, very scared of accepting new host keys. Since that is kind of impractical, SSH certificates also let you sign host keys, so that you only need to trust the certificate authority for a domain and then you won't see any warnings about unknown host keys when connecting to new machines on that network.

SSH certificates are a relatively new feature. As such they're not used nearly enough. Now that you know why they're a good thing, let's get 'em set up, starting with the root certificate.

Creating a Root Certificate

To sign anything, you need a certificate authority to sign them with. SSH does not use the more common X.509 certificates used in SSL as they're basically just an extension to the identity concept already in place in SSH. As such you don't need to mess around with obscure OpenSSL commands; like with identities, ssh-keygen is your friend! To create a new root certificate, create a new pair of keys first:

$ ssh-keygen -b 4096 -t rsa -f example-com-ca -C "CA key for example.com"

The options we used are pretty much the same, except that this time we didn't place certificates in your ~/.ssh/ folder. Why? Because those keys are not meant to be used as identities. There is one new option we didn't use last time:

-C "CA key for example.com" The -C option sets a comment in your key file. The default is user@host, but since you'll be dealing with a lot of keys at a time now it might be better to give the keys moe descriptive names.

Oh, and please note that most other guides will tell you to do these steps as root. There's no real need to generate keys as root - any ordinary user will do fine. So it's probably best if you do use an ordinary user account. Also, it doesn't matter where you generate the key pair - do it on your workstation if you can, not your server. Just remember to keep the signing keys safe - this one is probably one of those that you should use a password with, because this key is really powerful and you don't need to use it very often.

Signing Host Keys

The most straightforward use of your new signing key is to sign host keys. In SSH there is no real distinction between user and host keys, and as usual we'll use ssh-keygen for this. The command is as follows:

$ ssh-keygen -s example-com-ca -h -n host.example.com -V +52w -I host.example.com-key host-key.pub

This command contains quite a few new flags, so let's have a look at those:

-s example-com-ca Tells ssh-keygen to sign a public key with the private key example-com-ca. Substitute example-com-ca with whatever signing key you'd like to use. -h Sign a host key. Without this flag you'd be signing a user certificate. We'll get to that later. -n host.example.com Sets the host name for this new signed key to host.example.com. Replace with the host name of whatever machine the host key is intended to be used for. You could specify multiple host names by separating them with commas, e.g. -n host.example.com,ssh.example.com. -V +52w For how long the certificate will be valid. +52w means that the certificate will expire 52 weeks in the future, i.e. one year from now. You can also specify a range, e.g. +2w,+52w for a key that will become valid in two weeks and will expire in a year. The standard SSH date format applies, so you can use additional suffixes other than w and you can also specify explicit dates. See the Time Formats section of the sshd_config man page for further details on this. -I host.example.com-key Set an identifier for the signed key, which is used in logging (and revoking the certificate later, see below).

The final argument, host-key.pub, is the public key to sign. You always sign public keys, never private keys! The certificate will be placed in a new file host-key-cert.pub. For example, if you were to sign the RSA host key of the machine you're on right now, you could do that like this:

# ssh-keygen -s example-com-ca -h -n host.example.com -V +52w -I host.example.com-key /etc/ssh/ssh_host_rsa_key.pub

Note how you would have to do this as root, as the command will try to write a new file /etc/ssh/ssh_host_rsa_key-cert.pub and that directory is hopefully only writable by root.

Using Signed Host Keys

No matter where you signed the key, copy the certificate to the appropriate location on your SSH host; i.e. copy it to /etc/ssh/ssh_host_rsa_key-cert.pub if it's an RSA key and substitute the rsa part with the correct algorithm if you signed a different type of key. Next you need to tell the OpenSSH daemon on your SSH server to use the shiny new certificate by editing /etc/ssh/sshd_config:

HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub

Note how you don't need to tell the OpenSSH daemon anything about the CA - this is not necessary as it is the client side that verifies a server's host key, so the server just needs to know where its signature is.

This is all there is to do on the server side. On the client side, you need to add the signing public key to your user's ~/.ssh/known_hosts file. Add a new line like the following:

@cert-authority *.example.com contents-of-public-key-file

All of this needs to be on a single line, like always in the known_hosts file. The individual parts of the line are as follows:

@cert-authority Tells SSH that the following key is not the key for a single host, but rather is the expected signing key for a set of hosts. *.example.com A comma-separated set of of host name patterns for which this signing key is valid. This has the usual pattern format employed by SSH, where asterisks denote arbitrary character strings. IP adresses are also valid. contents-of-public-key-file This is literally the public key in your signing key pair.

Remember that all of this has to be on a single line. For example, the bottom of our Public Keys page lists the line to add for the becquerel.org network where this site is hosted. If you were to add this particular line and test it...

$ ssh stinger.becquerel.org Host key fingerprint is 62:2e:78:67:72:68:dd:05:2f:fb:0e:17:09:2d:1f:9d +--[RSA-CERT 819--+ | | | . . . | | + o E | | * o | | o S * | | . = o + . | | . * * + . | | o * + | | .o | +-----------------+ Permission denied (publickey).

You wouldn't see the last line if you had my private key, of course. Notice how ssh doesn't ask you to add a new host key - that's because it's verified against the signing public key. To verify that it's actually doing this, the -v flag is your fried:

$ ssh -v stinger.becquerel.org [...] debug1: ssh_rsa_verify: signature correct debug1: Server host key: RSA-CERT 62:2e:78:67:72:68:dd:05:2f:fb:0e:17:09:2d:1f:9d debug1: Host 'stinger.becquerel.org' is known and matches the RSA-CERT host certificate. debug1: Found CA key in /Users/mdeininger/.ssh/known_hosts:11 [...]

Notice how it says that the signature is correct and that it found [the] CA key in [the known_hosts file]. That's how you verify that it was using certificates; if it didn't know about the signing public key, it would instead say:

$ ssh -v stinger.becquerel.org [...] debug1: ssh_rsa_verify: signature correct debug1: Server host key: RSA-CERT 62:2e:78:67:72:68:dd:05:2f:fb:0e:17:09:2d:1f:9d debug1: No matching CA found. Retry with plain key [...]

Output like that means you didn't add the right CA key to your ~/.ssh/known_hosts, or that the host pattern was incorrect.

Signing User Keys

This final step is conceptually very similar to the host certificates: user certificates. Again we use ssh-keygen, like so:

$ ssh-keygen -s example-com-ca -n user -V +52w -I example.com-user id_rsa.pub

Note how this command is very similar to the one we used to sign host keys. The only differences are that the -n flag now specifies users and not host names, and this time you need to feed it your user identity's public key. Like last time it will produce a file called id_rsa-cert.pub. If you were to sign your own user key, the command would be this:

$ ssh-keygen -s example-com-ca -n user -V +52w -I example.com-user ~/.ssh/id_rsa.pub

And that's all there is to signing user keys. If your server were set up to allow you to log in with keys signed by this certificate, you'd be all set without any need to manually populate any authorized_keys files. So that's what you'll do next.

Using Signed User Keys

On the client side all you need to do is make sure that the id_rsa-cert.pub sits alongside your id_rsa.pub file, so we're done with this part. We still need to tell the server to accept keys signed by your CA, however. To do so, you need to copy your example-com-ca.pub to your server - /etc/ssh would be a good location, so I'll assume that's where you copied it to. You then need to edit your /etc/ssh/sshd_config file again and add an option like this:

TrustedUserCAKeys /etc/ssh/example-com-ca.pub

You could place multiple public keys in the file referenced to by TrustedUserCAKeys - one per CA that is allowed to sign user keys that you trust. As usual, restart your SSH server to have this option take effect. And that's everything you needed to do, you can now log on using your signed key, without needing to update the authorized_keys file for each user you want to log on as. Of course you can still do that, in addition to using the CA.

Separate User and Host CAs

It is often a good idea to use separate user and host certificate authorities. This is not strictly necessary, but it could increase security if either of the private keys were compromised or if users and hosts were managed by different departments. Feel free to just generate a second CA for your users like you did before creating your host keys and substitute the signing key as appropriate.

Revoking Identities and Certificates

The final thing to know about using identities and certificates is that you can revoke them. To do so, you would again need to edit your /etc/ssh/sshd_config and add the following option:

RevokedKeys /etc/ssh/revoked-keys

Make sure that the file listed here exists and is readable, otherwise public key authentication will be refused altogether! Depending on whether you used straight identities or certificates, you can either populate this file with a list of revoked public keys, or you can use ssh-keygen to populate this file with an OpenSSH Key Revocation List, or KRL for short. Any keys listed either way in this file will be refused access when trying to authenticate. It's like an inverse authorized_keys file. This is not particularly useful to force key rotation, but it is useful if you have a very specific key that got astray - or for blocking keys that are known to suffer from the old RNG bug and are inherently insecure.

If you want to use a KRL instead of a straight list of revoked public keys, OpenSSH's swiss army knife ssh-keygen is your friend, as usual. To generate or update a KRL, use the following command:

$ ssh-keygen -k -f revoked-keys -u -s example-com-ca public-key-or-file(s)

This usage has a couple of new options, so let's have a look at those:

-k This flag puts ssh-keygen in KRL mode, telling it to create or update a Key Revocation List. The new KRL is placed in the file specified by the -f flag, so in this example it would be placed in the revoked-keys file. -u If you specify this flag, the output KRL is appended to instead of being overriden with only the contents of the specified files. You should use this if you already have a KRL and just want to quickly revoke another public key. Note that if you specify this flag and the output file does not exist then ssh-keygen will fail with an error, so for your first run you should drop this flag.

The -s flag is optional and you only need to use it if you want to revoke a certificate by serial or id. If you only want to revoke public keys that you have then there's no need to specify the CA. The public-key-or-file(s) are the list of files that either contain public keys - one key per line - or revocation specifications. So, if you wanted to create a KRL with your own public key, you would invoke the following command:

$ ssh-keygen -k -f revoked-keys ~/.ssh/id_rsa.pub

... and now you have a new file revoked-keys in your current directory that contains this key. If you wanted to revoke an OpenSSH certificate by ID, you would create a new file ids-to-revoke with the following contents:

id: example.com-user

And the command to turn that into a KRL would be:

$ ssh-keygen -k -f revoked-keys -s example-com-ca ids-to-revoke

This usage lets you revoke keys without actually having the literal certificate or identity file on hand. You can specify multiple lines in your ids-to-revoke file with multiple id: commands. Instead of id: you can also use serial:, key: or sha1: to specify the key's serial, a plain public key or a key's SHA1 hash instead.

Remember to use the -u flag when updating a KRL and to actually distribute the KRL to the servers. scp and rsync are your friends, as usual.

Further Security Improvements

Now that you're an expert with certificates, there's a few additional things you can do to increase the security of your servers. Starting with their host keys.

Disable Unused Host Key Types

In your servers' /etc/ssh/sshd_config file, you will notice several HostKey directives. You should disable all of the host keys with algorithms that you don't use or which you don't trust. For instance on my servers, the block with host keys looks like this:

HostKey /etc/ssh/ssh_host_rsa_key #HostKey /etc/ssh/ssh_host_dsa_key #HostKey /etc/ssh/ssh_host_ecdsa_key

As you can see the DSA and ECDSA keys are commented out, meaning the SSH server will not use them. You could - and should - also remove these other private keys along with their public keys from /etc/ssh, as they serve no purpose. Although some distribution scripts may re-generate these upon restarting SSH.

Remember: the fewer features you enable, the fewer features can be exploited if it turns out there's a bug lurking somewhere.

Strengthen the Remaining Keys

The default host keys tend to be mere 1024 bits. There's no reason for that, so if you're paranoid you should create new host keys that are stronger:

# ssh-keygen -b 8192 -t rsa /etc/ssh/ssh_host_rsa_key

You should also regenerate your host keys if you suspect your host keys were generated when the weak RNG bug was in effect or if you suspect the private key has been compromised. Also remember to sign the public key anew and upload the certificate when you generate a new key, otherwise things will not work as intended.

Be advised that using keys stronger than 8192 bits with certificates will cause some versions of OpenSSH to ignore keys and fail. Some older versions may even be limited to 4096 bits. You will get very strange error messages citing that your certificate was not a valid certificate at all. So, best to hold off on those really strong keys for now. Also note that you should not assign a password to these keys, as you want your server to boot up automatically with SSH enabled and working.

Do not Keep Copies of the Certificates You Signed

Some guides will tell you to keep copies of the certificates you signed, or even to generate the public/private key pair for your user and hand the private key and certificate to them via a secure channel and to keep the files around so you can revoke them later.

This is not necessary, and is in fact a security concern. You do not need to have access to your users' private keys and you actually shouldn't in the first place, since it would allow you to impersonate them. Make your users generate their own key pairs and only ask them for the public key - which is the only thing you need to sign. There is also no need to keep a copy of the certificate; as long as you have the CA private key and the ID of the key which you assigned you can revoke the key using a KRL. Since you need not have the other information it is thus best to delete the public key file and the certificate once you've issued them.

Further Reading

Congratulations, you now have a tinfoil-hat level SSH server set up.

For further information on things you can configure in OpenSSH, read the following man pages:

$ man 5 ssh_config $ man 5 sshd_config $ man 1 ssh-keygen

If you need further guidance, you could also look at the following articles on the same subject:

If you have further queries, please use the comment section below or send me an email.

Background photo credit: bigpresh / Foter / Creative Commons Attribution 2.0 Generic (CC BY 2.0)