I’ve got a small infrastructure. I think, all told, I deal with something like 3 or 4 SSL certificates, maximum. Probably less. This means that I don’t deal with them very often, and last night was my first time dealing with intermediate certificates. Ye gods.

A quick introduction to how SSL certificates work may be in order…

First, SSL certificates deal with public key cryptography and key pairs. There is a private key that remains protected, and is used to create a public “certificate”, and can be used to “sign” (in other words, vouch for) other certificates.

Secure Socket Layers is all about trust. Making sure that the website you’re visiting is REALLY the website you want to visit is important on lots of occasions. The way SSL does this is by having an organization that you trust vouch for an organization that you’re not sure about.

Your web browser comes out of the box with several trusted certificates. These are the public certificates from Signing Authorities (SAs) that are trusted because…well, usually because they’ve spent a lot of money to put themselves in a position where they’re trusted.

The benefit of having these pre-existing trusted certificates means that whenever the private key owned by the trusted SA signs someone else’s certificate, your browser trusts that certificate, too.

If I, as standalone-sysadmin.com, created a certificate and presented it to your browser, your browser wouldn’t trust it. Why should it? Anyone can generate their own certificate. In order to have your browser trust my certificate, I have to get it signed by a CA that your browser trusts. Once that happens, then your browser will see “oh, this is his certificate, and it’s vouched for by someone I know, so I’m going to trust it”.

Back in the good old days (or at least before a year or two ago), Certificate Authorities (CAs) just used their root key to sign certs. This had the benefit of simplicity…the same key that generated the public root certificate pre-installed into your browser also signed the certificate being presented to it. Then things got more complex…

See, the benefit of root key signing was also the risk. Since a root key was trusted implicitly, if that key was compromised, it could be used to sign ANY certificate, and malicious people could present seemingly identical sites to unsuspecting people. It’s sort of like the equivalent of stealing the printing press and paper to counterfeit money.

The most likely way that the root key would become compromised is during the signing process. Since every certificate needed signed by the root key, the root key needed to be everywhere signing took place. The volume of certificates that get churned out by some of these CAs is huge, which meant that one hole in the system anywhere could be exploited to cause the whole system to come crashing down.

In order to avoid this problem, what CAs have implemented are “intermediate certificates”, which alleviate the necessity of keeping the root key around by creating “certificate chains”. The root key still creates the public certificate installed in your browser, but it no longer signs the certificates presented by web servers. Instead, the signed certificates refer to the intermediate certificate, and the intermediate certificate refers to the root key.

This picture from IBM does it much more justice than I can:

The problem for us is that the browsers don’t inherently know anything about the certificate chain…so we have to configure our web servers to present the entire chain of certificates. To facilitate this, lots of CAs have provided ready-made certificate chains in .pem files. I use Thawte, and you can get their intermediate bundles from here. Essentially, it’s just a text file:

$ cat SSL_PrimaryCA.pem

-----BEGIN CERTIFICATE-----

MIIERTCCA66gAwIBAgIQM2VQCHmtc+IwueAdDX+skTANBgkqhkiG9w0BAQUFADCB

zjELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJ

Q2FwZSBUb3duMR0wGwYDVQQKExRUaGF3dGUgQ29uc3VsdGluZyBjYzEoMCYGA1UE

CxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEhMB8GA1UEAxMYVGhh

d3RlIFByZW1pdW0gU2VydmVyIENBMSgwJgYJKoZIhvcNAQkBFhlwcmVtaXVtLXNl

cnZlckB0aGF3dGUuY29tMB4XDTA2MTExNzAwMDAwMFoXDTIwMTIzMDIzNTk1OVow

gakxCzAJBgNVBAYTAlVTMRUwEwYDVQQKEwx0aGF3dGUsIEluYy4xKDAmBgNVBAsT

H0NlcnRpZmljYXRpb24gU2VydmljZXMgRGl2aXNpb24xODA2BgNVBAsTLyhjKSAy

MDA2IHRoYXd0ZSwgSW5jLiAtIEZvciBhdXRob3JpemVkIHVzZSBvbmx5MR8wHQYD

VQQDExZ0aGF3dGUgUHJpbWFyeSBSb290IENBMIIBIjANBgkqhkiG9w0BAQEFAAOC

AQ8AMIIBCgKCAQEArKDw+4BZ1JzHpM+doVlzCRBFDA0sbmjxbFtIaElZN/wLMxnC

d3/MEC2VNBzm600JpxzSuMmXNgK3idQkXwbAzESUlI0CYm/rWt0RjSiaXISQEHoN

vXRmL2o4oOLVVETrHQefB7pv7un9Tgsp9T6EoAHxnKv4HH6JpOih2HFlDaNRe+68

0iJgDblbnd+6/FFbC6+Ysuku6QToYofeK8jXTsFMZB7dz4dYukpPymgHHRydSsbV

L5HMfHFyHMXAZ+sy/cmSXJTahcCbv1N9Kwn0jJ2RH5dqUsveCTakd9h7h1BE1T5u

KWn7OUkmHgmlgHtALevoJ4XJ/mH9fuZ8lx3VnQIDAQABo4HCMIG/MA8GA1UdEwEB

/wQFMAMBAf8wOwYDVR0gBDQwMjAwBgRVHSAAMCgwJgYIKwYBBQUHAgEWGmh0dHBz

Oi8vd3d3LnRoYXd0ZS5jb20vY3BzMA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQU

e1tFz6/Oy3r9MZIaarbzRutXSFAwQAYDVR0fBDkwNzA1oDOgMYYvaHR0cDovL2Ny

bC50aGF3dGUuY29tL1RoYXd0ZVByZW1pdW1TZXJ2ZXJDQS5jcmwwDQYJKoZIhvcN

AQEFBQADgYEAhKhMyT4qvJrizI8LsiV3xGGJiWNa1KMVQNT7Xj+0Q+pjFytrmXSe

Cajd1FYVLnp5MV9jllMbNNkV6k9tcMq+9oKp7dqFd8x2HGqBCiHYQZl/Xi6Cweiq

95OBBaqStB+3msAHF/XLxrRMDtdW3HEgdDjWdMbWj2uvi42gbCkLYeA=

-----END CERTIFICATE-----

This is where I started having a problem. See, I was looking at documentation that was telling me to use the SSL_CA_Bundle.pem file, which includes both the Primary Intermediate CA cert and the Secondary Intermediate CA cert. This was making my web server (actually a load balancer) puke.

The key? Only use the Secondary CA cert. I didn’t even figure it out. I went to bed in disgust. My boss Andrew played with it after I gave up and finally figured that out. How we were supposed to know? I’ve got no idea.

Also, why we had to use the Secondary CA and not the Primary CA? I don’t know that either. I examined my certificate with

openssl x509 -in mycert.pem -noout -text

and there’s no indication of the secondary CA being used, so I’ve got no idea. If you do know, I’d love to hear an explanation.

So yeah, I worked on this for a total of 5 or so hours yesterday, all told. It was my own personal hell, but the certificate is installed, we’ve learned how to deal with this in the future, and I don’t have to deal with this particular cert again until Sept 13th, 2013.