The Creative Network Session 11 — Debugging SSL verification failure inside of docker container

Previous sessions

NEXT TWO SESSIONS:

Session 12: Thursday, May 25th 9:00pm PST

Session 13: Tuesday, May 30th 9:00pm PST

Video for session 11:

During this session we worked together to debug a rather tricky problem: mysterious failure of SSL certificate verification from inside of docker container.

The problem started with React/Rails project I was trying to deploy with Docker. This project, including the Dockerfile, was working fine few weeks ago. I simply made some minor change in the client app and wanted to create a new image with my change, here is what the core part of Dockerfile look like:

FROM alpine:3.3 RUN mkdir /matching-tool

WORKDIR /matching-tool ENV BUILD_PACKAGES="curl-dev ruby-dev build-base" \

DEV_PACKAGES="g++ make bash zlib-dev libxml2-dev libxslt-dev tzdata yaml-dev sqlite-dev postgresql-dev mysql-dev" \

RUBY_PACKAGES="ruby ruby-io-console ruby-json yaml nodejs" \

RAILS_VERSION="4.2.3" RUN apk --update --upgrade add $BUILD_PACKAGES $RUBY_PACKAGES $DEV_PACKAGES \

&& apk del --update build-base openssl-dev curl-dev \

&& apk add --update libstdc++ RUN apk add --update curl vim COPY . /matching-tool RUN gem install bundler --no-document \

&& gem install bigdecimal --no-document \

&& bundle config --global --jobs 32 \

&& bundle config build.nokogiri --use-system-libraries \

&& bundle install --without test development doc --deployment --retry=3 EXPOSE 8080 CMD ["./run.sh"]

Then I do

docker build -t matching-tool .

I get this error:

Basically, all the gem install commands failed because the SSL verification with https://rubygems.org/ failed. None of the ruby gems can be installed.

In hindsight it looked look rubygems.org simply doesn’t recognize my ssl certificate. Following the direction from rubygems official site, I tried to manually downloading the latest trust certificate of rubygems, placing them in the correct ssl_certs directory, both inside of the docker container AND my local machine. Nop, still getting the same error.

By then I was pretty much stuck and reached out to Avi for help.

After some tinkering, we realized any https call coming from the inside container would fail, not just for rubygems server but for any https server. When I do curl --verbose https://www.google.com this request would also fail on SSL handshake. Outside of the container, all https requests work just fine. So what is the root cause?

It turns out that inside of my container, SSL certificate validation failed due to time drift of the container(time inside of container is inconsistent with the actual time). In SSL, clocks are used for certificate validation process, it’s important that the timestamp of both sides of SSL handshake falls within an acceptable range. Read here for more details about clock synchronization play in SSL communication.

I use Docker for Mac for development. There is a bug in their software that when you put the host machine to sleep and wake up again without restarting Docker itself, the time of any new container you start will be out of sync.

Check the below screenshot when I put my computer to sleep last night:

My machine is showing May 24 11:05am 2017 PST while the container is showing time as May 24 07:05:02 UTC 2017 , this is almost 11 hours drift, which is the amount of time I put my computer to sleep. Once I restart Docker, the time inside and outside of the container will be in sync:

Once the time drift is fixed, all htts connection were working fine.

In conclusion, the solution for my SSL certificate verification failure is to make sure the container time and actual time is in sync. In this case it’s fixed by simply restarting Docker(duh…this is one of the problem that will “fix itself” if I actually bothered to restart my computer, which will in turn restart Docker, but then I would have never found out the root cause).

Great thanks to Avi for lending his hand on this, I’ve learned a lot in this process.