Summary Over the last year, I rewrote the Socorro collector which is the edge of the Mozilla crash ingestion pipeline. Antenna (the new collector) receives crashes from Breakpad clients as HTTP POSTs, converts the POST payload into JSON, and then saves a bunch of stuff to AWS S3. One of the problems with this is that it's a pain in the ass to do development on my laptop without being connected to the Internet and using AWS S3. This post covers the various things I did to have a locally running fake AWS S3 service.

First, there was fakes3 Antenna is developed in a Docker environment, so I wanted to treat "fake AWS S3" as a service which ran in a container separate from Antenna. The first system I looked at was fakes3. That ran in a container and it persisted data to the file system, so I could mount that as a volume and in that way review what was actually getting saved by Antenna. I used that for a while since it met my needs, but then the authors switched the project to a license that wasn't open source at which point I didn't want to use it anymore.

Then, there was moto I skulked around for other solutions and ended up switching to moto. Moto was originally a mocking library, but recently added a moto-server wrapper which let you run it as a service. I didn't like any of the existing Docker images for Moto, so I rolled my own with this Dockerfile.motos3 file: FROM python:3.5.3 MAINTAINER Will Kahn-Greene <willkg@mozilla.com> # FIXME(willkg): Use a commit on master because 0.4.31 has bugs on buckets. Once a new # release has happened, we can switch to that. ENV MOTO_VERSION df84675ae67e717449f01fafe3a022eb14984e26 RUN cd /tmp && \ wget https://github.com/spulec/moto/archive/$MOTO_VERSION.tar.gz -O moto-$MOTO_VERSION.tar.gz && \ tar -xzvf moto-$MOTO_VERSION.tar.gz && \ cd moto-$MOTO_VERSION/ && \ pip install .[server] # Port that moto listens on EXPOSE 5000 # This spits out --help which isn't helpful, but you can specify what to do # by setting the command. ENTRYPOINT ["moto_server"] CMD ["--help"] At the time of this writing, the latest release of moto is 4.31 which had a bug which prevented me from creating buckets, so I used a commit on the master branch after the 4.31 release. (Also, note that I don't have a clue about best practices for Dockerfiles.) Then I added this to my docker-compose.yml file: services : # ... moto-s3 : build : context : . dockerfile : Dockerfile.motos3 command : s3 -H 0.0.0.0 -p5000 ports : - "5000:5000" This runs the moto-server Python program telling it to run the s3 service and bind to all the IP addresses and listen at port 5000. It exposes port 5000 so that other Docker containers can access it. I used that for a bit.