Methodology & Notes

Why Use Blogs & JSON?

Most simply, blogs are more than returning “Hello, world!”, and JSON is a very common use case. Good benchmarks need a similar load on each framework, and it needed to be a bit more load that printing two words to the screen.

Keeping Things the Same

One of the major themes I tried to stick to was keeping all of the blogs as similar to one another as possible, while still developing in the style and syntax of each framework. Many of the structs, like content generation, are repeated throughout verbatim, so that each framework has the same data models to work with, but facets like URL routing are sometimes starkly different to fit with the syntax and style of each framework.

Subtle differences

There are some subtle differences to note between the Server-Side Swift Frameworks.

Both Kitura and Zewo have issues building if there are any spaces in their absolute file paths. Xcode also has issues building with spaces in absolute file paths in any framework.

Zewo is using the 05–09-a Swift snapshot, which means that it has trouble building in release mode and has to be run in debug mode. All Zewo tests were performed with Zewo built and running in debug mode (without release optimizations) for this reason.

Static file handling is a point of debate amongst Server-Side Swift Frameworks. Vapor and Zewo both recommend using Nginx as a proxy for static file handling, then putting the framework behind that as the processing power for the backend. Perfect suggests using their built in handler, and I have not seen any comments on IBM regarding their own views on the subject. As this study was not about how well frameworks work with established server applications like Nginx, static file handling was used natively with each framework. You may be able to achieve better performance in both Vapor and Zewo if you choose to build with this in mind. This is also a secondary reason that I included JSON testing.

[Added September 1st with updated results] Zewo is a single threaded application. You can get extra performance out of it by running one instance of the application per available CPU, as they follow a concurrent, rather than multi-threaded model. For the purposes of this study, only one instance of each application was run.

Toolchains. Each framework is building off a different development snapshot toolchain from Apple. At the time of final testing they were/are:



- DEVELOPMENT-SNAPSHOT-2016-08–24-a for Perfect

- DEVELOPMENT-SNAPSHOT-2016-07–25-a for Vapor & Kitura

- DEVELOPMENT-SNAPSHOT-2016-05–09-a for Zewo

- DEVELOPMENT-SNAPSHOT-2016-08–24-a for Perfect - DEVELOPMENT-SNAPSHOT-2016-07–25-a for Vapor & Kitura - DEVELOPMENT-SNAPSHOT-2016-05–09-a for Zewo Vapor has a special syntax for running releases. If you simply execute the binary, you’re going to get some extra console logging that is meant to help with the development and debugging process. That has a little overhead. To run Vapor in release mode you need to add

--env=production

to the executable. i.e.

.build/release/App --env=production

[Added September 1st with updated results] When working with Zewo, even though you cannot build with swift in release mode on the 05–09-a toolchain, you can add some release mode optimizations by passing these arguments:

swift build -Xswiftc -O

Node.js/Express does not build, nor does it differentiate between debug/release

Static file handling is included in Vapor’s default middleware. If you are not using static files and want to optimize for speed, you must include (as I did in VaporJSON):

drop.middleware = []

Why Node.js/Express?

I decided to include a variation of the blog using the Express framework in Node.js. This is a good comparison because it has a very similar syntax to Server-Side Swift and is widely used. It helps establish a good baseline to show just how impressive Swift can be.

Developing the Blogs

At some point I started to call this “chasing the bouncing ball”. The current Server-Side Swift frameworks are under very active development, as is Swift 3, and each has a ton of changes from its previous versions. This became amplified by the frequent releases from all the Server-Side Swift Frameworks, as well as the Swift team at Apple. None of them were particularly complete in their documentation at this point, so I’m very thankful to members of the framework teams and the Server-Side Swift community at large for their contributions. I’m also grateful for the help that countless community members and framework teams gave me along the way. It was a ton of fun, and I would do it again without thinking.

As a side note, even though attribution was not required by the license, I feel it is nice to note that all the random royalty-free pictures included in the sources are from Pixbay, and were selected by me at random. Sources like this make demo projects much easier.

Hosting & Environment

To minimize any differences in the environment, I took a 2012 Mac Mini and gave it a clean install of El Capitan (10.11.6). After that, I downloaded and installed Xcode 8 beta 6, and set my command line tools to Xcode 8. From there I installed swiftenv, installed the necessary snapshots, cloned the repos, and cleanly build each of the blogs, again in release mode where possible. I never ran more than one at a time, and each was stopped and restarted in between tests. The test server specs are:

For development, I use a 2015 rMBP. I ran the build time tests here, as this is my real-life development machine and it made the most sense. I used wrk to get the benchmarks, and I did this over a thunderbolt bridge using a thunderbolt 2 cable between the machines. Using a thunderbolt bridge makes sure that you have an incredible amount of bandwidth and that you are not benchmarking the limitations of your router, instead of the task at hand. It’s also more reliable to serve the blogs on one machine, and to use a separate, more powerful machine to generate the load, ensuring you are capable of overpowering the server, so that you can be certain this is not a limitation. This also gives you a consistent testing environment, so I can say that each blog was run on the same hardware and in the same conditions. For the curious, the specs of my machine are:

Benchmarking Notes

For benchmarking, I decided to use a ten minute test with four threads, each carrying 20 connections. Four seconds is not test. Ten minutes is a reasonable timeframe to get plenty of data, and running 20 connections on four threads is a hefty load for the blogs, but not a breaking load.

Source Code

If you would like to explore the source code for the projects or do any of your own testing, I consolidated the code used for testing into one repository, found at:

https://github.com/rymcol/Server-Side-Swift-Benchmarking

Detailed Results

Build Time

I thought it would be fun to first take a look at build times. Build time can play a big role in day to day development, and while it has little to do with a framework’s performance, I thought it might be fun to explore real numbers vs. how long things felt while I was waiting.

W hat was run

For each framework,

swift build --clean=dist

and then

time swift build

were run, followed by a second test of

swift build --clean

then:

time swift build

This factors both a full build including pulling dependencies with SPM, as well as a regular, clean build with dependencies already downloaded.

How it was run

This was run on my local 2015 rMBP and the builds were all done in debug mode, because this is the normal process when developing Swift software.

Build Time Results