I’m a game programmer, and I practice productivity through unwillingness to do boring things with my time.

Game development has plenty of boring work to go around, so optimizing for years of life spent as Jonathan Blow expressed it, means you get to spend more time on things you actually enjoy.

I love playing games and I love making them, I just don’t fancy the work required to take the game code from my machine and package it up in a form that can run on my iPhone.

One especially painful part of this process has always been when someone needs to have their Apple Universal Device Identifier, or UDID for short attached to the game in order to test it on their device. Updating the UDID is generally a long-winded but trivial process where if you do everything right the game will run normally on the target device, and if you do it wrong you might spend an hour backtracking and googling for cryptic error messages.

We tried Unity Cloud Build first, but the build times were too unpredictable and it wasn’t possible for us to use the very latest patch versions of Unity, but critically automatic uploading to HockeyApp would still have required using an external server to pick them up after completion. Because the needs of all games and apps are different, it’s hard to wrap everything in a single system and share it as a whole, but perhaps by offering a bird’s eye view to what I did to automate the builds for our game Sociable Soccer I can hopefully save some a bit of time.

Nuke the entire site from orbit, it’s the only way to be sure

Now, I didn’t want to just reduce or speed up this work. This mindless mashing of buttons is time wasted, and I wanted to completely eliminate it so that I can get on with the interesting parts of development.

The ideal scenario: you just enter the UDID and magic happens.

We use Slack for most of our team communication. I’ve spent more than two decades on IRC so Slack in some ways is very familiar to me even if it’s new to most company cultures.

Just like IRC has a lot of autonomous scripts appearing as bots on the chat channels, this approach also works on Slack. The first step is to create a custom Slack integration to handle the UDID and pass it to the server.

The straightforward custom integration settings on Slack

When the Slack custom command is run on the client, the client passes it to the Slack server which in turn sends a customized webhook, i.e a user-defined HTTP callback to the build server. As we have a tiny development team, rather than setting up an expensive build server in the cloud the server in this case is just my desktop machine at the home office. The IP is a free dynamic IP allocated through noip.com.

Because iOS builds are practical to make only on a Mac, and because Apple’s desktop machines lag far behind in 3D acceleration and VR functionality, the home machine runs Windows 10 with a VMWare virtual machine for OSX. There are some gotchas in installing OSX inside a virtual machine, but you could always use a Hackintosh or an actual Mac to do the job.

Jenkins dashboard running inside the virtual machine

The actual builds on the server are triggered by polling the SVN every five minutes, but rather than launching a build directly the Slack message is first picked up by a simple PHP script running on a virtual host configured for the Apache server. To receive incoming webhook messages, the Apache server must be configured with a valid SSL certificate. This used to be quite painful, luckily there are now free providers and tools like certbot which made the whole thing much easier.

Trivial code for responding to Slack using CURL with PHP.

The PHP script does some basic validation for the Slack request parameters, stores the new UDID in a devices.txt file formatted following Apple’s specifications for adding multiple UDID numbers at once by uploading a plain text file. The script then uses CURL to send a message back to Slack informing of the successful result. At this point the script could have saved the file somewhere locally where the build server can find it, but in my case I store the devices.txt in the project root of our SVN hosted by Assembla. I’m not a fan of PHP but in this case it does the job, and the code for firing off the Slack notification is made easy by others who have done it before.

To infinity and beyond

Meanwhile, the server is running Jenkins, an open source Continuous Integration system built in Java that includes a ton of specialized plugins. Jenkins is configured to check changes in SVN every 5 minutes, and if changes are detected it will trigger a Jenkins project for each of the current five build types. The project could also be set up as a primary project with a single repository folder and the platforms as sub-projects, but this seemed the more effective approach for now because of how Unity caches things even if it burns through more disk space. The virtual machine currently fits in a 80GB virtual partition, which is not too bad.

Jenkins polling SVN every 5 minutes for possible changes

Each Jenkins project will first gather all the required software keys and environment parameters, then it will perform a SVN update on the working folder of the project. This ensures the build is always using the latest data in the game. In some cases Unity may have problems with making incremental builds, so forcing a clean build is usually recommended.

/Applications/Unity.app/Contents/MacOS/Unity -nographics -projectPath “${WORKSPACE}” -buildNumber “${SOCCER_BUILD_NUMBER}” -buildtarget ios -quit -batchmode -executeMethod BuildScript.PerformIOSBuild -logFile /dev/stdout

Using the Unity3D Builder plugin, Jenkins will call a build script inside Unity for each of the build types, launching Unity in batch mode with no graphics enabled. This does mean that when things break, visual cues are limited and you may be doing forensics work on error logs to figure out what happened.

Always look in the log file

The command line to launch Unity includes a few critical bits; the project path is set to Jenkins’ WORKSPACE variable, and the build number is generated with the handy Version Number Plugin. Finally, when launching the Unity batch process you’ll want to redirect the log file into stdout so that it’s included in Jenkins’ other output; otherwise the output disappears into the void resulting in some cryptic debugging. At the time, the logFile command line parameter was not documented, which added to my initial confusion. When the logs do work, they are very verbose.

Unity3D busy at work, first compiling the project scripts for its own use

Inside Unity, the build process is handled by a custom build script to take care of the continuous integration. The build script (BuildScript.cs) is also a part of the SVN project so this way we can inject conditional parameters and variables into the build itself, including the version number and in the case of an Unity issue workaround, the Android SDK and NDK root folder directories. Since preferences like the SDK and NDK path are stored and defined per user in Unity and the process runs under Jenkins, the paths appear empty when running in batch mode and the project will fail to build.

Workaround for the missing Android SDK/NDK paths on batch run of Unity

At this point Unity will also pass the code through the handy Obfuscator from BeeByte. While obfuscation is never a replacement for security, it does make it much less trivial to rip out the code from a Unity project on platforms running the Mono backend. Unity will internally use their proprietary IL2CPP to convert the Unity bytecode into C++ for most platforms, but it’s not available for Windows, and for Android you have to select it specifically from the project settings.

Wrapping up for delivery

Unity will process the build and produce either an APK calling Android SDK and NDK tools directly using the previously injected paths, or in the case of iOS an Xcode project file, which is customized slightly using the egoXproject plugin.

When building for iOS, instead of calling Xcode directly Jenkins is configured to launch a Fastlane script that eases working with Apple systems. Fastlane is a wonderful open source tool written in Ruby. I’ve never used the Ruby language before, but my experience with Fastlane was excellent. Fastlane is configured to perform actions like updating and downloading certificates as well as driving the build process.

Fastlane register_devices is first used to upload the devices.txt to the server so that the UDIDs can be included in provisioning profiles. While Fastlane now includes the innovative Fastlane match system for managing and distributing UDIDs, I didn’t want to set up a Git repository at this point so I used a small Fastlane spaceship script to automatically update the provisioning files to include the UDID that was only just added.

Using Fastlane Spaceship to update the provisioning profiles to include all devices.

Fastlane then injects the proper provisioning files into the Xcode project and launches the build using Fastlane gym. This way the build already includes the UDID for the device that was only just added.

Gym triggers the Xcode command line build with the appropriate parameters, packages the project and uses Apple’s CodeSign tool to sign it with your certificate. In our case, CodeSign uses certificates in jenkins_keys.keychain together with the provisioning files downloaded by Fastlane from Apple developer site. If you have an enterprise account on Apple’s development site you can add a separate user for Jenkins, or you can just let Jenkins and Fastlane use your own developer certificate. In either case, you might want to change the timeout value in the appropriate keychain so that it’ll remain active all through the build and packaging as Unity can sometimes take longer than the allocated default of five minutes. If the keychain locks or CodeSign doesn’t otherwise have access to it, you’ll get a rather cryptic error, “User interaction not allowed” which really just tries to say that Jenkins would have asked you to unlock the keychain with a password, but it couldn’t because it was running a batch process that doesn’t allow interactivity.

For me personally, this was one of the tougher parts of the puzzle. It might have helped if I was more familiar with OS X, which I am not.

Changing the default timeout in Keychain Access from Keychain Settings

After Gym has produced a valid IPA, Fastlane hockey uploads the file to our HockeyApp account. HockeyApp will update the distributed build based on the version number injected via the earlier via the build script, allowing the integrated HockeySDK to remind users of new versions being available, or sending out optional notifications by email or Slack. In addition to HockeyApp, Fastlane also supports integration with Testflight which is now a part of Apple.

Success! New build is made available on HockeyApp!

Whenever a build is complete and uploaded, the Jenkins Slack Plugin on the build server is configured to send a message back to Slack which is then forwarded to one of our team Slack channels.

Summary of steps needed

Create custom Slack integration to redirected dynamic IP

Configure Apache virtual host to run PHP and Jenkins, use certbot to set up valid SSL certificate

Create custom script to respond to slack message and store the UDID

Create Jenkins projects for builds for iOS and Android (Jenkins GUI)

Create Fastlane projects (“fastfile”) for iOS builds to ease provisioning

Configure Jenkins to poll SVN and update certificates using Fastlane

Create custom BuildScript.cs for Unity with to inject variables

Configure Fastlane to build project with Gym / Xcode

Configure Fastlane to upload IPA with HockeyApp

Motivation from Hyperbole and a half

While it’s hard to say how many hours of time are saved in total from never having to manually create and provision iOS builds for the game, the peace of mind from knowing the process will just work without having to do as much as press a button is immense. Besides the virtual machine and the hardware itself, tools used were all free and open source, and I was using most of them for the first time. If I can figure this out, you can. And if you can’t, there’s always Google and Stack Overflow.

Plus, being able to share a provisioned build by entering a UDID number into the Slack client on iPhone and having the build automatically available in just under 15 minutes on HockeyApp is kind of neat.

There’s more I could do with Jenkins, but I’d really just rather get on with the more important things, like building the spiritual sequel to Sensible Soccer. :)

Recent PC test build of our game Sociable Soccer.

Nailed it? Did I miss something obvious or fail to detail some part of the sufficiently? Let me know, as evolving the build systems is always a moving target and every new platform has its own quirks.