Abdel is currently taking his Master’s degree in Robotics and Artificial Intelligence. A late start notwithstanding – he only had his first PC at age 16 and first discovered the internet at 18 – Abdel has shown great passion and an apt mind for computer sciences. He already has ample experience developing and writing various kinds of software, from web APIs to robotics software, and is currently partaking in Google Summer Of Code, where he is working on the FreeBSD kernel. Abdel also curates the awesome robotics list

At Transloadit, it is our desire to give you the best possible encoding experience, and one of the main factors involved is making sure that we are using the latest encoding tools.

It's no secret that we are heavily relying on open source technology for this, and we therefore make a conscious effort to give back to the giants on whose shoulders we are standing, for instance through providing free hardware to the ImageMagick project.

Even though we have an intimate relationship with the project, it has proven to be very problematic for us to modernize our stack.

The 'fun' parts of maintaining encoding software

Oftentimes, our customers rely on behavior of the old version (even on its bugs, but more on that later), so "just upgrading" - which we naively tried to do a few years back - breaks their workflow. To ensure we can provide a consistent experience even between operating system upgrades, we are required to compile ImageMagick and all of its 31 dependencies. Yes 31, because by adding libraries such as pixman , pango and ufraw , we gain support for 131 more formats than what a standard apt-get install imagemagick would provide on Ubuntu. And by including libraries all the way down to zlib , we also shield ourselves from any backwards compatibility breakage that could occur as a result of operating system upgrades, thereby allowing our encoding capabilities to be the reliable constant upon which our customers can build their businesses.

Besides vast support for formats and this guaranteed continuity, the advantage of compiling is that it allows you to create a static binary. This is a single executable file that has every dependency baked into it. This way, it becomes easy to run very different versions of the same software alongside one another without any conflicts, giving our customers an upgrade path that is opt-in. This is how we have been providing upgrades to our video encoding software.

The problem, however, that we are having now with upgrading our image stack, is that the process of compiling ImageMagick and its 31 dependencies would take two hours, and that is under perfect conditions on a beefy server. If one particular library version conflicted with another, you may only find out after 90 minutes, after which you would have to think of a possible fix and then try all over. That can hardly be considered workable or maintainable, and it certainly does not amount to the level of agility that we aspire to have in our development.

The introduction of Docker containers, we thought, would solve some of this struggle in that we could at least run different versions alongside each other easier, and speed up our build times through the use of layers. Also, instead of having single massive binaries to recompile, we could just recompile a single shared library if needed. Our initial joy was tempered when we learned that there was no way to start containers without dangerous system privileges. This meant we either had to keep containers running and let our non-privileged API user proxy commands to those, or be okay with giving this user the privilege to start containers. Both options weren't great, to say the least. Proxying would be quite fragile, and by handing out such privileges, an attacker could theoretically mount any directory on the host machine as a --volume into the container, and then read/write/delete whatever they wanted. We later also learned about the work that was being done in the runC project to launch containers without root access (and we'll likely take that approach in the future) but it wasn't yet at the level of maturity that we needed.

We also considered running a separate cluster for the new version, but this would have some serious costs in terms of additional hardware, maintenance and operational overhead.

We dove deep into all these different directions, but ended up having to back out every time since there were serious drawbacks. How should the future of our stack maintenance look? What solution is both reliable and mature enough to give us guaranteed continuity while maintaining our agility?

Nix

For a long time, we had also been looking at Nix as a possible solution. We have been building our software with a simple compile framework called depmake that Felix wrote back in 2012 and which already implemented some of Nix' basic ideas and qualities. It has been working very well for us all these years, but Nix takes these, and additional concepts, much further.

Nix is a functional programming language that lets you define how to build software in a declarative fashion. By making use of a global cache, you only have to compile things that are unique to your environment, but in many cases, you can just pull the resulting software straight from this cache. This can greatly improve build times for the simple fact that there is no building *waves hand*

After a good amount of learning and experimentation, we figured out how Nix and depmake could co-exist, so that we can start making use of the new, without throwing away the old.

Future

Using Nix really opens the floodgates to potential new software and is going to make it extremely easy for us to roll out a new stack, customize it, ensure it keeps working in the same way forever, and allow multiple versions of the same software to exist alongside one another.

The bulk of the work in providing a new stack to our customers has now moved from building to testing it, which is a great improvement.

We are still considering wrapping our Nix-built software in containers and running them root-less, but this would primarily be to deliver additional security on top of the containment that we already provide.

Will we ever deprecate our old versions? Yes. But we'll do so with fairly lengthy grace-periods during which we gently nudge people towards the new.

Without further ado.. version two!

We are quite proud to present to you our new ImageMagick stack nicked v2.0.3 (not SemVer, any change can potentially break backwards compatibility). It has been in private beta for two weeks already and will now enter public beta. It is based on the latest ImageMagick version that Nix provides but with custom additions such as libraw so that our customers can keep enjoying those 131 extra formats.

Besides RAW and all the other formats that we already support, customers will be happy to learn that the new stack comes with support for both WebP and DjVu formats.

We have updated our supported formats page so you can more easily discover differences between our stack versions. Simply click the "Compare" button for a clear overview.

For your convenience, here are the full lists:

Gained read support: 3FR, 3G2, 3GP, CANVAS, CLIP (previously write-only), FILE, FTP, HALD, HTTP, IIQ, JNX, MAC, MEF, NRW, PES, RAW, RMF, RW2, SCREENSHOT, WMF, WMZ.

Gained write support: BRF, DDS (previously read-only), INLINE (previously read-only), ISOBRL, ISOBRL6, JSON, SPARSE, UBRL, UBRL6

Gained read/write support: AAI, BGRA, BGRO, CAL, CALS, DXT1, DXT5, G4, GROUP4, HDR, JPE, JPS, MASK, MKV, PNG00, PNG48, PNG64, PSB, RGF, SIX, SIXEL, TIFF64, VIPS, WEBP

Besides more formats, our new ImageMagick version also enables HDRI (High Dynamic Range Imaging) by default, so you can expect more accurate image processing results.

Aside from these customer-facing upgrades, we're quite happy about the fact that the new version also has some operational benefits for us, such as the plugging of a number of security vulnerabilities, memory leaks, and so forth.

Notable breaking changes

Our previous version of ImageMagick had a bug in colorspace handling, resulting in darker images and the incorrectly reporting of sRGB vs RGB. This is now fixed, but if you found a way to work around this bug, you are essentially relying on it and will therefore need to un-compensate when you choose to upgrade.

We no longer support reading PGX and JPX files, and for the formats: BRG, GBR, GRB and RBG, it is now recommended to create an RGB format and then swap channels afterwards.

For reference, you can find a full changelog at the bottom of this post.

Upgrading

Transloadit recommends to start using the new ImageMagick stack for testing and non-critical workloads. You can do this by adding imagemagick_stack: "v2.0.3" in your /image/resize Robot Step .

Our apologies for taking so long to deliver on this upgrade, you can expect much timelier updates thanks to our new way of stack herding!

We'd love to hear your findings, so do let us know what you think via the well known speech bubble on our site!

Changelog

For reference, here is the full changelog: