The migration

Birds got migrations right way before software engineers did

So, without further ado, let’s dive into the actual migration. How to accomplish it correctly, efficiently, and without breaking too much? These are the steps you should follow:

Use automatic migration tools, if available, to get a global overview of the affected parts of your application and automate boring stuff. Prepare a detailed, independent plan for every deprecated functionality or library. Create isolate modules to interact with external dependencies: such a refactor will be insanely useful for the current migration, and will simplify future ones as well. Develop new features in a backward compatible way: if the migrated module is both compatible with the old and the new version of the language, you enter a healthy feedback loop which allows you to iterate quickly, and see the light at end of the tunnel. Don’t rush. Seriously, don’t do it. Don’t let the excitement for the upgrade take over you and make you forget everything else.

With that in mind, we collected some ideas and started working on a plan to make the migration as smooth as possible. All the steps described below include a comparison between how we estimated the tasks in terms of time and complexity, and how they actually turned out to be.

Step 1: automatic migration tools

Estimate: easy. Reality: easy

We used the wonderful elm-upgrade tool to have a glimpse of how much stuff would be broken after transitioning to Elm 0.19.

By just running the command elm-upgrade in the root folder of our project, the elm-package.json file was analyzed and we were asked which libraries we wanted to update, remove or keep. At the end of the process, the elm.json file was created with all the new dependencies, along with a log that listed all the other libraries that instead failed the update, due to incompatibility issues with Elm 0.19.

Then, it proceeded to refactor all our application code by replacing deprecated syntax (such as the exclamation mark ! for tuples), old module imports, and so on. In practice, it tried to automate all the changes that a couple sed commands would have resolved.

This process took a couple minutes, and left us with a diff that touched hundreds of files. It would have been a huge mistake to just commit and keep going with the migration. As I said above, it needs to be an incremental process, and the compatibility-breaking refactor made by elm-upgrade was one of the last steps that needed to be done.

This is the upgrade-warnings.txt file produced by the first run of elm-upgrade , listing all the libraries that were not ported to Elm 0.19, along with all the deprecated functionalities we were using from the standard library:

WARNING! 4 of your dependencies have not yet been upgraded to

support Elm 0.19.

- https://github.com/justinmimbs/elm-date-extra

- https://github.com/mgold/elm-date-format

- https://github.com/saschatimme/elm-phoenix

- https://github.com/simonh1000/file-reader



Here are some common upgrade steps that you will need to do manually:



- NoRedInk/elm-json-decode-pipeline

- [ ] Changes uses of Json.Decode.Pipeline.decode to Json.Decode.succeed

- elm/core

- [ ] Replace uses of toString with String.fromInt, String.fromFloat, or Debug.toString as appropriate

- elm/time

- [ ] Read the new documentation here: https://package.elm-lang.org/packages/elm/time/latest/

- [ ] Replace uses of Date and Time with Time.Posix

- elm/html

- [ ] If you used Html.program*, install elm/browser and switch to Browser.element or Browser.document

- [ ] If you used Html.beginnerProgram, install elm/browser and switch Browser.sandbox

- elm/browser

- [ ] Change code using Navigation.program* to use Browser.application

- [ ] Use the Browser.Key passed to your init function in any calls to Browser.Navigation.pushUrl/replaceUrl/back/forward

- elm/url

- [ ] Changes uses of Navigation.Location to Url.Url

- [ ] Change code using UrlParser.* to use Url.Parser.*

- elm/random

- [ ] Change references to Random.Pcg.* to Random.*

The log speaks for itself, and the top section clearly suggests that most of the work will need to be done on three areas:

How we deal with files

The communication to the backend through phoenix sockets

Date and times

In fact, most of the incompatibilities described in the latter section are either trivial, or easy but a bit time consuming. With that in mind, we decided to start dealing with something that would be compatible with both versions of Elm: the removal of native modules.

Step 2: native modules

Estimate: easy. Reality: medium

As Evan stated in the post I already linked above, native modules and effect managers are prohibited in Elm 0.19, and need to be replaced with pure Elm versions or Javascript ports.

We already removed a couple native modules before starting the migration, mostly for UUID generation and similar stuff, and it was an operation that took minutes at most.

Refactoring the code that was using the Elm file-reader library took a bit more time, as we were not that familiar with the FileReader browser API. Turns out that it’s really simple, and we only had to do a Javascript port that subscribed to the FileSelected Elm message, and sent in response the base64-encoded contents of the file. You can see a working example here.

While developing these features, the cool thing was that we could immediately test them on the codebase using Elm 0.18, being confident that it would have worked on Elm 0.19 too!

Then it came the time to think about the removal of elm-phoenix , the library that wraps Phoenix sockets and allows us to communicate with the Elixir backend. We employed websockets for two different things: querying the GraphQL server using Absinthe, and sending push notifications.

It turned out that elm-phoenix implemented an effect manager itself, and furthermore it used the deprecated websocket library underneath. We couldn’t use none of this libraries, so we had to look for some solid alternatives.

There is a cool websocket client for Elm 0.19, or we could always resort to Javascript ports. However the main problem wasn’t about the implementation, but about the scale of the refactor: we had hundreds of queries, and rewriting them without an effect manager would have taken days, possibly weeks.

At this point we were clueless and discouraged from the possibility of this huge refactor, until a coworker had a fantastic idea: change the transport.

Step 3: websockets

Estimate: impossible. Reality: easy

Websockets are cool and fast, but do you really need them for GraphQL queries? Probably not.

The fact that Elm 0.19 bundles the http library as an effect manager, helped a lot towards the decision of using HTTP as a transport for our queries. Also, it meant that we just had to refactor the one, main function responsible for making queries and decoding the response and/or error messages. Changing the backend took minutes as well, thanks to the huge flexibility of Absinthe.

We had just done in a couple hours what seemed to be taking weeks of work. That was the real game changer for us.

At this point we only had to reimplement socket initialization and error handling, which in our case just meant wrapping Phoenix sockets, since they already implement all these features internally.

Handling push notifications was as easy as we initially thought. We had only about a dozen different messages, but it would have been as easy even with an order of magnitude more. It’s all about making a couple of Javascript ports, declaring all the messages we are interested in handling, and have Elm call the appropriate actions.

That’s all for native modules. I have to admit that this stuff resulted in a week with more bugs than usual, mostly due to websockets. The Phoenix sockets library abstracted away all the details, and it turned out we didn’t have a thorough knowledge of websockets, especially about error handling, reconnections, auth issues, and so on. But bug fixes came, and we only had one last step before migrating.

At this point, we were completely sure that only date and time stuff needed to be ported, and it needed to be done in a totally different, non-Elm 0.18 compatible way (as explained below). Thus, we run elm-upgrade on the project and continued working from there.

Step 4: date and time

Estimate: easy. Reality: hard

Elm 0.19 completely changed the date/time handling, by introducing two new types, Posix and Zone , for representing respectively time in UTC format and timezones. The big difference with Elm 0.18 is that we now have a single type to deal with dates and times, rather than two separate ones that often clash together.

The library is very frugal and makes simple operations like parsing a date incredibly hard. In fact, we had to resort to an external library just for the purpose of parsing dates in ISO8601 format. That’s because you only get the Time.millisToPosix function, which basically accepts a UTC Unix timestamp and transforms it into a Posix . You’ve got a date as a string in ISO8601 format? You better write a parser yourself.

The Zone module does not allow you to specify timezones when parsing or formatting dates: you only get utc , or you can compute the local timezone using the here task. And that’s all. Do you want to make an app that displays the current time in three different timezones? You have two choices: either you are a package author and implement your own custom timezone or you use a 20KB external library that does it for you.

In a part of the application we had a couple <input type="datetime-local"> , which returns selected dates as string using this format: YYYY-MM-DD HH:mm . We knew in advance they were computed using italian timezone, and we had to parse them, make some computations, and have the output again in italian timezone. There is literally no easy way to do that. We had to do a Javascript port that calls new Date(input).toISOString() , which returns a string in ISO8601 along with timezone, and can be fed to the external library that parses it into a Posix .

After way more work than we initially estimated, we ended up with a custom Date module that implements all the parsers and formatters that we use throughout the application. It really helped cleaning the code from calls to various external dependencies, but we still had to use three or four different libraries inside the central module, due to lack of basic features in the elm/time package.

Final steps

Estimate: easy. Reality: easy

The migration was 99.9% done. All we needed to do was refactor the navigation, as explained in the official docs, and use the new APIs in a couple more places, like application startup, regex parsing, and so on.

Finally, we could send the application to the QA team for testing, and wait for them to tell us that we broke pretty much everything :-)

Jokes aside, the migration took some weeks to complete. We broke a lot of stuff. But the most important thing was that we proceeded iteratively, from the start to almost the end (date and times modules excluded), so that no one in the team was focusing on Elm 0.19 alone. Everyone would in fact work on features that would be useful for both versions of the app.

So, was it worth it at the end? I’ll leave the numbers speak for me.

Benchmarks

The benchmarks were executed on a machine with an Intel i7–8550U CPU and 16GB RAM. The Elm webapp was in the range of 50 kLOC.

Commands were run by preceding the time command, and only real value is reported. Full benchmarks here and here.

Initial build times

The commands were executed after downloading elm dependencies locally, and deleting the elm-stuff folder before each run, so the cache is empty.

Incremental build times

The commands were run after doing a clean build, then making the described changes and finally build again.