Testing

Ember’s test infrastructure is great, especially the async helpers, but we wrote most of our tests before component integration tests or ember-cli-mirage or pretender were available, and we did too much as acceptance tests that could have been unit tests. We now have hundreds of mockjax fixtures for acceptance tests, which are huge pain to reason about and maintain, especially since our backend APIs are in flux. For new tests, we’re doing more component integration tests, and writing factories instead of fixtures.

Our focus on acceptance tests, combined with Ember’s not-incredibly-fast initial render times, means that our tests are slow. Run in serial, a complete test run takes 40 minutes in CI. Ouch. We now run tests in four containers in parallel, which gets us to ~12 minute CI runs. We are rewriting large chunks of the project now, and are planning to do way fewer acceptance tests, way more component integration tests.

Ember’s strengths lead to some problems

Ember’s clean component patterns meant that we made a lot of focused components with a single responsibility. We then composed them with nesting and loops to make complex user experiences. However, broad deep trees of nested components cause performance problems in 1.13, but it looks like there won’t be extensive improvements to this particular performance problem until Glimmer 2 lands in Ember 2.7. To get fast render performance, we basically need to de-Ember the slowest parts. We’ve developed several techniques for improving performance:

Tuning individual parts of the app to have less nesting. Flatter component hierarchies render faster.

We moved a lot of data loading into parallel async calls because the concept of promises and loading states is deeply embedded in the Ember API; we don’t block the initial render and load data sooner.

We wrote an ember-cli-react addon, which gives a big speedup in rendering performance, although we haven’t yet tested it on Android. This is similar to Discourse’s approach; they replaced the Ember renderer with their own for their most important component. Using React means we didn’t have to write our own renderer and the performance is great.

Each of these card thumbnails has three nested components: teacher avatar, step counter, and a status badge.

Computed properties pose a similar issue; they allow us to build interesting UIs quickly, but there is a non-trivial cost; in one case, computing which shade of green to draw a rectangle requires analyzing fifty or more objects on the client side. Computed properties are great for the prototyping part of the application lifecycle but once we’ve decided on the UI, we can move intensive calculations to the server.

Dependencies

The extensive add-on ecosystem has allowed us to leverage great code that we didn’t have to write ourselves. We simply ember install ember-something-awesome and immediately have access to new functionality. However, we also have a new dependency to deal with, which has its own dependencies. For some add-ons, this isn’t a big problem because they have few dependencies or their dependencies are only used at build time. Where it gets confusing is npm dependencies that add bower dependencies, or bower dependencies on common packages.

We often end up with complaints from bower about unmet dependencies, but it’s unclear whether those complaints actually matter. Was a dependency unnecessarily locked down, or is there actually a problem that will happen if I run these two libraries together? When I added in ember-moment, for instance, I had to upgrade momentjs from 0.5.x to 5.x. Our test suite probably doesn’t have much coverage for time formatting, it’s the sort of thing I trust the library to handle for me, so I needed to ask the QA team to keep an eye out for date formatting issues. I think that particular update worked out okay, and ember-moment was much better than our format-date helper, so, disaster averted. But repeat that process across twenty dependency updates, and then framework updates, and there’s always risks.

It’s hard to find large projects that are open-source that we can look to for inspiration in dealing with these large-app problems. Do other people have these same problems with test times? With upgrades? I had conversations at EmberConf that revealed that other organizations do have Ember apps of similar size, but they’re all closed-source, with Discourse a notable exception. If you know of other large open-source Ember apps, please mention them in the comments.

Conclusions

All of this is fixable and the Ember community is already working on most of it. The keynotes and talks at EmberConf this year demonstrated that the Ember core team cares about the same things that we care about: engines to decompose monolithic apps, FastBoot for initial render speed, and most importantly, a massive rewrite of the rendering engine to beat React even with deeply-nested repeated components.

Big thanks to the team at AltSchool for teaching me Ember, and especially to Jeffrey Biles of emberscreencasts.com for technical review and encouraging me to write this series.

We are sticking with Ember for this summer’s big development push, and working hard to teach ourselves the newest best practices. We are hoping to get to Ember 2.0 in time for the new school year. Want to help? We’re hiring! Any questions, or do you have similar stories to tell? Please leave a comment below.