I recently worked on a performance project for a client. In this article I will detail the context of the project and explain what we learned and implemented, hoping it can help some of you.

Some context

The app:

An internal financial tool for a big company letting managers view and update workload costs per projects and departments. Therefore a lot of dynamic data computation and representation through tables. The main page of the app is a full synthesis in a table letting the user drill down to a single project cost, while displaying the total amount per departments that own the project.

The stack:

Angular 1.4.8

Loopback 2.22.2

PSQL 9.3.14

The pain point:

The synthesis page was developed in such a way that all the data treatment is done in the front-end at first load, and then no matter which filter you click on, the calculation being already done, just needs to be displayed. Hence the page loading is extremely slow.

The impact:

In Production on corporate computers, the synthesis page took tens of seconds to load (or even crash)

On a development environment, the page took 17 seconds to load

In the end, most users preferred to export the data in an excel file rather than using the app

Our investigation

The challenge was exciting, especially since we did not know either the app, the problem it answered or its code.

First advice, before rushing for fixes:

1- Take the time to understand the overall purpose of the app, and the business complexity of its features by talking with the project stakeholders and interviewing users

2- Acquaint yourself with the code base

3- Investigate the possible reasons for performance issues

In order to understand what was taking so long, and how our improvements would impact the performance, we used different reporting tools:

Timeline (Chrome dev console)

Purpose: Understand what from loading, scripting, rendering or painting was the slowest.

Result: The responsibility was well shared between loading (API calls) and scripting (Javascript and Angular).

Network (Chrome dev console)

Purpose: In the network tab of the console, we got a detail of what was queried to the server, and if those queries were all necessary, unique and optimized.

Result: Several queries were sending the full data-table instead of the necessary elements. One same query could be made several times with the same data but for different components of the page.

Loopback Explorer

Purpose: Strip the global context out in order to investigate specific sensitive queries performance. Start from heavy queries found in the network console and query them directly to understand what's problematic.

Result: Some queries were taking above 5 seconds to send back a result because they were requiring heavy join from the ORM.

Batarang (Chrome extension)

Purpose: Angular performance tool, to know the number of $digest cycles and active watchers.

Result: With more than 200,000 watchers for the last 30 $digest cycles, we sure had an issue here.

Server profiling

Purpose: Understand how the server/hardware handles the app.

Result: Our application was sharing its server with others. We could see with a top that the CPU was skyrocketing on single requests, and mainly that the RAM was fully taken. By lack of time and knowledge, we did not use any more sophisticated tool regarding profiling, but it is worth spending time on it: it was in the end the main source of our problem.

The solutions

With all these answers in hand, it was time to prioritize our actions, and start developing the improvements. From what we learned, each aspect (Front-End, Back-End, Server) of the app needed an equal attention.

Staging ISO-prod

It is extremely important to have a staging environment ISO to production, in order to directly feel the pain it might end up doing in production, and avoid unpleasant surprises the day your app officially opens to final users.

Ergonomy

Despite many efforts you will make in order to have a more performing app, the one true rule is to pay attention to its ergonomy. A bad ergonomy will very often be the main reason why an app is heavy. The most incriminating part of the app, the one we were working on, was such a dense and complicated feature, that both users and developers were losing themselves in its complexity. If the ergonomy had been tailored and simplified for user needs, the developers would not have had so much trouble working on the app, the code structure would have been evenly simplified and the app would have been lighter.

Angular

ng-if vs ng-show

Our table had to include several embedded ng-repeat to display properly the table tree. In the leaves of this tree, one directive was hidden with ng-show (depending on the users chosen filters). The issue in doing that is that Angular will generate those elements, and keep them in the DOM. It weighs heavy on performance, by just setting a ng-if instead, we were able to win precious seconds.

One time binding

A special type of data binding between the controller and the view in Angular: this special $scope variable is processed and rendered at controller construction, and then the binding is "cut". If the variable changes in the controller, the front won't be re-rendered. This binding allows for a lighter component state, hence lighter $digest cycles.

ng-repeat

It is known that Angular is struggling with ng-repeat . As much as possible if you can, find a way to avoid having too many ng-repeat , especially embedded in one another, like we had.

Heavy $scope

The scope of the main table controller was crowded. Too many variables and functions were assigned to it. Not each of these needed to be bound to the $scope . Sometimes it was only done so in order to be able to test the function. We decided to create additional services for these functions, first they were easier to test, the controller was clearer, and the scope lighter, hence lighter $digest cycles.

$scope functions in view

Many functions were called during the view rendering, directly placed in the HTML through { myFunctionInTheHTML } . It generated more complication during the render, and even more if a re-render was necessary. We decided to refactor this as much as possible in order to have most of the processing done directly in the controller.

Loopback / Ops

Table relations with Loopback

Loopback ORM joins data from different tables in Javascript instead of SQL. As our operations required to treat a big amount of data, we decided to rewrite the queries in SQL directly.

Limit the number of similar API calls

The table was accompanied by many filters. The lists of some of those filters were the same but one different request was made to fill each of them. We decided to cache the first call and use the cache to fill up the other filters.

Limit the data sent by the API to the necessary

At page loading, around 5 big queries were made to the API, and each of them was retrieving full tables from the database. We selected the exact fields and filters necessary during API call-time to lighten the retrieved objects.

Set NODE_ENV to “production”

When you set NODE_ENV to "production", Express/Loopback cache view templates, cache CSS files generated from CSS extensions and generate less verbose error messages.

Cluster your node.js

If your server has a CPU with many cores, don't hesitate to see if it wouldn't be interesting to share the traffic weight on each of them through node.js clustering. We decided not to do that, because we were mainly having issues with the RAM of our server.

Cache request results

With Nginx for instance, you can cache the result of requests. Your app won't have to repeat the same operation several times.

Upgrade the server

If your server is struggling and you were not capable to improve the perf of your app with all the above solutions, maybe it's time to upgrade your server material power. We had to do so, and it changed the lives of our users.

Conclusion

Working on performance is a scary moment. It is like facing the great unknown, without knowing if we will come back from it. I hope that if you see yourself in a situation close to ours, this article will give you precious leads on what and how to investigate.

This article is aimed at helping the community. If you saw something that might deserve some more details or other solutions I did not detail, please don't hesitate to react to this article directly, and let us know what we could have done better.