What’s the Problem?

Performance is (unfortunately) Emergent, meaning that individual systems performance may not inform the larger combined systems performance. Even if this is not strictly 100% true, it’s difficult to predict the performance of end user system until the system is fully put together and running on hardware that will be servicing the production workload. This has 2 important implications:

Difficult to predict performance (Favor Observation)

System Specific (Local Doesn’t Predict Remote)

Because of this, we often see the mantra: Make it Work then Make it Fast emerge. This suggests that we shouldn’t worry about performance until we have a working project. While I think this is better advice than the other end of the spectrum, there is plenty of low overhead work we can do to design performance in.

Finally! Performance is:

Defined by clients, and a proxy for their experience! Performance is the executable validation of a design. Performance will happen either Explicitly, under our control and guidance or implicitly, hitting against unforeseen user expectations or unforeseen physical limitations (such as the speed of light).

Ensuring Code is Performant

Discover

What are the known or discoverable constraints?

Explicit Constraints

These will often be viewed from a client’s perspective. These are often contractual ie a customer is paying for a certain level of service:

Response must be served with 100ms

99.99% of requests need to be < 1 second

The two most common sources for these are:

Contractual

Product Management

Implicit Constraints

Implicit constraints are performance limits hidden in requirements. If a program is doing daily reports or hourly rollups, the interval of those actions may defined the expected limit. These are constraints that are often end user usability experience (UX) focused. For example if an action is blocking in the user path there is a ~100ms limit for the site to feel “responsive”.

Physical Limitations of Systems

Physical limitations are often dictated by the minimum achievable performance in the physical world! If certain latencies are required between largely separated locations performance will run up against the speed of light! There are other performance critical constraints based on the current speed of hardware, seen in the “Latency Numbers Every Programmer Should Know” infogram.

Reasonable Ranges

Reasonable ranges are formed based on knowledge of physical system limitations and experience of observing systems in production. Suppose that there is a go service that accepts and HTTP connection, makes a get request to redis and then writes and closes the HTTP connection. The service is being tested locally by running both the service and redis and driving traffic using apache benchmark. Locally is a 2019 macbook pro with 6 core and 16GB of ram. On the first test the service is only able to handle 50 requests per second! Based on experiences this is unreasonable. This service should be able to push a couple hundred requests per second easy.

Design

Design phase is focused on materializing the structure of a concrete implementation. This is where discovered constraints are combined with implementation strategies. Most of the performance work should be done here, as this is where the solution is most malleable and has the least cost associated with changing.

Measurement Strategy

First step is to define a measuring strategy:

What is being measured?

Where is it being measured?

How will it be collected?

And finally what will we use to determine if this is “performant”? Google’s strategies for choosing good SLOs highly align with this step.

Theoretical Performance

Theoretical Design takes into account theoretical performance and runtime complexity of implementation algorithms and datastructures:

It also includes system specific implications of performance, like implications of schema choices on database query execution.

Isolatable Components — Testable Design

This brings us to the code level. There are a number of techniques that reduce friction in isolating components for individual benchmarks and in providing stub (highly controllable) implementations for performance testing: