In a recent blog, we examined the performance difference between the runtimes of languages that AWS Lambda supports natively. Since that experiment was specifically interested in the runtime differences of a ‘warm’ function, the ‘cold start’ times were intentionally omitted.

A cold start occurs when an AWS Lambda function is invoked after not being used for an extended period of time resulting in increased invocation latency.

Since the cold start times of AWS Lambda is an important performance consideration, let’s take a closer look at some experiments designed to isolate the variables which may impact the first-time invocations of functions.

Testing methodology

From my experience running Lambda functions in production environments, cold starts usually occurred when an AWS Lambda function is idle for longer than five minutes. More recently, some of my functions didn’t experience a cold start until after 30 minutes of idle time. Even if you keep your function warm, a cold start will occur about every 4 hours when the host virtual machine is recycled — just check out the metrics by IO Pipe.

For testing purposes, I needed a reliable method for consistently ensuring a cold start of an AWS Lambda function. The only surefire way to create a cold start is by deploying a new version of a function before invocation.

For the experiment, I created 45 variations of AWS Lambda function. Using the Serverless framework setup below, it was easy to create variants of the same function with different memory sizes.

I recursively deployed all 45 functions and invoked each of them programmatically using the simple script below.

The deployment and invocation loop took about three minutes. To collect a meaningful amount of data points, I ran the experiment for over 24 hours.

My initial hypothesis

I based the hypothesis on my knowledge that the amount of CPU resources is proportional to the amount of memory allocated to an AWS Lambda function.

C# and Java would have higher cold start time Memory size affects cold start time linearly Code size affects cold start time linearly

Now it was time to see if the experiments supported my hypothesis.

Experiment # 1 — Cold start time by runtime & memory

To evaluate the the impact of memory on cold starts, I created 20 functions with 5 variants — using different memory sizes for each language runtime. The supported languages are C#, Java, Node.js, and Python.

I created 5 variants of the same hello world function (using different memory sizes) for each language runtime

After running the experiment for a little over 24 hours, I collected the following data — here’s the results: