Illustration created for “A Journey With Go”, made from the original Go Gopher, created by Renee French.

ℹ️ This article is based on Go 1.13.

While developing in Golang, a mutex can encounter starvation issues when it consistently tries to acquire a lock that it is never able to get. For this article, we’ll look at a starvation issue affecting Go 1.8 that was solved in Go 1.9.

Starvation

In order to illustrate a starvation situation with a mutex, I will take the example made by Russ Cox about the issue in which they discuss mutex improvement:

This example is based on two goroutines:

goroutine 1 holds the lock for a long time and briefly releases it

goroutine 2 briefly holds the lock and releases it for a long time

Both have a cycle of 100 microseconds, but since goroutine 1 is constantly requesting the lock, we could expect it will get the lock more often.

Here is an example, done with Go 1.8, of the lock distribution with a loop of 10 iterations:

Lock acquired per goroutine:

g1: 7200216

g2: 10

The mutex has been acquired ten times by the second goroutine, while the first one got it more than seven million times. Let’s analyze what is happening here.

First, goroutine 1 will get the lock and sleep for 100 microseconds. When goroutine 2 tries to acquire the lock, it will be added to the lock’s queue — FIFO order — and the goroutine will go into waiting:

Figure 1 — lock acquisition

Then, when goroutine 1 finishes its work, it will release the lock. This release will notify the queue to wake goroutine 2 up. Goroutine 2 will be marked as runnable and is waiting for the Go Scheduler to be run on a thread:

Figure 2— goroutine 2 is awoke

However, while goroutine 2 is waiting to run, goroutine 1 will request the lock again:

Figure 3— goroutine 2 is waiting to run

When goroutine 2 tries to get the lock, it will see it has a hold already and will go into waiting mode, as seen in figure 2:

Figure 4— goroutine 2 tries again to get the lock

The acquisition of the lock by goroutine 2 will depend on the time it takes for it to run on a thread.

Now that the problem has been identified, let’s review the possible solutions.

Barging vs Handoff vs Spinning

There are many ways to deal with a mutex, such as:

Barging. This is designed to improve throughput. When the lock is released, it will wake up the first waiter and give the lock to either the first incoming request or this awoken waiter:

barging mode

It is how Go 1.8 was designed and reflects what we have seen previously.

Handoff. When released, the mutex will hold the lock until the first waiter is ready to get it. It reduces the throughput here since the lock is held even if another goroutine would be ready to acquire it:

handoff mode

We can find this logic in the mutex of the Linux Kernel:

Lock starvation is possible because mutex_lock() allows lock stealing, where a running (or optimistic spinning) task beats the woken waiter to the acquire. Lock stealing is an important performance optimization because waiting for a waiter to wake up and get runtime can take a significant time, during which everyboy would stall on the lock.

[…] This re-introduces some of the wait time, because once we do a handoff we have to wait for the waiter to wake up again.

In our case, a mutex handoff would perfectly balance the lock distribution between the two goroutines, but will decrease the performance since it would force the first goroutine to wait for the lock even if it is not held.

Spinning. If a mutex is different from a spinlock, it can combine a bit of its logic. Spinning could be useful when the waiter’s queue is empty or when the application makes intensive use of mutexes. Parking and unparking goroutines have a cost and could be slower than just spinning waiting for the next lock acquisition:

spinning mode

This strategy is used by Go 1.8 as well. When trying to acquire a lock already held, the goroutine will spin few times if the local queue is empty and if the number of processors is greater than one — spinning with one processor would just block the program. After spinning, the goroutine will park. It acts as a fast path in case the program is intensively using the lock.

For more information about how locks are designed — barging, handoff, spinlock— in general, Filip Pizlo made a must-read article “Locking in WebKit”.

Starvation mode

Prior to Go 1.9, Go was combining barging and spinning mode. In version 1.9, Go solved the issue previously explained by adding a new starvation mode that will lead to a handoff during the unlocking mode.

All goroutines that wait for the lock for more than one millisecond, also called bounded waiting, will be flagged as starving. When flagged as starving, the unlock method will now hand off the lock to the first waiter directly. Here is the workflow:

starvation mode

The spinning is also deactivated in starvation mode since the incoming goroutines will not have any to acquire a lock that is reserved for the next waiter.

Let’s run the previous example with Go 1.9 and the new starvation mode:

Lock acquired per goroutine:

g1: 57

g2: 10

The results are now more fair. Now, we could wonder if this new layer of control has an impact on the other cases when the mutex is not in a situation of starvation. As we can see in the benchmarks (Go 1.8 vs. Go 1.9) for the package, the performance did not decrease for the other cases (performance slightly changes with a different number of processors):