I need a golang Levenshtein distance library, and I need a fast one. There are plenty of implementations out there. Which is the best?

I found these:

I then wrote a simple benchmark, one for each library. This isn’t a terribly scientific test if the libraries have different strengths (perhaps one is faster with longer strings?), but it gives us something to start with. I’ve also, perhaps foolishly, assumed they all actually work.

func BenchmarkLevenshtein_arbovm(b *testing.B) {

s1 := "frederick"

s2 := "fredelstick"

total := 0 b.ReportAllocs()

b.ResetTimer() for i := 0; i < b.N; i++ {

total += arbovm.Distance(s1, s2)

} if total == 0 {

b.Logf("total is %d", total)

}

}

Here are the results:

BenchmarkLevenshtein_dgryski-8 3000000 422 ns/op 80 B/op 1 allocs/op BenchmarkLevenshtein_texttheater-8 1000000 1287 ns/op 1200 B/op 11 allocs/op BenchmarkLevenshtein_kse-8 5000000 373 ns/op 0 B/op 0 allocs/op BenchmarkLevenshtein_honzab-8 500000 3419 ns/op 192 B/op 2 allocs/op BenchmarkLevenshtein_arbovm-8 3000000 442 ns/op 80 B/op 1 allocs/op BenchmarkLevenshtein_agnivade-8 2000000 629 ns/op 192 B/op 2 allocs/op

Unfortunately kse, dgryski and honzab have no licence, so I can’t use them. arbovm is my best bet. (@dgryski’s implementation is almost identical to arbovm’s. Both are based on http://en.wikibooks.org/wiki/Algorithm_implementation/Strings/Levenshtein_distance#C).

But arbovm does an allocation for each distance calculation. It isn’t just that I don’t like allocations. My application involves doing ~1 billion distance calculations in a fairly tight loop, so any allocations will stress GC. My application has a small number of long-lived goroutines, so I can add a per-goroutine context structure to hold memory for the distance calculation.

The allocation is for an []int to hold intermediate results. Each call to Distance(s1,s2) only does one allocation, and it is done with once the call is complete. In the future we can hope the Go compiler will spot this slice does not escape the function and allocate it on stack instead of on heap.

So I can introduce a context structure as follows to solve my problem.

type Context struct {

intSlice []int

} func (c *Context) getIntSlice(l int) []int {

if cap(c.intSlice) < l {

c.intSlice = make([]int, l)

}

return c.intSlice[:l]

}

I can then either alter Distance(s1, s2 string)int to have an additional parameter Distance(c *Context, s1, s2 string) int or I can make Distance a member of Context : func (c *Context) Distance(s1, s2 string) int . This is just a matter of taste.

Then I create the context once for each goroutine, and call context.Distance() as many times as I like without ever causing an allocation.

Here’s the final benchmark (I’ve also switched to go 1.8rc2, which makes quite an improvement). We’ve gone from 316ns and 1 allocation to 265ns and 0 allocations. My small tweaks are here.

BenchmarkLevenshtein_philpearl-8 5000000 265 ns/op 0 B/op 0 allocs/op BenchmarkLevenshtein_arbovm-8 5000000 316 ns/op 80 B/op 1 allocs/op

Please press the little heart if you’ve enjoyed this: I’m powered by internet points.

By day, Phil fights crime at ravelin.com. You can join him: https://angel.co/ravelin/jobs