Deep Spelling

Rethinking spelling correction in the 21st century

I implemented my first spelling corrector years ago based on Peter Norvig’s excellent tutorial — a spelling corrector in 21 lines of Python code.

And it sucked.

So I tried to fix it. I piled on double metaphone phonetic similarity, unicode support, multi-word expressions, weighted Damerau-Levenshtein edit-distance, efficient Trie and smart caching.

And it still sucked.

And the reason it failed to go beyond a simple toy is very simple — I am not Google (and neither are you).

Even in its simplest form, spelling a short word took a long time — say ~0.1 seconds. That might be ok for some uses, but when you are dealing with real-time chat, faced with the upcoming bot gold rush, interactivity reigns supreme. And don’t forget that spelling is only a single chain in the long processing pipeline of delivering value via Natural Language Understanding, dialog management, business logic and of course the actual application.

The root cause of the abysmal performance is that the speller is trying to brute-force its way to the right solution. Here is the core of Norvig’s code:

Nasty Brute Force

The function looks at every possible edit to the input — a deletion of any character, a transposition of any 2 adjacent characters, replacing any character in the input with a random character or simply inserting a random character. And for each of the results in the set of edited strings — it calculates every possible edit again!

The result is a challenging amount of computation that needs to happen, and which grows exponentially with respect to the length of the input string.