TSP Algorithms and heuristics

Although we haven’t been able to quickly find optimal solutions to NP problems like the Traveling Salesman Problem, "good-enough" solutions to NP problems can be quickly found [1].

For the visual learners, here’s an animated collection of some well-known heuristics and algorithms in action. Researchers often use these methods as sub-routines for their own algorithms and heuristics. This is not an exhaustive list.

For ease of visual comparison we use Dantzig49 as the common TSP problem, in Euclidean space. Dantzig49 has 49 cities — one city in each contiguous US State, plus Washington DC.

Dantzig49

The optimum solution to Dantzig49.

1: Greedy Algorithm

A greedy algorithm is a general term for algorithms that try to add the lowest cost possible in each iteration, even if they result in sub-optimal combinations.

In this example, all possible edges are sorted by distance, shortest to longest. Then the shortest edge that will neither create a vertex with more than 2 edges, nor a cycle with less than the total number of cities is added. This is repeated until we have a cycle containing all of the cities.

Greedy Algorithm

Although all the heuristics here cannot guarantee an optimal solution, greedy algorithms are known to be especially sub-optimal for the TSP.

2: Nearest Neighbor

The nearest neighbor heuristic is another greedy algorithm, or what some may call naive. It starts at one city and connects with the closest unvisited city. It repeats until every city has been visited. It then returns to the starting city.

Nearest Neighbor Heuristic

Karl Menger, who first defined the TSP, noted that nearest neighbor is a sub-optimal method:

"The rule that one first should go from the staring point to the closest point, then to the point closest to this, etc., in general does not yield the shortest route."

The time complexity of the nearest neighbor algorithm is O(n^2) . The number of computations required will not grow faster than n^2.

3: Nearest Insertion

Insertion algorithms add new points between existing points on a tour as it grows.

One implementation of Nearest Insertion begins with two cities. It then repeatedly finds the city not already in the tour that is closest to any city in the tour, and places it between whichever two cities would cause the resulting tour to be the shortest possible. It stops when no more insertions remain.

Nearest Insertion

The nearest insertion algorithm is O(n^2)

4: Cheapest Insertion

Like Nearest Insertion, Cheapest Insertion also begins with two cities. It then finds the city not already in the tour that when placed between two connected cities in the subtour will result in the shortest possible tour. It inserts the city between the two connected cities, and repeats until there are no more insertions left.

Cheapest Insertion

The cheapest insertion algorithm is O(n^2 log2(n))

5: Random Insertion

Random Insertion also begins with two cities. It then randomly selects a city not already in the tour and inserts it between two cities in the tour. Rinse, wash, repeat.

Random Insertion

Time complexity: O(n^2)

6: Farthest Insertion

Unlike the other insertions, Farthest Insertion begins with a city and connects it with the city that is furthest from it.

It then repeatedly finds the city not already in the tour that is furthest from any city in the tour, and places it between whichever two cities would cause the resulting tour to be the shortest possible.

Farthest Insertion

Time complexity: O(n^2)

7: Christofides Algorithm

Christofides algorithm is a heuristic with a 3/2 approximation guarantee. In the worst case the tour is no longer than 3/2 the length of the optimum tour.

Due to its speed and 3/2 approximation guarantee, Christofides algorithm is often used to construct an upper bound, as an initial tour which will be further optimized using tour improvement heuristics, or as an upper bound to help limit the search space for branch and cut techniques used in search of the optimal route.

For it to work, it requires distances between cities to be symmetric and obey the triangle inequality, which is what you'll find in a typical x,y coordinate plane (metric space). Published in 1976, it continues to hold the record for the best approximation ratio for metric space.

Christofides Algorithm

The algorithm is intricate [2]. Its time complexity is O(n^4)

8: 2-Opt

A problem is called k-Optimal if we cannot improve the tour by switching k edges.

Each k-Opt iteration takes O(n^k) time.

2-Opt is a local search tour improvement algorithm proposed by Croes in 1958 [3]. It originates from the idea that tours with edges that cross over aren’t optimal. 2-opt will consider every possible 2-edge swap, swapping 2 edges when it results in an improved tour.

2-Opt

2-opt takes O(n^2) time per iteration.

9: 3-Opt

3-opt is a generalization of 2-opt, where 3 edges are swapped at a time. When 3 edges are removed, there are 7 different ways of reconnecting them, so they're all considered.

3-Opt

The time complexity of 3-opt is O(n^3) for every 3-opt iteration.

10: Lin-Kernighan Heuristic

Lin-Kernighan is an optimized k-Opt tour-improvement heuristic. It takes a tour and tries to improve it.

By allowing some of the intermediate tours to be more costly than the initial tour, Lin-Kernighan can go well beyond the point where a simple 2-Opt would terminate [4].

LKH

Implementations of the Lin-Kernighan heuristic such as Keld Helsgaun's LKH may use "walk" sequences of 2-Opt, 3-Opt, 4-Opt, 5-Opt, “kicks” to escape local minima, sensitivity analysis to direct and restrict the search, as well as other methods.

LKH has 2 versions; the original and LKH-2 released later. Although it's a heuristic and not an exact algorithm, it frequently produces optimal solutions. It has converged upon the optimum route of every tour with a known optimum length. At one point in time or another it has also set records for every problem with unknown optimums, such as the World TSP, which has 1,900,000 locations.

11: Chained Lin-Kernighan

Chained Lin-Kernighan is a tour improvement method built on top of the Lin-Kernighan heuristic.

It takes an existing tour produced by the Lin-Kernighan heuristic, modifies it by "kicking" it, and then applies Lin-Kernighan heuristic to it again. If the new tour is shorter, it keeps it, kicks it, and applies Lin-Kernighan heuristic again. If the original tour is shorter, it kicks the old tour again and applies Lin-Kernighan heuristic.

Chained LKH

Depending on its implementation it may stop when there are no more improvements, or when it has reached a time limit, or a tour of a maximum length, etc.

Being a heuristic, it doesn't solve the TSP to optimality. However it is a subroutine used as part of the exact solution procedure for the state of the art Concorde TSP solver [5].

Closing Thoughts

This is not an exhaustive list, but I hope the selected algorithms applied on Dantzig49 can give a good impression of how some well-known TSP algorithms look in action.

The Traveling Salesman Problem

The Traveling Salesman Problem is one of the most studied problems in computational complexity. Given a set of cities along with the cost of travel between them, the TSP asks you to find the shortest round trip that visits each city and returns to your starting city.

Nobody has been able to come up with a way of solving it in polynomial time. We’re not sure if it's even possible.

Harvard's Hassler Whitney first coined the name "Travelling Salesman Problem" during a lecture at Princeton in 1934. It became known in the United States as the 48-states problem, referring to the challenge of visiting each of the 48 state capitols in the shortest possible tour. Alaska and Hawaii weren’t US states back then.

Dantzig49

Dantzig49 was the first non-trivial TSP problem ever solved. It’s a variant of Whitney’s 48 states problem, using one city for each state, plus Washington DC. The road distances used in Dantzig49 were those available on a Rand McNally map, so not all cities were state capitals.

There are (n-1!)/2 possible tours to any TSP problem, so Dantzig49 has 6,206,957,796,268,036,335,431,144,523,686,687,519,260,743,177,338,880,000,000,000 possible tours (~6.2 novemdecillion tours). If you ask a computer to check all of those tours to find the shortest one, long after everyone who is alive today is gone it will still be trying to find the answer.

The large (factorial) brute-force search space of the TSP doesn’t inherently mean there can’t be efficient ways to solve the TSP. There are other problems that have even larger search spaces, yet we have algorithms that can efficiently solve them. The Minimum Spanning Tree problem is one example. But without an efficient algorithm for the TSP, this factorial search space contributes to the TSP’s difficulty.

The optimum solution to Dantzig49. One of 6,206,957,796,268,036,335,431,144,523,686,687,519,260,743,177,338,880,000,000,000 possible tours

It was solved in 1954 by Danzig, Fulkerson and Johnson. They introduced novel techniques, enabling them to solve Dantzig49 without inspecting all possible tours. They did it by hand, using a pin-board and rope. Their work paved the way for new heuristics.

NP and P

We can't quickly find the optimal solution to a TSP problem. Our best-known exact solving techniques can take a long time for even a modest number of cities. Specifically, we can't solve them in polynomial time. We also can't quickly verify the solutions even when we have them. This makes it an NP-Hard problem. It has a variant that can be written as a yes/no question. That 'decision' variant is NP-Complete . NP-Complete problems also can't be solved in polynomial time, but their solutions can be verified in polynomial time.

Not all problems take too long to solve, though. We group the problems that we can quickly solve (in polynomial time) as P .

It could be possible that a quick method for solving an NP-Complete problem exists, and we just haven't found it yet, making P=NP. Or, it could be impossible for a quick method to exist. Knowing which one of these two possibilities is true is a million dollar question [6][7].

The TSP's solvability has implications beyond just computational efficiency. One of the unsolved questions in Economics is whether markets are efficient. There is proof that markets are efficient if and only if P = NP [8]. This has implications on the type of economic policies governments enact. Free market vs regulated market, small government vs big government, etc.