ERRATUM 12/03/2019 : a mistake made its way into the algorithm, making it break in some circumstances. If you’re using the algorithm and you’ve seen it not working, this might be why. I address the issue in the relevant place in the article.

As a programmer, maybe you have come to a point where your game engine or library was missing that one feature that you needed. Then comes the dreaded moment where you have to look all over the internet for code written by people that tackled the issue before you (I see you, StackOverflow users). While there is nothing wrong with that (I sure am doing it), often times this is something that you could have done yourself, even for pretty theoretical stuff like the ones that pertain to geometry or shuffling. I’m one of those people that constantly want to understand how everything works, and what better way to understand an algorithm than to come up with it yourself, reinventing it on the fly if it already exists ?

Stating our sample problem

In this article, we will go through several steps that I think are pretty efficient in coming up with an algorithm of your own to solve problems. In order to apply them to something, we will keep a sample problem on the side : convex decomposition of simple polygons. It sounds all scientific and complicated, but it’s not too hard.

A simple polygon is a polygon that doesn’t self-intersect and doesn’t have any hole ; a convex polygon is a polygon where all angles are below 180°. Of course, a convex polygon is a simple polygon, and a simple polygon is a complex polygon (the most general class of polygons), but the converse is not always true.

One good thing about convex polygons is that testing collisions between two arbitrary convex polygons is really easy (we won’t cover it here), whereas testing collisions between two arbitrary complex, or even simple polygons is obscenely difficult. This is where convex decomposition comes into play : if we can divide a simple polygon into several smaller convex polygons, colliding with it is equivalent to colliding with at least one of the polygons of the decomposition. Because of this, our sample problem will be the following : given an array of points describing a simple polygon, return an array of arrays of points describing convex polygons that “add up” to the source polygon.

Know what you’re working with

When designing algorithms, the most important thing is to identify the problem that you are trying to solve, that is, what your algorithm is supposed to do in the first place. While that sounds silly, it’s not just what your algorithm should do, but more what it takes as input and gives as output (including data structures when relevant). Most of the time, knowing what data structures you are going to work with can help shape your reasoning. For example, if you are designing a sorting algorithm, chances are you are going to want an array as an input, but what should the output be, a new array, or nothing and the sorting is in-place (directly modifying the source array) ?

If you read my previous article on that one curves algorithm, it’s a good example of an algorithm whose input and output is sort of hard to caracterize. In fact, you could say that the algorithm takes a function of a float and an integer as input, with the function describing the parametric curve and the integer being the number of steps of the sampling ; it shall then return another function of a float as output, that is, the “curve” function of the article, which is basically a more regular version of the initial curve.

For our sample problem, and as we have stated before, our algorithm will take an array of points as input and return an array of arrays of points as output. The input will be the vertices of a simple polygon, and the output will be the vertices of all the convex polygons in the convex decomposition of the source polygon.

Brain and paper first

While this doesn’t only apply to the making of algorithms (or even programming), starting off with paper and pencil (pen for those who like to live dangerously) is one of the best ways to wrap your head around a problem. Of course, programming is no exception, so let’s jump right in and start from the beginning.

First off, most of the time you want to draw (or write out, depending on what you’re working with) simple cases that you then solve yourself, without thinking too much about what you’re doing in order to keep an intuitive approach. While doing that, or after you’re done, you should analyze your way of thinking about it and organize it all in steps. The idea is that whether you want it or not, you are executing an algorithm ; your brain is a computer too, just the most powerful computer known to man. So powerful that it can look at itself running and understand how it works ; so powerful that it already runs the algorithm that you are trying to write, but somehow it doesn’t understand it (because you don’t !). However, you can execute the algorithm step by step to get to know how it works. To test out that theory, let’s go back to our sample problem and use our simple polygon example.

I invite you to actually draw this polygon, and at this point stop reading and attempt to divide this polygon into smaller shapes so that you can never find inner angles above 180°. I gave you my solution on the picture above, but since everybody thinks differently, we may not come up with the same shapes at the end. It’s completely fine, in fact, convex decomposition of a single polygon is not unique !

Now that we just casually applied a computational geometry algorithm in seconds, without actually knowing it, all using the most powerful computer in the known universe, we can think back at how we just did that and divide it into steps. Again, because we all think differently, the exact steps may vary from one person to the other ; I will apply this to my reasoning, yours should be close enough.

First, I asked myself “why isn’t this convex already ?” ; the answer came fast enough : some inner angles are over 180° (exactly two of those, indicated in the picture by arrows), which by definition prevents the polygon from being convex. In order to correct that, I then thought that I just had to cut the angle in order to obtain two smaller angles that ideally wouldn’t be over 180°. This is obtained by linking the faulty vertex with some other vertex in the polygon. Repeat that for all faulty vertices, and we have our convex decomposition !

Okay so that all went pretty fast. What actually happened ? At this point, we can write an embryo of an algorithm in an English-like language called “pseudo-code”. It’s not exactly an English sentence, but it’s not quite programming either. It just looks like programming enough that it gets your “think like a programmer” engine going.

while there is a faulty vertex link one with another vertex in the polygon end while

We understand from this that we need to be able to tell if a vertex is faulty or not ; this is simply done by testing if the two edges to which the vertex belongs make an angle over 180° or not. There’s another, more serious problem though : which vertex do we choose to link to the faulty vertex ? Let’s think again at how we did it last time …

How I did it is that I tried to include as many vertices as possible into each polygon, as to minimize the number of polygons in the decomposition. This is generally a good idea, as it is more efficient to handle a rectangle than 2 triangles that together add up to a rectangle ; although they describe the same shape, you have 4 vertices in one case and 6 in the other. The way you do this is by testing every vertex in order, starting at the faulty vertex, until you find one that makes your polygon not convex anymore ; you then select the last one that kept it convex. This is always possible because a triangle is always convex, so in the worst case you’ll get a triangle. Now our pseudo-code looks like this :

while there is a faulty vertex while the next vertex forms a 180° or lower angle with the faulty vertex go to the next one if the angle formed by this new vertex is over 180°, stop and select the one before end while link the faulty vertex to the selected vertex end while

Now, this test could lead to a couple of hairy situations : what if the vertex right after our faulty vertex is faulty itself ? What if we run into a faulty vertex during our test ? The second question kind of solves itself from the observation that you can’t have a faulty vertex in a convex polygon ; you must immediately stop the search and select it, so that the edge that you add by cutting the angle turns it into a valid vertex. The first question can be solved from intuition : when we solve the problem by hand, this never happens because we willingly start with either the leftmost or rightmost faulty vertex, obviously not one in the middle of other faulty vertices. In code, this simply translates to starting the search from a faulty vertex that we know has a valid neighbour. This will always be possible, because a simple polygon always has at least one valid vertex (that is, non-faulty). Find it, use it to get rid of one faulty vertex, repeat. Our pseudo-code now looks like this :

while there is a faulty vertex choose one that has a valid vertex after it while the next vertex forms a 180° or lower angle with the faulty vertex go to the next one if this new vertex is invalid, stop and select it else if the angle formed by this new vertex is over 180°, stop and select the one before end while link the faulty vertex to the selected vertex end while

And in action, could give something like this :

Looks pretty good !

One more question remains : right now, we are storing our polygons as arrays of vertices, not edges ; what does “linking vertices” mean then ? We are not adding or removing any vertex from the polygon, so we can’t possibly modify the array of vertices … or can we ?

Let’s look at how we thought about it when we did it by hand : once the edge was drawn, we completely disregarded the part that was made convex, and solely focused on the remaining vertices. However, we did keep caring about the edge that was just added, and still included both of the vertices making it up in the search. This has a pretty clear interpretation : we just cut the polygon in two parts, one convex part and one simple part, and we simply stopped caring about the convex part to actually re-apply the exact same algorithm on the new, smaller simple polygon !

Now we understand what “linking” means : it’s basically creating a new polygon from all the vertices between your start and end points (respectively green and red in the picture, “between” in the sense that we are moving along the polygon), subtracting that polygon to the source polygon, and calling the algorithm again on the resulting smaller batch of vertices. Remember that both vertices that make up the edge that was added must be present in both polygons !

Every time we do a full iteration of the algorithm and you get one convex and one simple part, we shall add the convex part to an array : this is the array that the algorithm will ultimately return when it has completed, the array of all convex components of the original source polygon. As for the simple part – you guessed it – we call the algorithm on it again.

ERRATUM 12/03/2019 : as it turns out, this is not enough to guarantee that the resulting polygon is convex. Consider this example :

As you can see, the top-right polygon is not convex. The reason for this is that when searching for the first vertex that makes our polygon not convex, we don’t check that the current vertex can actually be seen from the base vertex.

However, as it turns out, merely checking visibility is not enough as you can get stuck in an infinite loop. The more straightforward (and probably less effective) solution is to simply call makeConvex on both polygons instead of only the remaining one. Fortunately, calling that on an already convex polygon is pretty cheap, so the hit on performance shouldn’t be too bad (the algorithm stays sub-quadratic in complexity).

Our pseudocode now looks like this :

function makeConvex choose a faulty vertex that has a valid vertex after it while the next vertex forms a 180° or lower angle with the faulty vertex go to the next one if this new vertex is invalid, stop and select it else if the angle formed by this new vertex is over 180°, stop and select the one before end while push in an array all vertices between the faulty vertex and the selected vertex, both included remove from the source polygon all vertices between the faulty vertex and the selected vertex, both excluded return the concatenation of both results of makeConvex called on the new polygon and the remaining one end function

And this is it, we are done ! Our brain and paper session is over ; after this point, any further work is implementation-related, so I will leave it to you ! As with the curves algorithm, I will provide soon a working implementation with code and screenshots.

End word

Keep in mind that this pseudo-code defines a somewhat naive approach and that it is not optimized. The point is not to make the most efficient algorithm, but to learn how to make your own, so that as you do more and more of those, one day you will have the right idea for a clever algorithm that maybe nobody had thought about before. If after reading this article, you are thinking that maybe you will start thinking about designing your very own solution to a certain problem before you start looking online, then I will feel like I have reached my goal.

Mattias Refeyton

ERRATUM 12/03/2019 : a mistake made its way into the algorithm, making it break in some circumstances. If you’re using the algorithm and you’ve seen it not working, this might be why. I address the issue in the relevant place in the article.