One problem I faced while developing the planets for Epic Lander, was that at some zoom levels all vertexes of the planets could be visible, and as such culling by a bounding box tree was made useless, and even added more overhead. After spending a lot of timing optimizing small parts of the rendering system, I decided the best option would be to create different sets of geometry for rendering based on the zoom level.

My first attempt at the algorithm, was essentially the standard downsampling algorithm used to change the pitch of an audio sample. Although it seemed like it would be a good idea, given how it smooths as it reduces the number of points, it was actually very noticeable. When zooming in the geometry would shift, and the effect was unpleasant and confusing.

After studying this effect, and wondering what was wrong with it, I realized that the problem was that downsampling caused all the vertexes to move, due to the fact the vertexes were being averaged with their close neighbours.

At this point I had established two things,

1. Moving vertexes is very noticeable,

2. But, vertexes must be removed to improve speed

That means that I have to remove some vertexes, without moving any of the original ones. And given the fact that I’m trying to minimalize noticeability, I want to remove the vertexes which will be missed the least.

After having a look at the terrain, it was easy to see that the biggest points were much more noticeable than the smoother areas, and so the points that needed to be removed were the ones where the angle from the point before, and the point after was minimal.

Explained in pictures:

Given your initial terrain, which consists of vertexes and edges:

Calculate the angles between the each vertex and it’s adjacent vertexes:

Remove the first N (in this case 6) vertexes, ordered least noticeable first (Using this metric, where the angle is closest to PI)

Recreate edges through the deleted vertexes:

Although I have not tested this method for terrain in 3D games, I believe it should work. In the case of having a 2D height map instead of a 1D one, you will likely want to store 4 pointers of the adjacent vertexes for each vertex, and then update the links of adjacent vertexes when a vertex is removed. One drawback of this method is that a uniform grid will no longer be uniform, however for quick spatial queries combining an array with a linked will be nearly as fast.