Registration is hard. The most effective alignment technique, Iterative Closest Point (ICP) is both the most and least powerful tool available. It is the most powerful because, when it works, it can get everything exactly lined up better than anything else. It is the least powerful because it only works when already 98% aligned. I recommend it even so, having tried other techniques, and for now just leaving the other 98% to a human.

Above: Before ICP. Note how misaligned the carpet edge is between the three clouds.

Iterative Closest Point

ICP works really well within its limits, and people recommend it all the time, but people don’t talk about those limits very well. The more similar two clouds’ points are, the more misaligned they can be before aligning them. In my case, with two wholly unique clouds that only share some overlap, this means they must be fairly closely aligned (in distance and rotation) before alignment can work. My initial attempts, where both clouds were aligned on all but one axis, but shared very few similar points, led to significant failures. They became exactly aligned on those other axises, but made no attempt at all to fix that remaining axis. In my latest code, a user must align the clouds as closely as they can, and then ICP takes care of the rest. This works best when the clouds share some similarities, but are not exactly the same.

The tutorial image vs my results with their exact code. Not encouraging.

Robust Alignment

The most disappointing development for this post was learning that even the best rough alignment techniques have significant limitations. PCL has a tutorial for most things, but when none of the example code worked as shown (especially when designed specifically for extreme cases) it indicates certain limitations of the field. Alignment is possible, we know that, but currently it is such a significant undertaking that it exceeds the scope of this phase of the project. And that’s a bummer - it would be great to automatically align any point clouds we want to merge, but currently that is beyond the system’s abilities for the time I can invest looking into this.

Each keypress took about 2-3 seconds to update in the scene. Yuck.

Human Element

And that led to me giving up on my goal of a fully automated alignment process. Instead, now we have the uglier (but functional) approach of having someone manually line up multiple clouds before a finer alignment. It’s pretty ugly, it takes a while, and it’s not perfect - but it works and it gets us to a point where ICP can kick in and do its job. The end results are not perfect, but far better than anything we had before, and look fairly elegant too. In short: we now have something that works, even if it’s not quite what we wanted.

The treasure chest looks fine. The “desk and dresser” on the right however…

Next Steps

Oh man, surface reconstruction reared its ugly head as soon as I gave it an aligned point cloud. It looks like I have my work cut out for me still, as my attempts to apply older reconstruction techniques led to even worse results, and will require some major new revisions/research to eliminate the relief-like effect that we’re seeing right now. I’m not looking forwards to a deep dive into another single subject matter, likely to just lead to another rescinding of prior goals, but at least we have a functional form of alignment now. Let’s hope that by next time I can say the same for reconstruction.