Above is a .off point cloud of my beautiful mug, clad in orange shirt, captured from the side by a kinect 2.0. It’s a bit rough, but for a first shot at capturing said data in libfreenect2, it worked out pretty well.

So last week involved a huge crunch on starting to learn point cloud file formats, how libfreenect2 makes that data accessible, and how to display it in Unity. Luckily, all three are more straightforward than you might think.

The source image looked something like this. Except with less smiles.

File formats for point clouds don’t look to be one for efficiency right now or minified data, but they sure make it easy to work in. Object File Format was what I settled on after surfing the net for a bit and going through PCL’s documentation (and .pcd does look like a better file format long term) and Meshlab’s examples (which let me practice with premade .off files). It’s straightforward, already has online examples in case my data goes bad, and looks like it will work with libfreenect2 and our Unity plugin.

Libfreenect2′s accessible data made point clouds look much easier to work with than creating and texturing 3d models, although I had done neither before from data. Their API is fairly straightforward, and included a function called getPointXYZRGB(). Once the data is registered (combining camera color and depth) you can grab the pixel by its x and y (512x424 for kinect 2.0 depth) and get back a position in 3d space and a color - the same parameters most point cloud file formats use (”x y z r g b") including the one we already had.

Unity plugins then offer two options currently, Point Cloud Viewer and Tools ($75) and Point Cloud Free Viewer ($0). For now, we have implemented the latter - which the reviews describe quite accurately as “only a few changes enable the reading of any point cloud format offered by the competition“ (ie. it almost works, and you can fix it yourself). There’s no quick reading of files, and it’s very prone to errors that need cleaning if you mess with the content files - but it can ultimately get the job done. I’m hoping to switch to the alternative soon, but for now this proves that point clouds from libfreenect2 data can be displayed in Unity.

The “at least I tried” merged point cloud so far.



Next Steps

Miraculously, capturing two point clouds at once ended up being trivial. Getting the two to display together in a meaningful way, that’s another beast to tackle. Above you can see my only attempt at putting two clouds together so far, and results are far from pretty. But that is the next step - getting the two to line up correctly so we have meaningful data to view from multiple angles.

Animation comes after that, where I will likely accost my computer’s drive with a file for every frame until functionality is proven, and then refine for efficiency and elegance from there. But even so, at this point, things are looking very good for kinect 2.0 videogrammetry. Now to see if all this data is intelligible when animated, or if we need more accuracy than this device can provide.