This is my biggest post yet! Sequenced point clouds are both capturing and playing back in real time for me - the biggest milestone yet for this project. Ashok even helped me get it running in VR on the Vive too, and that turned out fantastic looking there (though looks worse in gif format). It’s a pretty dumb implementation right now and not fit for long term storage, but for proof of concept, this is great!

Stack

The stack on this is still pretty clean: it’s just libfreenect2 for capture, outputting multiple .off files every second, and while it doesn’t appear to be going a full 30fps for the kinect 2 (not tracking those numbers yet) it’s certainly getting enough data to make something half decent for a first take. Then Point Cloud Viewer takes all those separate .off files, loops through importing them, and then loads them all at once to enable/disable them in order for playback.

Limits

The limitations right now are the .off files, alignment, and an artificial cap on sequence length. The files are all string based so they’re much slower to write than binary, but we can fix that by writing to binary at capture time and then converting later (or ideally giving Point Cloud Viewer binary file support). Alignment is a huge issue as it is currently done by hand, so you can see the offset issues near the end where instead I should be using some form of point matching here, something PCL might be able to fix (haven’t read that one yet). The artificial cap on length is just 100 frames - we can lift that cap at any time, but it is better to do so when we have more control over recording and the write speeds.

Naming

Filenaming was an issue that came up to get to this point (this was a particularly large crunch week actually for all this) but after not finding any videogrammetry file formats in my searches, I based it resembling camera standards. Currently it is “date_time_frame_device” so an example of “20160503_1742_000089_01″ represents Device 01′s capture of frame 89 at a time of 5:42pm on May 3rd, 2016. Long term, I’d prefer to have all frames within one file (and all devices too) but this keeps things a bit more legible while just a proof of concept.

What Next?

So, we did it! It is entirely possible to use multiple kinect 2.0s to capture animated point clouds of people and play them back in VR using tools available today.

Next from here is actually splitting up that capture data into two parts: an HD rgb image for each frame, and a depth image for each frame. This will make that same data more useful long term, as storing it in its most raw format means we can revisit it later with more advanced tech and improve the quality of these captures - and that I can work from home on the data without devices. Capturing point cloud data is likely easier since point clouds are actually less data (rgb is only 512x424 then instead of 1080p) but if we want to reconstruct any textured meshes, the HD images will help a lot.

Also on the todo list is throwing money at Microsoft for another Kinect or two, and seeing if USB 3 expansion cards are needed for additional buses or controllers (which I have heard is an issue).

But for today, I am satisfied. This is the realization of an idea three years in the making. So for me, even in all its pixelated glory, this is just awesome.