Finals hit pretty hard this semester, but on top of that was my first opportunity to do some real field work with this tech. Classmate Martzi Campos had a physical installation for her thesis project, a narrative driven Escape Room-like game, that had to be dismantled (possibly forever) before the end of Finals. I took this as an opportunity to crunch and try and capture this moment in time before it was gone, hoping to bring my framerate up from 0.5fps to a half decent speed and to add HD image capture so we could reverse project textures later onto meshes. And I’m happy to say, at 8fps now with full HD pictures per frame too, I managed to actually pull something off! Video below.

So here are the results so far. Beautiful Corner, Martzi’s thesis project, as a few seconds of the 6min capture we got for a full playthrough. This is a bare bones representation, as Unity + Point Cloud Viewer Free can only handle a hundred or two frames of animation right now before hitting memory issues, (point clouds are big!) and it’s all still in point cloud form rather than the cleaner/intended textured meshes, but the data is all there. These captures included 2.8k frames per device (6min at 8fps), and one pointcloud and one 1920x1080 JPEG for every frame. No, it’s not fully efficient yet - but it let us capture the thesis project in some 3d motion scan form before it was dismantled, and doing that in a way that replays half decently was my real goal here.



The installation itself.

Efficiency

It took a lot to get to that framerate with that data though, and the efficiency strides were huge even to get here. This was a 12x increase in capture speed, while also doubling the data recorded! How? Binary storage and post processing, at the good recommendation of Ashok - all data is almost immediately dumped now (although point clouds are still processed before dumping, but that will likely change soon, just fixing some bugs there) and all compression, string writing, and anything else is done in post. That means quick capture speeds, and then waiting 10x the length of your capture again in post processing time when you try to quit. Our 6min of footage took over an hour to export - not ideal, but it got us to 8fps before striking the set.

So many kinects!

Capture

Capturing the day of was a real pain too! (Did I say I cut it close?) Not only did this mean some last minute shipping charges to get two more Kinect to Windows Adapters, and a scramble around USC to find more Kinect 2.0s, but also the terrible realization that four kinects did not just work like they should have, and only three could run on the current hardware with the current settings. Even libfreenect2′s suggested fixes for buffer sizes did no good, so we ended up with four kinects and only three in use. A bummer, but thankfully, not the end of the world. Microsoft’s Holoportation setup makes a lot more sense to me now though: to get decent captures of people in the space, we ended up having to sacrifice the ceiling and overhead lamps in the capture, and had all three kinects angled more down than up. Having secondary devices below/looking up would have greatly improved the capture.

Not pictured: tolerant faculty posing for calibration.

Filesize

Oh, and those 2.8k frames? They come in at about 20GB total right now - and that’s after JPEG compression. Midcapture, it’s closer to 50GB with all the redundant binary files that store empty points. Even properly exported, OFF files are not a small format either, so I’ll be looking into PCL’s PCD next and seeing if Mimesys’ Q3D is an open source format too. Ideally: we would capture as compressed HD video and 512x424 single channel depth video, and export only as mesh sequences and UV textures from video files - but that’s a few months off from right now. Next, OpenCV for video capture maybe?

And the ladders that served as last minute tripods.

Conclusion

Ultimately, wow, this stuff is crazy possible. To even get three kinects running at 8fps in less than a week from where I was before (admittedly with a serious number of crunch hours) shows how close this stuff is to realization. And there’s lots of competition I never realized exists out there too. But we’re so close to a videogrammetry future here that you can really taste it - capturing straight depth videos instead of point clouds should make this even faster, as should capturing video instead of individual JPEGs. If we can hit that 30fps mark, a Kinect 2.0′s maximum capture rate, I think we’ll have as strong of a setup as we need to both 1) prove this is cool stuff and people do want it, and 2) start making some seriously cool captures of the present on a widely distributed scale. We’re at the cusp of capturing the world as spaces in motion, and I can’t wait to see what people do with this tech next.

Thanks also to Martzi Campos for letting me capture and show Beautiful Corner! I’m excited to have the opportunity to help see a live/immersive piece like this captured in animated volumetric form.