This is the big stuff folks. I sat down and made bulk rendering functions and bulk mesh display functions in my PCL and Unity programs respectively, and the results are really cool. Texture alignment is not perfect, I had to switch reconstruction techniques from Poisson to Greedy Triangulation, the PCDs are still pretty noisy before reconstruction, and the average render takes 20min (for one frame) with Unity only capable of displaying about 100 frames in one scene before it starts crashing… but oh man. There is something really cool about walking around in a video and choosing what angle to view it from. Full video footage showing this after the break, along with lots of new developments.

The Beautiful Corner capture is great for walking around in, even with holes.

The Fun Stuff

First, what worked. Watch the video above for another walkthrough in Martzi’s space as she walks in, but now with textured polygons everywhere. You will quickly learn the pros and cons of this specific capture now: having three kinects at three out of four corners of the room means a quarter of our needed points are missing… and that shows up pretty badly with reconstruction. Point clouds left a lot to the imagination, which our brains would fill in automatically, but rendered polygons and textures mean her face missing is a much bigger problem than it used to be. It shows a lot about how to do or not do future captures now that we finally see this one more fully rendered.

Even so, walking around the space as she moves through it is so cool. The surfaces make things feel like you have a sense of navigation, even though it is a captured space (and now with more clarity since less is left to imagine). Microsoft’s Holoportation of people is cool, but I’m really glad my interest in capturing spaces too is paying off as much as it is. I can’t wait to travel to places I’ve never been just by people capturing footage like this from around the world and uploading it online. Imagine Valve’s Destinations, but animated.

Greedy triangulation is not the hole filling reconstruction that Poisson offered.

The Bummer Stuff

Alright, so the biggest issue in videogrammetry rears its ugly head again: reflections. Everything feels… pasty, like clay, since nothing is reflective like it should be. The blue treasure chest in real life has a light shine to it. The lamp’s lighting reflects off the surface of the desk differently depending on your angle. None of this applies in videogrammetry, since we’re not capturing lightfields but video from a few angles to reconstruct into polygons.

I am lucky then in that I get to work at USC’s MxR studio, and other people are working on similar issues. It sounds like someone else has found a cool half-fix for this, so hopefully there will be more news on that soon, but that is a discussion for another day.

The rollercoaster of emotions that is my notes document.

The Tech Stuff

A lot has happened in the last month, so I’m going to try and squeeze it down to just the highlights for the sake of still releasing blogposts regularly instead of getting backlogged trying to document everything. Something is likely wrong in my kinect parameters for reprojection, as you can see in the textures being about a half foot off from where they should be. Poisson really liked merging surfaces together, but too often they were surfaces that shouldn’t go together and it caused these “bubbles” where shapes appeared out of no where by collecting some noisy stray points, so I switched to the more conservative Greedy Triangulation which fills in almost no holes. It looks alright, but some smoothing is much needed.

Everything takes a LONG time right now - rendering at 20min / frame is pretty far from Microsoft’s streaming FVV technology. I am not using any parallel stuff for that yet though, so we should be able to do 4-8 frames per 20min soon (or a frame every 2.5min ideally, which is at least much more doable). Importing into Unity is not yet automated, so I have to update a bunch of materials by hand with the correct shaders (unlit and transparent) to get you this quick proof of concept / example footage today. That footage of me waving is some 36 frames, and that was a two day turnaround from capture to playback (12hr render times will do that). But the higher density point clouds do look a lot better, so that was definitely worth it. The 100 frames of Martzi footage took more than half a week. Luckily, once you have the gigs of data on your computer, loading time and playback take only a few seconds - so users won’t experience much in terms of technical issues if size is not a problem for them.

A timelapse of Martzi walking into the space: every frame at once in Unity.

Next Steps

So some great news - I got the go ahead to work on this for credits towards graduating (did I mention I’m pursuing a Master’s?). This is going to give me more opportunities to collaborate on this stuff and keep giving it hours every week, which given the code’s momentum, I’m very excited about. Parallel is next on the tech side for faster renders, along with fixing texture alignment and reducing noise so these renders come out cleaner than ever. (And maybe we can finally see someone’s face)

I’d love to get the framerate higher first so we can get higher density captures by added more devices (we’re still only hitting three kinect 2.0s, and I think four will be the magic number) without hitting unplayable framerates, or the hope of someday hitting 24fps, but that might require abandoning libfreenect2 to see if PCL can handle multi kinect captures. We shall see! Until then, I’d rather get these renders a little more cleaned up first. Microphone support through the kinects is important, but I’m not sure how high to prioritize that vs the rest of this right now since I don’t know its processing costs. If it’s trivial, I may give it a shot sooner rather than later.

But still, this project has finally reached the point of its original intention: animated textured reconstruction, even if it’s not perfect yet, and that’s something worth celebrating for sure. And once framerates are higher, captures will be denser and creating some experiences that are really worth sharing, and I can’t wait to see those.