As augmented reality (AR) technologies barrel toward ubiquity, a key concern among ethicists remains preserving privacy in data on which the underpinning AI models train. In order to localize themselves in space, AR apps require access to device cameras, which they use to construct digital maps of their surroundings. These maps are often stored persistently either on-device or remotely, and although images aren’t retained along with them, malicious actors with access to them could theoretically infer geometry, appearance, the layout of private spaces, knowledge about objects contained in those spaces, and other sensitive information.

That’s why a team at Microsoft and academic collaborators are investigating techniques that obfuscate data without sacrificing performance or accuracy. In a pair of papers scheduled to be presented at the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2019) conference next week in Long Beach, California, they demonstrate that 3D point clouds and camera localization features are prone to privacy attacks, and they offer a solution that involves transforming the 3D points such that scene geometry is concealed.

In the first of the two papers (“Revealing Scenes by Inverting Structure from Motion Reconstructions”), scientists from Microsoft Research and the University of Florida show that the aforementioned AR point clouds, reconstructed using structure from motion or simultaneous localization and mapping (SLAM), retain enough information to rebuild detailed scene images even after the source pictures have been discarded. Their novel AI system reconstructs color images in the scene given 2D projections of sparse 3D points and their features, even in the absence of attributes like visibility and in cases where the points are irregularly distributed.

Image Credit: Microsoft Research

The second paper (“Privacy Preserving Images-Based Localization”) — the work of ETH Zurich and Microsoft’s Mixed Reality & AI lab — posits a privacy-preserving approach to AR localization that lifts 3D points in maps to randomly oriented 3D lines that pass through the original points. The new representations bear little resemblance to the original scenes on their face, but still allow for precise positioning because of the close correlation among the 2D image points and the 3D lines.

The teams say they’re investigating algorithms for camera localization services that will run on the cloud, and they believe that the mitigation technique in the second paper will enable users to share maps with AR apps in a privacy-preserving fashion.