We just returned from a weeklong AI+MR Hackathon at the Microsoft Reactor space in Redmond where we integrated a variety of Microsoft's cognitive services into HoloLens and Windows Mixed Reality (WMR) Immersive (VR) experiences. We couldn't believe how easy it is to build solutions around the LUIS Language Understanding cognitive service, as well as the Microsoft Translator APIs. Other teams had similar success with Face recognition and Custom Vision services.

Within 3 days we had a comprehensive framework for easily integrating LUIS into Unity projects, and we integrated language translation into our audio chat capability within Prism by Object Theory. If any participant speaks a different language, the audio is automatically translated to the desired language as a thought bubble next to the speaker's avatar.

We left the hackathon realizing that with very little effort, we could have been benefiting from LUIS in all of our HoloLens solutions we've delivered over the past three years. We offer extensive voice commands in all of our apps to make them easier to use without a keyboard and extensive menus. We struggle to remember all of them and we've evolved better commands over time from app to app, which means even more variations to remember. To help, we include a "Show Voice Commands" voice command to show a list of all supported commands in each app. But this is not enough.

With LUIS, we train its machine learning backend to recognize a wide variety of ways to say the same command and it quickly learns how to map all of them to the right intent. For example in our Prism app, we can easily train it to recognize "Open the session manager", "Change sessions", "Start a new session", "See my sessions", and "Connect to a session" as all meaning open the panel for managing sessions. Better yet, it's easy to further train the LUIS engine over time based on actual utterances that have been used in the app by actual users, by scanning the anonymous list of utterances captured by LUIS, and mapping any unrecognized utterances to the correct intent.

Conclusion: If you're developing VR or MR applications, I highly recommend using LUIS to make them more user friendly. We definitely plan to embrace LUIS across the board and Translator for Prism.

Next steps: We plan to implement an utterance-to-intent mapping cache so whenever the device is not connected to the cloud, it can still benefit from understanding a wide variety of utterances during these offline periods.

The LUIS Unity framework, to which we contributed at the hackathon, is open source and available at https://github.com/Microsoft/mixedreality-azure-samples/tree/feature/LUIS

All links:

https://github.com/Microsoft/mixedreality-azure-samples/tree/feature/LUIS/





– Michael Hoffman, Co-founder, Object Theory

Twitter: m_the_hoff | LinkedIn: mthoffman | Web: objecttheory.com



