Get your existing app ready for HoloLens 2

07/29/2020

9 minutes to read





In this article

Overview

This guide is tailored to help developers with an existing Unity application for HoloLens (1st gen) port their application for the HoloLens 2 device. There are four key steps to porting a HoloLens (1st gen) Unity application to HoloLens 2.

The sections below detail information for each stage:

Step 1 Step 2 Step 3 Step 4 Download latest tools Update Unity Project Compile for ARM Migrate to MRTK v2

Prerequisites:

It's highly recommended that you use source control to save a snapshot your applications original state before starting the porting process. Additionally, we recommend saving checkpoint states at various times during the process. It can also be helpful to have another Unity instance of the original application to compare side-by-side during the porting process.

Note Before porting, ensure you have the latest tools installed for Windows Mixed Reality development. For most existing HoloLens developers, this involves updating to the latest version of Visual Studio 2019 and installing the appropriate Windows SDK. The content that follows dives further into different Unity versions and the Mixed Reality Toolkit (MRTK) Version 2. For more information, please see Install the tools.

Migrate project to the latest version of Unity

If you're using MRTK v2, Unity 2019 LTS is the best long-term support path with no breaking changes in Unity or in MRTK. You should assess any plugin dependencies that currently exist in your project, and determine whether or not these DLLs can be built for ARM64. If a hard dependency plugin cannot be built for ARM64, you may need to continue building your app for ARM.

After updating to Unity 2019 LTS, it's recommended that you update particular settings in Unity for optimal results on the device. These settings are outlined in detail under recommended settings for Unity.

It should be reiterated that the .NET scripting back-end is being deprecated in Unity 2018 and removed in Unity 2019. Developers should strongly consider switching their project to IL2CPP.

Note IL2CPP scripting back-end can cause longer build times from Unity to Visual Studio, and developers should set up their developer machine for optimizing IL2CPP build times. It might also be beneficial to set up a cache server, especially for Unity projects with a large amount of assets (excluding script files) or constantly changing scenes and assets. When opening a project, Unity stores qualifying assets into an internal cache format on the developer machine. Items must be re-imported and re-processed when modified. This process can be done once and saved in a cache server and consequently shared with other developers to save time, as opposed to every developer processing the re-import of new changes locally.

After addressing any breaking changes from moving to the updated Unity version, you should build and test your current applications on HoloLens (1st gen). This is a good time to create and save a commit into source control.

Compile dependencies/plugins for ARM processor

HoloLens (1st gen) executes applications on an x86 processor while the HoloLens 2 uses an ARM processor. Therefore, existing HoloLens applications need to be ported over to support ARM. As noted earlier, Unity 2018 LTS supports compiling ARM32 apps while Unity 2019.x supports compiling ARM32 and ARM64 apps. Developing for ARM64 applications is preferred, as there is a material difference in performance. However, this requires all plugin dependencies to also be built for ARM64.

Review all DLL dependencies in your application. It is advisable to remove any dependency that is no longer needed from your project. For remaining plugins that are required, ingest the respective ARM32 or ARM64 binaries into your Unity project.

After ingesting the relevant DLLs, build a Visual Studio solution from Unity and then compile an AppX for ARM in Visual Studio to test that your application can be built for ARM processors. It is advised to save the application as a commit in your source control solution.

Important Application's using MRTK v1 can be run on HoloLens 2 after changing the build target to ARM, assuming that all other requirements are met. This includes making sure you have ARM versions of all your plugins. However, your app won't have access to HoloLens 2 specific functions like articulated hand and eye tracking. MRTK v1 and MRTK v2 have different namespaces that allow both versions to be in the same project, which is useful for transitioning from one to the other.

MRTK Version 2 is the new toolkit on top of Unity that supports both HoloLens (1st gen) and HoloLens 2. It is also where all the new HoloLens 2 capabilities have been added, such as hand interactions and eye tracking.

Check out the following resources for more information on using MRTK version 2:

Prepare for the migration

Before ingesting the new *.unitypackage files for MRTK v2, it is recommended to take an inventory of 1) any custom-built code that integrates with MRTK v1 and 2) any custom-built code for input interactions or UX components. The most common and prevalent conflict for a mixed reality developer ingesting MRTK v2 involves input and interactions. It is advised to begin reading and understanding the MRTK v2 input model.

Finally, the new MRTK v2 has transitioned from a model of scripts and in-scene manager objects to a configuration and services provider architecture. This results in a cleaner scene hierarchy and architecture model, but requires a learning curve for understanding the new configuration profiles. Thus, please read the Mixed Reality Toolkit Configuration Guide to start becoming familiar with the important settings and profiles to adjust to the needs of your application.

Perform the migration

After importing MRTK v2, your Unity project most likely has many compiler-related errors. These are commonly due to the new namespace structure and new component names. Proceed to resolve these errors by modifying your scripts to the new namespaces and components.

For information on the specific API differences between HTK/MRTK and MRTK v2, see the porting guide on the MRTK Version 2 wiki.

Best practices

Prefer use of the MRTK standard shader.

Work on one breaking change type at a time (ex: IFocusable to IMixedRealityFocusHandler).

Test after every change and use source control.

Use default MRTK UX (buttons, slates, etc.), when possible.

Refrain from modifying MRTK files directly; create wrappers around MRTK components. This action eases future MRTK ingestions and updates.

Review and explore sample scenes provided in the MRTK, especially HandInteractionExamples.scene.

Rebuild canvas-based UI with quads, colliders, and TextMeshPro text.

Enable Depth Buffer Sharing or set focus point; prefer to use a 16-bit depth buffer for better performance. Ensure when rendering color, to also render depth. Unity generally does not write depth for transparent and text gameobjects.

Set Single Pass Instanced Rendering Path.

Utilize the HoloLens 2 configuration profile for MRTK

Testing your application

In MRTK Version 2, you can simulate hand interactions directly in Unity as well as develop with the new APIs for hand interactions and eye tracking. The HoloLens 2 device is required to create a satisfying user experience. You are encouraged to start studying the documentation and tools for greater understanding. MRTK v2 supports development on HoloLens (1st gen) and traditional input models, such as select via air-tap can be tested on HoloLens (1st gen).

Updating your interaction model for HoloLens 2

Once your application is ported and prepped for HoloLens 2, you're ready to consider updating your interaction model and hologram design placements. In HoloLens (1st gen), your application likely has a gaze and commit interaction model with holograms relatively far away to fit into the field of view.

Steps to update your application design to be best suited for HoloLens 2:

MRTK components: Per the pre-work, if you added MRTK v2, there are various components/scripts to leverage that have been designed and optimized for HoloLens 2. Interaction model: Consider updating your interaction model. For most scenarios, we recommend switching from gaze and commit to hands. With your holograms typically being out of arms-reach, switching to hands results in far interaction pointing rays and grab gestures. Hologram placement: After switching to a hands interaction model, consider moving some holograms closer to directly interact with them, by using near-interaction grab gestures with your hands. The types of holograms recommended to move closer to directly grab or interact are small target menus, controls, buttons, and smaller holograms that fit within the HoloLens 2 field of view when grabbing and inspecting the hologram.

Every application and scenario are different, and we’ll continue to refine and post design guidance based on feedback and continued learnings.

Additional caveats and learnings about moving applications from x86 to ARM

Straight-forward Unity applications are simple because you can build an ARM application bundle or deploy directly to the device for the bundle to run. Some Unity native plugins can present certain development challenges. Because of this, you must upgrade all Unity native plugins to Visual Studio 2019 and then rebuild for ARM.

One application used the Unity AudioKinetic Wwise plugin and that version of Unity did not have a UWP ARM plugin, which caused a considerable effort to rework sound capabilities into the application in question to run on ARM. Ensure that all required plugins for your development plans are installed and available in Unity.

In some cases, a UWP/ARM plugin might not exist for application-required plugins, which blocks the ability to port the application and run it on HoloLens 2. Contact your plugin provider to resolve the issue and provide support for ARM.

The minfloat (and variants such as min16float, minint, etc.…) in shaders might behave differently on HoloLens 2 than on HoloLens (1st gen). Specifically, these guarantee that at least the specified number of bits will be used. On Intel/Nvidia GPUs, these are largely treated as 32 bits. On ARM, the number of bits specified is actually adhered to. That means in practice, these numbers might have less precision or range on HoloLens 2 than they did on HoloLens (1st gen).

The _asm instructions don’t appear to work on ARM, meaning any code using _asm instructions must be rewritten.

The SIMD instruction set is not supported on ARM because various headers, such as xmmintrin.h, emmintrin.h, tmmintrin.h, and immintrin.h, are not available on ARM.

The shader compiler on ARM runs during the first draw call after the shader has been loaded or something the shader relies on has changed, not at shader load time. The impact on framerate can be noticeable, depending on how many shaders need to be compiled. This has various implications for how shaders should be handled, packaged, updated differently on HoloLens 2 vs HoloLens (1st gen).

See also