Hand menu

08/27/2019

5 minutes to read





In this article

The hand menu is one of the most unique UX patterns in HoloLens 2. It allows you to quickly bring up hand-attached UI. Since it's accessible anytime and can be shown and hidden easily, it's great for quick actions.

You'll find our recommended best practices for working with hand menus in the list below. You can also find an example scene demonstrating the hand menu in MRTK.

Best practices

Keep the number of buttons small

Due to the close distance between a hand-locked menu and the eyes and also the user's tendency to focus on a relatively small visual area at any time (the attentional cone of vision is roughly 10 degrees), we recommend keeping the number of buttons small. Based on our exploration, one column with three buttons works well by keeping all the content within the field of view (FOV) even when a user moves their hands to the center of the FOV.

Utilize hand menu for quick action

Raising an arm and maintaining the position could easily cause arm fatigue. Use a hand-locked method for the menu requiring a short interaction. If your menu is complex and requires extended interaction times, consider using world-locked or body-locked instead.

Button / Panel angle

Menus should billboard towards the opposite shoulder and middle of the head: This allows a natural hand move to interact with the menu with the opposite hand and avoids any awkward or uncomfortable hand positions when touching buttons.

Consider supporting one-handed or hands-free operation

Do not assume both of the user's hands are always available. Consider a wide range of contexts when one or both hands are not available, and make sure your design accounts for those situations. To support a one-handed hand menu, you can try transitioning the menu placement from hand-locked to world-locked when the hand flips (goes palm down). For hands-free scenarios, consider using a voice command to invoke the hand menu.

Avoid adding buttons near the wrist (system home button)

If the hand menu buttons are placed too close to the home button, it may accidentally trigger while interacting with the hand menu.

Hand menu with large and complex UI controls

When the menu is world-locked, make sure to provide a way to move the menu, and close the menu when it's no longer needed. Make the menu movable by providing handles on the sides or top of menu. Add a close button to allow the menu to close. Allow for the menu to re-attach to the hand when the user hand faces the user. We also recommend requiring that the users gazes at their hand to prevent false activations (see below).

Large menu that shows a usability issue

World-locked menu on hand drop

Manual grab & pull to world-lock the menu

How to prevent false activation

If you use just palm-up as an event to trigger the hand menu, it may accidentally appear when you don't need it (false-positive), because people move their hands both intentionally (for communication and object manipulation) and unintentionally. To reduce false activations, add an additional step besides the palm-up event to invoke the hand menu (such as fully opened fingers, or the user intentionally gazing at their hand).

Require Flat Palm

By requiring a flat open hand, you can prevent false activation that might occur as the user manipulates objects or gestures while communicating within an environment.

Require Gaze

By requiring the user to gaze at their hand (either with eye gaze or head gaze), it prevents false activations due to the user having to intentionally direct their attention to the hand as a secondary activation step (with a tunable distance threshold used to allow for user comfort).

Hand menu placement best practices

In human anatomy, the ulnar nerve is a nerve that runs near the ulna bone. The ulna is a long bone found in the forearm that stretches from the elbow to the smallest finger.

Below are 2 recommended placements based on our explorations:



A. Ulnar inside palm

This position is reliable because the hands do not overlap each other. This is critical for accurate hand detection and tracking.

B. Ulnar above hand

This location is comfortable for users because they don't need to raise their arm too much to interact with the hand menu. We recommend placing menus 13cm above the palm and align the buttons inside the ulnar palm. Read more about the optimal button size



For technical reasons we recommend this location with one required implementation: the developer will need to freeze the menu once the user's opposite hand gets close to interacting with it. This will avoid jitteriness from overlapping hands and also allows for a faster targeting of the buttons.



HoloLens 2 cameras identify hands accurately when they are separate from each other. Any overlapping hands can cause hand menus move away from the anchor location.



Menu positions that are not recommended

We have done user research with different menus layouts and locations, the following menu locations are NOT recommended, find the cons of each study below:



Above the arm

1 - Difficult to maintain good hand tracking

2 - Causes user fatigue due to unnatural position

Above fingers

1 - Hand fatigue due to holding out hand for a long time

2 - Hand tracking issues on index and middle fingers



Above-center palm

1 - Hand tracking issues due to overlapping hands

2 - Hand fatigue due to holding hands for long time in order to interact with menus Top fingertip

1 - Hand tracking issues

2 - Hand fatigue from holding hand above normal posture

3 - Issues pressing buttons with other fingers by accident due to limited space between fingers



Back of the arm

1 - Can trigger home button by accident

2 - Not a natural or comfortable position

Hand menu in MRTK (Mixed Reality Toolkit) for Unity

MRTK provides scripts and example scenes for the hand menu. The HandConstraintPalmUp solver script allows you to attach any objects to the hands with various configurable options. MRTK's hand menu examples include useful options such as flat palm and gaze requirement for preventing false activation.

You can try hand menu examples in HoloLens 2 with MRTK Examples Hub app.

See also