MacKenzie, I. S., Kauppinen, T., & Silfverberg, M. (2001). Accuracy measures for evaluating computer pointing devices. Proceedings of the ACM Conference on Human Factors in Computing Systems - CHI 2001, pp. 9-16. New York: ACM. [software]

Accuracy Measures for

Evaluating Computer Pointing Devices I. Scott MacKenzie1, Tatu Kauppinen2, & Miika Silfverberg2 1Department of Computer Science

York University

Toronto, Ontario, Canada M3J 1P3

smackenzie@acm.org 2Nokia Research Center

P.O. Box 407

FIN-00045 Nokia Group, Finland

tatu.kauppinen@nokia.com, miika.silfverberg@nokia.com



Abstract

In view of the difficulties in evaluating computer pointing devices across different tasks within dynamic and complex systems, new performance measures are needed. This paper proposes seven new accuracy measures to elicit (sometimes subtle) differences among devices in precision pointing tasks. The measures are target re-entry, task axis crossing, movement direction change, orthogonal direction change, movement variability, movement error, and movement offset. Unlike movement time, error rate, and throughput, which are based on a single measurement per trial, the new measures capture aspects of movement behaviour during a trial. The theoretical basis and computational techniques for the measures are described, with examples given. An evaluation with four pointing devices was conducted to validate the measures. A causal relationship to pointing device efficiency (viz. throughput) was found, as was an ability to discriminate among devices in situations where differences did not otherwise appear. Implications for pointing device research are discussed. Keywords: Computer pointing devices, performance evaluation, performance measurement, cursor positioning tasks

INTRODUCTION

The pointing device most common in desktop systems is the mouse, although others are also available, such as trackballs, joysticks, and touchpads. Mouse research dates to the 1960s with the earliest publication from English, Engelbart, and Berman [6]. The publication in 1978 by Card and colleagues at Xerox PARC [4] was the first comparative study. They established for the first time the benefits of a mouse over a joystick. Many studies have surfaced since, consistently showing the merits of the mouse over alternative devices (e.g., [7, 9, 13]). This paper focuses on the evaluation of computer pointing devices in precision cursor positioning tasks. The primary contribution is in defining new quantitative measures for accuracy that can assist in the evaluations.

PERFORMANCE EVALUATION

The most common evaluation measures are speed and accuracy. Speed is usually reported in its reciprocal form, movement time (MT ). Accuracy is usually reported as an error rate - the percentage of selections with the pointer outside the target. These measures are typically analysed over a variety of task or device conditions.

An ISO standard now exists to assist in evaluating pointing devices. The full standard is ISO 9241, "Ergonomic design for office work with visual display terminals (VDTs)". Part 9 is "Requirements for non-keyboard input devices" [8]. ISO 9241-9 proposes just one performance measurement: throughput. Throughput, in bits per second, is a composite measure derived from both the speed and accuracy in responses. Specifically,

(1)

where

(2)

The term ID e is the effective index of difficulty, in "bits". It is calculated from D, the distance to the target, and W e , the effective width of the target. The use of the "effective" width (W e ) is important. W e is the width of the distribution of selection coordinates computed over a sequence of trials, calculated as

(3)

where SD x is the standard deviation in the selection coordinates measured along the axis of approach to the target. This implies that W e reflects the spatial variability (viz. accuracy) in the sequence of trials. And so, throughput captures both the speed and accuracy of user performance. See [5, 10] for detailed discussions.

NEW ACCURACY MEASURES



Figure 1. A "perfect" target-selection task

In practice, this behaviour is rare. Many variations exist and all occur by degree, depending on the device, the task, and other factors. In this section, we identify some of these behaviours and formulate quantitative measures to capture them.

We are not suggesting that it is wrong to report error rates. Rather, our goal is to augment this with more expressive measures of accuracy - measures that can assist in characterizing possible control problems that arise with pointing devices.

Movement Variability

Consider the trackball's means to effect pointer motion. To move the pointer a long distance, users may "throw" the ball with a quick flick of the index finger, whereas more precise pointer movement is effected by "walking" the fingers across the top of the ball. These behaviours, which are not possible with other pointing devices, may affect the pointer's path. Such effects may not surface if analyses are limited to movement time or error rates.

Dragging tasks are particularly challenging for trackballs. This has been attributed to an interaction between the muscle groups to effect pointer motion (index finger) vs. those to press a button (thumb) [11]. In the study cited, however, only movement times and error rates were measured. Since these are gross measures (one per trial), their power in explaining behaviour within a trial is limited. Here we see a clear need for more detailed measures that capture characteristics of the pointer's path.

Several measures are possible to quantify the smoothness (or lack thereof) in pointer movement, however analyses on the path of movement are rare in published studies. (For exceptions, see [1, 12].) One reason is that the computation is labour-intensive. The pointer path must be captured as a series of sample points and stored in a data file for subsequent analysis. Clearly, both substantial data and substantial follow-up analyses are required.

An example of a task where the path of the pointer is important is shown in Figure 2. When selecting items in a hierarchical pull-down menu, the pointer's path is important. If the path deviates too far from the ideal, a loss of focus occurs and the wrong menu item is temporarily active. Such behaviour is undesirable and may impact user performance.



Figure 2. The importance of pointer path

Several measures are now proposed to assist in identifying problems (or strengths) for pointing devices in controlling a pointer's movement path. Figure 3 shows several path variations. Note that the pointer start- and end-point are the same in each example. Clearly, accuracy analyses based only on end-point variation cannot capture these movement variations.

We begin by proposing several simple measures that require only that certain salient events are logged, tallied, and reported as a mean or ratio.

Target Re-entry (TRE ). If the pointer enters the target region, leaves, then re-enters the target region, then target re-entry (TRE ) occurs. If this behaviour is recorded twice in a sequence of ten trials, TRE is reported as 0.2 per trial. A task with one target re-entry is shown in Figure 3a.



Figure 3. Path variations. (a) target re-entry (b) task axis crossing

(c) movement direction change (d) orthogonal direction change

An example where target re-entry was not used, yet may have helped, is Akamatsu et al.'s evaluation of a mouse with tactile feedback [2]. This study found a main effect on fine positioning time - the time to select the target after the pointer entered the target region. With tactile feedback, users exhibited a lower fine positioning time than under the no feedback, auditory feedback, and colour feedback conditions. A measure such as target re-entry may also serve to reveal differences among on-target feedback conditions, for example.

Other counts of path accuracy events are possible, and may be relevant, depending on the device or task.

Task Axis Crossing (TAC ). In Figure 3b, the pointer crosses the task axis on the way to the target. In the example, the ideal path is crossed once, so one task axis crossing (TAC ) is logged. This measure could be reported either as a mean per trial or a mean per cm of pointer movement.

TAC may be valuable if, for example, the task is to trace along a pre-defined path as closely as possible.

Movement Direction Change (MDC ). In Figure 3c, the pointer's path relative to the task axis changes direction three times. Each change is logged as a movement direction change (MDC ).

MDC and TAC are clearly correlated. One or the other may be of interest, depending on the task or device.

Orthogonal Direction Change (ODC ). In Figure 3d, two direction changes occur along the axis orthogonal to the task axis. Each change is logged as one orthogonal direction change (ODC ). If this measure is substantial (measured over repeated trials), it may signal a control problem in the pointing device.

The four measures above characterize the pointer path by logging discrete events. Three continuous measures are now proposed: movement variability, movement error, and movement offset.

Movement Variability (MV ). Movement variability (MV ) is a continuous measure computed from the x-y coordinates of the pointer during a movement task. It represents the extent to which the sample points lie in a straight line along an axis parallel to the task axis.

Consider Figure 4, which shows a simple left-to-right target selection task, and the path of the pointer with five sample points.



Figure 4. Sample coordinates of pointer motion

Assuming the task axis is y = 0, y i is the distance from a sample point to the task axis, and y[overbar] is the mean distance of the sample points to the task axis. Movement variability is computed as the standard deviation in the distances of the sample points from the mean:

(4)

For a perfectly executed trial, MV = 0.

Movement Error (ME ). Movement error (ME ) is the average deviation of the sample points from the task axis, irrespective of whether the points are above or below the axis. Assuming the task axis is y = 0 in Figure 4, then

(5)

For an ideal task, ME = 0. As with MDC and TAC, ME and MV are likely correlated. One or the other may bear particular merit depending on the movement characteristics of the device.

Movement Offset (MO ). Movement offset (MO ) is the mean deviation of sample points from the task axis. Assuming the task axis is y = 0 in Figure 4, then

(6)

Movement offset represents the tendency of the pointer to veer "left" or "right" of the task axis during a movement.

For an ideal task, MO = 0. Several movement responses, and the relative values of movement variability, error, and offset are shown in Figure 5.