The Test

With the help of some indispensable technical guidance from Viget’s Nate Hunzaker, I built a Rails app to administer an unmoderated icon recognition test on the Web. It’s easier just to take the test yourself than for me to explain it, but here’s my attempt at a brief description:

Before the test, participants are asked to familiarize themselves with the 20 labeled icons. Then, participants are led through 24 icon recognition trials in which they are presented with the name of one of the 20 icons and asked to select the matching icon as quickly and accurately as possible from a circular array of 19 other distractor icons. Each trial uses one of the 20 icons as its target, and the first four trials are considered warm-ups, which are excluded from data analysis. The test app randomly determines the sequence of target icons, the position of each icon in the array, and whether all icons are shown in a solid or hollow style. The test takes about five minutes to complete.

Go ahead and take the test yourself at icon-test.net for a better idea of how it works. I've been told that it's actually kind of fun!

Results

Over a 10-day data collection period, 1,260 tests were completed, for a total of more than 25,000 individual icon recognition trials. The demographics skewed towards young (18-40), tech-savvy Apple users. The mean selection time was almost exactly three seconds, with a standard deviation of 1.5 seconds.

Averaged across all 20 icons, hollow icons were selected about 0.1 second slower than solid icons, which would seem to support Johnson’s assertion that hollow icons require more cognitive effort to recognize than solid icons (if only a very small amount more). However, the picture isn’t yet complete.

I haven’t mentioned yet that my study actually included another aesthetic variable in addition to icon style: icon color. Each trial in the test actually showed icons in one of four style-color combinations: solid black-on-white, hollow black-on-white, solid white-on-black, or hollow white-on-black. (Side note: the question of whether or not black and white should be considered colors sparked a lively debate in the designers' chat room at Viget.)