The four variations are shown above. The first was a screenshot of the current homepage to be used as a baseline. The other three variations consisted of our proposed redesign plus two slightly more conservative variations with the search field shown by default.



About 1,400 people accepted our pop-up invitation and took the test. For many of the projects we work on, it might take weeks to recruit that many test participants, but thanks to WRAL.com’s massive amount of traffic, we reached that number in less than four hours.

Results

You can see how each variation performed below. The first thing to notice is the remarkable consistency of success rates for three of the four designs. (The success rate was defined as the percentage of participants who clicked in the areas shown below.) For all except Variation 2, the success rates are essentially exactly the same, which is pretty amazing.

Variation 1. Baseline Variation 2. Hidden Input Variation 3. White Box Variation 4. Blue Box Success Rate 86%

95% CI [81, 90%] 73%

95% CI [67, 78%] 86%

95% CI [81, 90%] 85%

95% CI [80, 90%] Task Time 3.8 sec.

95% CI [3.5, 4.1] 7.8 sec.

95% CI [7, 8.7] 4.2 sec.

95% CI [3.9, 4.5] 4.9 sec.

95% CI [4.5, 5.4] Task times were calculated using the geometric mean in order to calculate confidence intervals. Chalkmark does not report confidence intervals, so I found helpful calculators for task success and task time. Task time calculations include successful selections only.

The second notable thing is the comparatively poorer performance of the design with a hidden input. Around 13% fewer participants clicked within the correct area, and those who did took about twice as long to do so than in the baseline design.

One thing to consider is that at least some of the participants were already familiar with the current design and may have used that search box before. When you consider that natural advantage of familiarity, Variations 3 and 4 can reasonably be considered more successful than the baseline design.

Analysis

First, a few caveats. First-click testing is a super easy way to compare the recognizability of elements in a fictional interface with a potentially large sample, but it has its shortcomings. Aside from the extremely narrow scope allowed by first-click testing, we can’t ask people why they did what they did or even confirm that they understood what they were doing.

Indeed, the heatmaps that Chalkmark provides are as confounding as they are insightful. People clicked on headlines, navigation items, faces, and in empty spaces. It would seem that some people simply forgot they were taking a test and carried out the task that they came to WRAL.com for in the first place. Other people likely clicked accidentally, not realizing that their first click would immediately end the test.

However, these errant clicks don’t invalidate the test, because they show up in every variation in a consistent pattern. For example, compare Variations 3 and 4, which are identical except for the color of the search box. The stray clicks, while hard to explain, are amazingly consistent.

In some cases, the heatmaps offer real insight. Look at how many people clicked in the blank space at the very top of Variation 2, the hidden input design. In the absence of a search box, some participants seemed to click where they thought a search box should be.