To convince journalists about the audio quality of its new HomePod smart speaker (here’s my review), Apple did something smart: Before we were given our review units, we were required to attend a listening session. Mine was held in Apple’s New York City public-relations loft, a mockup of an apartment.

Four speakers were on a counter against a wall: Sonos One ($200), Google Home Max ($400), the HomePod ($350), and the Amazon Echo Plus ($150).

The PR person could switch playback from one speaker to the other without missing a beat. They even had a halo light rigged to turn on behind whichever speaker was playing, so you’d know which was which.

There was not a shred of doubt: In this side-by-side comparison, the HomePod sounded better than its competitors.

Most of the reviews, including mine, said the same thing: that the HomePod isn’t as smart as the other smart speakers (among other problems, its voice control is limited to iTunes and Apple Music — no Spotify), but that it sounds amazing.

What Hi-Fi (a British audiophile site): “The HomePod is the best-sounding smart speaker available—and by quite a margin.”

(a British audiophile site): “The HomePod is the best-sounding smart speaker available—and by quite a margin.” Pocket-Lint (tech site): “The best sounding speaker of its type.”

(tech site): “The best sounding speaker of its type.” The Verge: “It sounds far better than any other speaker in its price range.”

“It sounds far better than any other speaker in its price range.” Tech Crunch: “HomePod is easily the best sounding mainstream smart speaker ever. It’s got better separation and bass response than anything else in its size.”

Still, when I tweeted about the test, a couple of people were suspicious of the setup, which of course was entirely controlled by Apple. What was the source material? What was the wireless setup?

An Apple rep told me that the test songs were streaming from a server in the next room (a Mac). But each speaker was connected to it differently: by Bluetooth (Amazon Echo), Ethernet (Sonos), input miniplug (Google Home), and AirPlay (HomePod), which is Apple’s Wi-Fi-based transmission system.

Since the setup wasn’t identical, I wondered if it was a perfectly fair test. (Bluetooth, for example, may degrade (compress) the music it’s transmitting, depending on the source and the equipment.)

So I decided to set up my own test at home.

The setup

I hid the four speakers behind a curtain — a sheet of thin, sheer fabric that wouldn’t affect the sound. It took me a Sunday to figure out how to get the A/B/C/D switching to work seamlessly, but I finally managed it: All four speakers would be streaming from Spotify, all four over Wi-Fi. I’d use the Spotify app’s device switcher to hop among speakers without missing a beat.

View photos The speakers hidden. More

I chose five songs, each with different styles, instrumentation, and sonic demands:

“Star Wars: Imperial March.” Full orchestra, full volume, full of brass.

Full orchestra, full volume, full of brass. “Havana” (Camila Cabello). Current pop hit. Distinct bass, drums, piano, and voice. Lots of rhythm.

Current pop hit. Distinct bass, drums, piano, and voice. Lots of rhythm. Brandenburg Concerto No. 3 in G Major. All strings, full range of pitches and dynamics.

All strings, full range of pitches and dynamics. “Hallelujah” (Pentatonix). A cappella ballad, five voices, very exposed and close to the mikes.

A cappella ballad, five voices, very exposed and close to the mikes. “Helpless” (from “Hamilton”). Broadway pit band, pop sound, female harmonies.

In these kinds of tests, volume matching is incredibly important, for a couple of reasons. As Tom’s Hardware puts it: “First, if sources are at different levels, they’re easy to tell apart. From there, the test is no longer blind. Second, us humans tend to prefer (all other factors being equal) louder sources.”

For my dress rehearsal the night before, I volume matched them as best I could by ear.

The panelists at the dress rehearsal were my wife Nicki and my friend Mike, a professional guitarist who spent years as an audio technician for big-name touring bands.

I gave each panelist a score sheet, with room for notes, and asked them to rank the four speakers, 1 through 4, after each listening test. I sat at the laptop to control the tests; I played the same section of each song for about 20 seconds on each speaker. Panelists were free to ask for re-plays, or to hear any speaker again, or to hear two speakers in a different succession.

At the end of the rehearsal, I asked the listeners to choose a winner, based on how many first-place finishes they’d marked down. Both Nicki and Mike declared the HomePod to have the best sound, hands down.

The test

The next day, the Yahoo film crew arrived. Our sound recordist, Dave, used level meters to help me volume-match the speakers more precisely.

My five panelists included Darwin, a professional violinist who spends a lot of time listening to recordings on nice gear; Julie, an entrepreneur and homeowner who is precisely the target market for these speakers; Dana and Tori, high-schoolers who haven’t yet begun to lose their ability to hear high frequencies; and Rob, a sound technician for Yahoo.

Story continues