You rarely hear them on the radio. Most fans can’t spell, let alone pronounce, their name. They don’t play gigs on Friday nights and their manager says they sometimes are mistaken for a death-metal group by fans who come across the cover art for some of their music collections. Instead of portraying the group’s five members, the covers feature five glowing bars and the mysterious letters: “PTX.”

Yet Pentatonix, a young a cappella group hailing mostly from Texas, may be one of the few acts to sell a million copies of a 2014 album this year, selling more than 615,000 copies of “That’s Christmas to Me” since last month, according to Nielsen SoundScan. Several weeks ago the album bumped Taylor Swift’s “1989” from the No. 1 spot at Apple Inc.’s iTunes Store, though Ms. Swift has since regained the top position.

The group’s unlikely rise to stardom—which caught most of the music industry by surprise—highlights not only the influence of YouTube but also the power of a distinctive sound. The latter is particularly valuable in a world where pop stars generate most of their income from hit singles and are wary of taking the musical risks they once could on full-length albums.

One reason why Pentatonix was able to preserve their unconventional sound is because until last year, no big record label wanted anything to do with them. Even after sweeping NBC’s “The Sing-Off” in 2011, the group was dropped by Sony’s Epic Records—which had an option to sign winners. So they started making their own records and simple YouTube videos, putting together quirky a cappella arrangements of popular hits without executives weighing in or pushing them toward a commercial sound.

The group hung on to the name they had coined before appearing on “The Sing-Off,” even though NBC had warned them they would regret it, said Pentatonix member Scott Hoying. The 23-year-old Mr. Hoying, who says he became an “a cappella nerd” as a freshman at the University of Southern California, persuaded two childhood friends to audition with him for the TV singing contest and then recruited two more singers. Until this year, the group distributed its music through Sony’s Madison Gate Records label, which usually handles film soundtracks.