The creator of Harlem Shake could end up returning 100% of his earnings from the viral video hit after neglecting to clear two of the song's most recognisable samples.

The No 1 song in America begins with a line in Spanish: "Con los terroristas," someone growls, as the crescendo builds. At the 15-second mark, another voice entreats listeners to "Do the Harlem shake". This is a key moment for the thousands of videos that turned this obscure dance track into a YouTube sensation – the signal that transforms each clip into a rowdy, air-humping bonanza.

Unfortunately, Harlem Shake's 23-year-old creator Baauer didn't record either of these vocal snippets. And he didn't pay to license them. Instead, the two artists he sampled are demanding remuneration for what has become a worldwide hit.

"We can turn around and stop that song," Javier Gómez, the former manager for reggaeton artist Hector El Father, told the New York Times. "[It's] a clear breaking of intellectual property rights." El Father is the voice that purrs "Con los terroristas" ("With the terrorists") at the opening of Harlem Shake. The line first appeared on 2006's Los Terroristas, and Gómez says it is one of El Father's "trademarks".

As for doing the Harlem Shake, that advice was taken from a 2001 rap song called Miller Time. "It just got stuck in my head for a while, so I used it," Baauer told the Daily Beast. A rapper called Hennessy Youngman, from the group Plastic Little, delivers the line. According to the New York Times, Youngman told Baauer he was happy someone was "doing something useful with our annoying music".

But happiness doesn't pay the rent. Both El Father – whose real name is Hector Delgado – and Youngman – who was born Jayson Musson – are negotiating licensing terms with Baauer's label. "Hector will get what he deserves," Gómez said. The price could be very high: Harlem Shake has sold more than 816,000 copies in the US and earned more than £50,000 from YouTube.

Baauer's worst-case scenario is what happened to the Verve in the late 90s. After using music from a Rolling Stones song, the Verve ended up relinquishing the complete publishing royalties for Bitter Sweet Symphony, their best-known single.