Music recommendation is not doing its job, says Paul Lamere, director of developer community at 'machine listening' specialists The Echo Nest. However long the long tail of music, mechanical recommendation is biased towards what the majority of listeners do because brand new bands have very little audience and little related data.

Musician unloading in Austin, Texas, for SXSW

It means that if you listen to the 'short head' of music - the mainstream, most listened to artists - you are likely to be referred to similar artists that are also in the short head, like Britney Spears to Christina Aguilera. He said 48% of short-head recommendations push back to the short head, and that in 2007 music retail data on 4m sold tracks showed 13% were either American Idol or Disney artists.

"We still haven't seen the promise of the long tail. It's not that [The Long Tail author] Chris Anderson was wrong but the music recommenders aren't doing their job - they are pushing people towards the short tail."

Recommendation can get stuck in corners

Hype Machine creator Anthony Volodkin said the problems are the same for film and book recommendations, but that another problem is that if recommendations aren't accurate enough, users tend to leave. That explained 'dark corners' of recommendation sites where a critical mass of people create a kind of recommendation cul-de-sac. And someone people love to hate, like Napoleon Dynamite, aren't properly represented if they have a majority of five and one-star ratings.

Recommendation would also benefit by bring in extra details to explain why something is being recommended. Why would recommend Johnny Cash to someone looking at Nine Inch Nails? Because Johnny Cash covered a Nine Inch Nails song. "It could get you into a whole new genre you never would have discovered otherwise - and great albums like Fulsom Prison. But it's hard for collaborative filtering to do because all they know is what poeple bought."

Recommendation can also be gamed. When religious broadcaster started blaming the gay community for some of society's problems, it prompted a campaign on Amazon where people visited the page for his book, Six Steps to Spiritual Revival, and then went to the page for the Ultimate Guide to Anal Sex for Men. The site noticed the traffic and flagged up the sex guide as a related title of interest...

More sinister gaming is already happening though. Lamere said he's already seen profesionals at music companies trying to game recommendation systems to push sales for certain artists.

How could music recommendation be improved?

Lamere said semantic recommendation is one route. A tag cloud of words related to one artist can be compared to another, with surprisingly similar results. Another route is comparing complex visualisations of a track.

Official SXSW vehicle

He also showed a 3D map of his music collection, developed when he worked in research at the Sun Microsystems Lab, which shows each track as a dot, and its location in the 'galaxy' is determined by an analysis of its sound, so classical is clustered together, rock, country and so on. It's not perfect - it thinks distorted guitar is the same as harpsichord, for example - but it is far more accurate than collaborative recommendation.

As well as social context, sites need to do a better job at presenting new music so that long tail is really discoverable. They need to "create meaning", said Volodkin, and that's something traditional, human editors have always been better at. Editors of something like Pitchfork might be quite divisive but their opinion is meaningful. Their strong editorial voice still resonates with readers, despite the number of music startups that have entered the space since Pitchfork launched in the nineties.

Maybe technology isn't the right answer, said Volodkin. There's still no better recommendation than that of a friend, but the best services will be those that combine social recommendation. It's not enough that other people have listened to, recommended or bought one artist - who are those people, and how many of them were there?

"The really meaningful sites will represent all the activity going on out there - the news, blogs, Twitter posts, what people are sharing," said Volodkin. "It will all come together around people."