Rock’s flight into seriousness in the 1970s had many ill effects. There was prog rock, jamming, not releasing singles – and the idea that the couple of decades since Elvis had produced enough music of sufficient worth to produce a canon. In 1974, like a university English department sending out a reading list to undergraduates, NME polled its writers and published its list of the top 100 albums of all time. The Beatles’ Sgt Pepper’s Lonely Hearts Club Band was No 1, Bob Dylan’s Blonde on Blonde was No 2, the Beach Boys’ Pet Sounds was No 3 – you could imagine just such a top three being published today.

A few period pieces aside – it’s a long time since Spirit, Frank Zappa, Johnny Winter, Joe Cocker or Country Joe and the Fish featured in a generalist greatest albums list – it set a template for the pop canon that has remained largely untouched for more than 40 years, by adhering to certain rules.

1. The canon is based around rock music

Pop has a place, but only the pop music of youth, which for NME’s 1970s writers meant Motown, or early rock’n’roll. You can see the rock-centrism in the fact that the only pre-1955 music allowed is blues, rock’n’roll’s wellspring. There’s no place, for example, for Frank Sinatra. Later polls would realise this was a bit limiting, and expand the remit to include jazz (or, in practical terms, Miles Davis’s Kind of Blue and John Coltrane’s A Love Supreme; nothing by Billie Holiday or Ella Fitzgerald, though), and, later, hip-hop. And being based around rock music means the canon is overwhelmingly white, despite black musicians having driven mainstream trends since the Gershwins, Cole Porter, Irving Berlin et al started to pick up and incorporate jazz in their writing.

2. The canon is male

Five albums on the 1974 list were fronted by women, and that includes the Velvet Underground and Nico, which is a bit of a stretch. When NME listed the 500 best albums of all time in 2013, the number of female-fronted albums in its top 100 had increased to a mighty eight. Again including that first Velvets album.

3. The canon is measured by albums

In the critical hive mind of the past, the album is the novel of music, the only serious way to assess worth. One hugely progressive feature of the NME poll, and its 1985 successor, was to allow greatest hits albums, on the grounds that some acts had strings of great singles but no great albums (though why they thought Aretha Franklin was one such, I have no idea). But generally, 60s artists who made albums of three astounding hits and 11 pieces of filler have been marginalised, as well as the whole 40 years of dance music since disco, where the primacy of the single track for the dancefloor is paramount.

Neutral Milk Hotel – better than Nas? Photograph: Merge Records

I’m not just talking about the NME polls of 1974, 1985 and 2013. The website Best Ever Albums has aggregated all the lists it can find – 34,000 of them, from best evers to best of years – to draw up a poll of polls, a canon of canons. The highest placed album that is not indisputably a rock record is at No 38: Kind of Blue. Kanye West’s My Beautiful Dark Twisted Fantasy is the highest-placed hip-hop record, at No 51. The highest-place album fronted by a woman is Portishead’s Dummy, at No 82. It takes until No 98 for a female solo artist to appear (Joni Mitchell, Blue). Most jawdroppingly, the highest placed album by a woman of colour is The Miseducation of Lauryn Hill, which appears at No 442. Which makes it a less significant record than, for example, Hospice by the Antlers (No 207). This is a canon in which the most artistically influential and commercially successful music of the last 30 years – hip-hop – is profoundly undervalued. For what it’s worth, I love Neutral Milk Hotel more than I love Nas, but I’d have to be a fool to claim their In the Aeroplane Over the Sea trumps Illmatic for “greatness”.

At this point, you’d have to be the most intransigent of rockists to think the canon – as defined by its previous iterations – has any meaning at all. It’s a canon drawn up overwhelmingly by white men, working for magazines or papers that considered themselves to be generalist titles but were, by virtue of their staffing and their readership, really rock titles that occasionally delved into other areas. And that’s no canon at all.

So now what?

Music has changed irrevocably. You can see that by looking at festival bills, where hip-hop and R&B artists sit alongside rock bands, the never-the-twain tribal divisions of the past dissolved. You can see it in the fact that online distribution of music has created a generation gap in music again, by creating an audience of teenagers who consume everything online, who listen to music that has bypassed the traditional gatekeepers of press, radio and labels. And you can see it in the fact that music journalists – the people who traditionally define the canon – are no longer overwhelmingly men. That more women and people of colour are now creating the narrative around pop is surely a good thing, even if it sometimes feels uncomfortable for those of us brought up on the certainties of Led Zeppelin and a narrative that begins with Robert Johnson and traces a line through the Beatles and the Smiths.

I suspect we may be at the end of the age of the canon, for now at least. For all those mixed bills at festivals, and playlists that place Beyoncé next to Black Midi, I suspect best-ever lists, from here on, will be specialised. They are more likely to go by genre, in specialist titles: the best hip-hop album ever, the best metal album ever, the best electronic album ever. It’s simply easier to do it that way than to attempt to evaluate the relative worth of Lemonade against Led Zeppelin IV. Maybe some magazines will have the occasional readers’ vote, which they’ll then have to fiddle to make sure Catfish and the Bottlemen’s fanbase don’t skew it and vote The Balcony higher than Blonde on Blonde.

Beyonce and Jay-Z – new additions to the canon? Photograph: PictureGroup/Rex/Shutterstock/PictureGroup/REX/Shutterstock

More to the point, which titles have the authority to attempt to define greatness nowadays? NME or Rolling Stone, the magazines that have previously seen themselves as the arbiters of the subtly different British and American canons? Not them, I fear, with NME having lost its print edition, and Rolling Stone in a decline that seems irreversible. Pitchfork has, doubtless deliberately, steered clear of anything overarching, instead selecting its greatest albums of different decades, lists that are admirably wide-ranging, though they have a baffling blindspot for reggae.

Maybe in five or 10 years’ time a new canon will emerge, once the generation that has dominated music media for the past 30 years or so finally all but disappears, because it’s the composition of the voters that determines the outcome. It will be a canon that retains some of the old elements: I don’t see Bob Dylan, the Beatles, the Stones or David Bowie going anywhere in a hurry, though perhaps fewer of their albums would fill the lists. I think some acts will suffer: I’m sure Morrissey’s continuing outbursts will colour people’s views of the Smiths (and I’m not certain Radiohead will retain their current lofty position: they are at No 1 and No 6 in the Best Ever Album aggregator). Some of the marginal things will disappear completely (Wilco, Sufjan Stevens, Bon Iver – that kind of thing; all those indie acts who made one beloved album).

I suspect Prince and Michael Jackson will see their statuses elevated. I think an awful lot more women – especially women of colour – will find themselves elevated: Tori Amos, Erykah Badu, Mary J Blige, Janet Jackson, and – the nemesis of Guardian comment threads – Beyoncé. Much more hip-hop will be included, and not just the hip-hop that crosses over and earns approval from rock fans. Kendrick Lamar, Jay-Z, Beastie Boys and Public Enemy will be joined by more Kanye, more Wu-Tang solo albums, more of-the-moment oddities (which would themselves disappear in due course). Chic will assume their rightful place in the pantheon. Various forms of African music will enter the lists, not just Fela Kuti; ditto the many offshoots of dance music.

It won’t be a list that reflects my tastes, and I’m fine with that. My canon is dead. Long live the new canon, whatever it might be.