Among the many remarkable things about Lorde’s 2013 debut album Pure Heroine were the lyrics of a song called Tennis Court. Written when Ella Yellich-O’Connor was 15 years old – and already, it would appear, the smartest and most self-aware writer in pop – it offered the same kind of pinpoint-sharp observations of her teenage peers’ lives as the rest of Pure Heroine (“it’s a new art form showing how little we care”), but one verse also cast a wary eye to her own future. If her musical career was successful, she noted, it would automatically remove her from her suburban environment and social group, the very things that had inspired her songs to date (“weird social situations and cliques, girls vying for attention, the archetypes of being a teen,” as she told the Guardian not long after the album’s release). “And what then?” the lyrics of Tennis Court wondered: “how can I fuck with the fun again when I’m known?”

Four years later – with more than5m album sales, an endorsement from David Bowie who called her “the future of music” and a level of celebrity that means a rumour Lorde started a secret Instagram account reviewing onion rings became international news – Melodrama answers the question. You can see how much has changed just by reading the credits. A recording studio in New Zealand swapped for one in New York, her solitary collaborator on Pure Heroine Joel Little supplanted by a battalion of heavy-hitting names from the backroom of pop: sometime Taylor Swift collaborator and Fun guitarist Jack Antonoff; Kuk Harrell of Rihanna, Beyonce and Justin Bieber fame; Frank Ocean and Zayn Malik producer and songwriter Malay; S1 (Symbolyc One), who’s worked with Beyonce and Madonna.

It’s the kind of supporting cast that could give a fan of Pure Heroine pause. One of the things that was striking about Lorde’s debut was that it appeared to come out of nowhere. Its incisive critiques of pop’s champagne-splashing VIP-area excesses – “I’m kind of over being told to put my hands in the air” – sharpened by the fact that it was a pop album made outside the machine of mainstream pop.

Melodrama, by contrast, comes from deep within it. At its weakest moments, you’re struck by the sense of Lorde struggling to assert her individuality amid some pretty generic music. You could almost imagine any of her mainstream pop peers singing a couple of the songs, although ‘almost’ is the operative word here. Even with the album’s weakest song, Homemade Dynamite – not bad exactly, but nothing melodically or sonically to set it apart from the rest of the Radio 1 playlist – she’s wont to remind you that you’re in the presence of a superior class of lyricist. It’s not just that she unexpectedly throws a There Is a Light That Never Goes Out-ish car crash into a song about copping off with someone at a party, it’s the way she does it. “Might get your friend to drive, but he can hardly see – we’ll end up painted on the road in red and chrome, all the broken glass sparkling,” she sings, adding dolefully: “I guess we’re partying.”

Melodrama album cover, by Brooklyn-based artist Sam McKinniss. Photograph: Lorde/Republic Records

The moments that do feel a little commonplace on Melodrama are vastly outweighed by songs where her uniqueness shines through. There’s ample evidence of an ability to twist pop into something entirely her own: the addictive, weirdly muted take on Giorgio Moroder-esque electronic disco found on Supercut; the off-kilter Hard Feelings, replete with an instrumental passage scarred by howling feedback. The ballads on pop albums are seldom the highlight, but they are here. Liability is based on a fantastic descending chord sequence not unlike that of Bowie’s All the Young Dudes. Meanwhile, Writer in the Dark certainly isn’t the first song on which a latterday female singer-songwriter attempts to channel Kate Bush, but it may well be the first one that doesn’t make you want to die of embarrassment on their behalf – the tune is wonderful, her cracked, raw vocal really affecting.

She scrupulously avoids the trap that other writers in her position frequently fall into – replacing their initial source of inspiration with songs that tell you that fame isn’t all it’s cracked up to be and that touring is boring. Instead she proves as adept at chronicling the messy entanglements you invariably encounter in your late teens and early 20s as she was at depicting suburban ennui. The album is shot through with ruminations on serious relationships that turn out to have been less serious than was thought, and the creeping disquiet that accompanies what Perfect Places calls “graceless” druggy hedonism: “the terror and the horror when we wonder why we bother,” as she puts it on Sober II (Melodrama). Her ear for a fabulous image remains intact – “well, summer slipped us underneath her tongue” – as does her sly self awareness and wit. It seems highly unlikely that any pop song this year will have a better chorus than that found in The Louvre: “We are the greatest, they’ll hang us in The Louvre – down the back, but who cares? It’s still The Louvre.”

At moments like that, Melodrama sounds less like a troubled attempt to follow up a huge debut album than a cocky challenge being issued to her musical contemporaries. For all its odd misfires, it makes a great deal of the stuff that sits alongside it in the charts look pretty feeble by comparison. If that sounds like faint praise, it isn’t meant to be: if it was easy to make hugely successful mainstream pop music as smart as this, then everybody would be at it. And they patently aren’t.