Nick Cave and the Bad Seeds’ 18th album was casually announced, a week before its release, in answer to an online query from a fan on Cave’s Red Hand Files website. That says a lot about how Nick Cave has transformed himself over the last 12 months. Previously an entertaining but guarded interviewee, he has, more or less, thrown himself open to the public. His website began with Cave posting the words “You can ask me anything” online. He’s subsequently answered dozens of fans’ questions, from the trivial to the metaphysical, eloquently and at length.

His most recent tour was effectively its live incarnation, based around an audience Q&A, conducted without a moderator. Anyone who has attended an artist Q&A where a particularly banzai fan has ended up in possession of the microphone knows what a bold high-wire act that is. The accidental death of his son Arthur in 2015, he said, has led him to “see people in a different way”, giving him “a deep feeling toward other people and an absolute understanding of their suffering”.

And yet, for all his new-found empathy and openness, some things about Nick Cave remain enigmatic. What, for example, are we supposed to make of Ghosteen’s cover, a kitschy, fairytale landscape illustration that looks not unlike one of those leaflets Jehovah’s Witnesses hand out at shopping centres? His description of the album is also pretty gnomic: the eight songs on the first album are “the children”; the two lengthy tracks and spoken word piece on album two are “their parents”.

Anyone seeking a more prosaic description of Ghosteen’s contents might note that, sonically, it continues and extends the weightless, drifting style of its two predecessors. It’s not entirely clear how Bad Seeds drummer Thomas Wydler passed his time in the studio, given that this time around, there are virtually no rhythms – with the more punishing of Warren Ellis’s tape loops and electronics replaced by warm analogue synthesiser that gives proceedings a faintly proggy feel. Indeed, Ghosteen occasionally feels like an infinitely warmer, sweeter sibling of 2016’s Skeleton Tree. While that album forced its most beautiful melodies to pick their way gingerly through minefields of explosive noise and eerie, discordant soundscapes - or to submit to Cave singing them in an alarmingly parched, numb voice – here they’re bolstered by soft, pillowy harmonies and stately piano. Cave’s voice, meanwhile, sounds rich. At an age when singers tend to start losing some of their range, his appears to be getting wider: it’s hard to imagine the twentysomething who snarled and howled his way through Saint Huck or Your Funeral My Trial daring to attain the high notes he achieves on The Spinning Song.

Artwork for Ghosteen by Nick Cave and the Bad Seeds. Photograph: Ghosteen Ltd

The result is perhaps the most straightforwardly beautiful set of songs that Nick Cave and the Bad Seeds have ever recorded, which fits with the album’s lyrical themes. When you listen to Ghosteen, the “children”/ “parents” description makes sense. Cave has talked about how his son’s death altered his work, how he found a way to “write beyond the trauma … to propel [myself] beyond the personal into a state of wonder”. And, despite the appearance of regular Cave themes – the first thing you hear is him invoking Vegas-era Elvis – Ghosteen appears to depict that process. For all the cinematic attractiveness of their arrangements and the title track’s opening meditation on the beauty of the world, the long songs on album two seem darker in tone, filled with sleepless nights, departures and desperate fantasies of escape: “I’m waiting for peace to come,” repeats one ambiguous line in Hollywood. The songs on album one, meanwhile, sound like the result of working through those emotions: lighter, calmer, concerned with empathy and faith. You feel the shift most clearly on Sun Forest, in which Cave does his patented apocalyptic thing – screaming horses, burning trees, hanging bodies, Jesus mad with grief – before the lyric suddenly flips – “the past pulls away and the future begins and I say goodbye to all that, / and the future rolls in like a wave… and the past, with its savage undertow, lets go” – and ends up exhorting the listener: “Come on, everyone!”

Most listeners seemed to take Skeleton Tree as an extended treatise on grief, ignoring Cave’s insistence that the songs predated the loss they were presumed to be about. In fairness, you could see why, given how bleak and disturbing it sounded. But if Ghosteen is the album people supposed its predecessor to be, it sounds like nothing like you might expect of an album informed by tragedy.

On one level, it shouldn’t be surprising that it’s as good as it is: Nick Cave and the Bad Seeds have been in a career-high purple patch since the last double album they released, 2004’s Abattoir Blues/The Lyre of Orpheus. Nevertheless, listening to Ghosteen, it’s very hard indeed not to be taken aback.

• This review previously misstated the year of the death of Cave’s son. This has been corrected.