For major studios, August is where the bodies are buried. The blockbusters they felt confident about releasing in May, June and July have mostly come and gone, save for a few still ravaging the landscape, picking off repeat viewers still lured by the air conditioning. The new movies tend to be mistakes or misfits, gambles that have either come up snake eyes or are shrouded in uncertainty, because they don’t fit into the usual boxes. If one of them turns out to be a late-summer sensation, there will have been no precedent for it, no tried-and-true formula for what constitutes a Hollywood hit.

M Night Shyamalan’s The Sixth Sense was – and remains – unclassifiable. It’s usually labeled a horror film, and there’s plenty of evidence to support that reading: a moody Philadelphia haunted by unsettled spirits, a few macabre shocks and a long history of cinematic ghost stories like The Innocents and The Haunting, about souls still trapped in a purgatorial no man’s land between the living and the dead. It could also be seen as an American answer to Wim Wenders’ Wings of Desire, with Bruce Willis taking the Bruno Ganz role of an angel observing – and occasionally interceding in – the sad, lonely lives of mortals below. Or, to quote Stephen Holden’s pan in the New York Times, it could be dismissed as “supernatural schmaltz”, with a touchy-feely agenda more in line with films like Ghost or Field of Dreams.

Whatever the case, The Sixth Sense struck a chord with audiences in August 1999, thanks in part to the most (not) talked-about twist to hit the culture since The Crying Game seven years earlier. It feels great to be able to have that twist in the open now – Bruce Willis was dead the whole time! And the kid who sees dead people never brought it up for some reason! – because it gets at why the film moved so many people after giving them a few mild shocks first. Through the story of a nine-year-old who can see and counsel the deceased, Shyamalan taps into the powerful fantasy that we can still control our own narrative, even after death. All the loose ends can be tidied up – justice granted, relationships mended, a final “goodnight” uttered – and everyone can rest in peace, whether they’re six feet under or sleeping next to an empty divot on the bed.

Haley Joel Osment as Cole Sear. Photograph: Ron Phillips/Hollywood/Kobal/Rex/Shutterstock

For Shyamalan, it was a last chance to define himself as Steven Spielberg’s heir apparent after stalling out twice with Praying for Anger, his forgotten 1992 directorial debut, and Wide Awake, a Denis Leary/Dana Delany/Rosie O’Donnell dramedy that bombed in 1998 after three years of reshoots, edits and general psychological torment under the Miramax chief Harvey Weinstein. He even had Spielberg’s producing partners, Frank Marshall and Kathleen Kennedy, along for the ride. He never lacked for confidence – in fact, it was his confidence that reportedly rankled Weinstein – but The Sixth Sense feels so much like the first M Night Shyamalan film, in style and tone and twistiness, it’s as if his previous work never happened at all. Like a true movie magician, he made his career struggles disappear.

The Shyamalan that emerges in The Sixth Sense may resemble Spielberg in his storyboard pristine imagery, but the real audacity of the film is his commitment to a somber tone, which goes far beyond the moody trappings associated with horror films. He’d carry it over to even greater effect with his 2001 follow-up, Unbreakable, but his overcast Philadelphia is less about creating a frightening atmosphere than imposing a deeper, more melancholy ambience over the action. The look allows Shyamalan to toggle back to chills on occasion, especially during a stretch where ghosts bombard its young hero with nightly visitations, but he wants to suggest the grief, confusion and misgivings associated with death.

The performances all fall in line, starting with Bruce Willis as Dr Malcolm Crowe, a celebrated child psychologist who’s nonetheless tormented by the case of a child he couldn’t help. That child has grown up to shoot Malcolm and himself – the former fatally, we later discover – but in his despair, Malcolm tries to redeem himself by seeking out another patient with similar symptoms of acute anxiety and mood disorder. That kid is Cole Sear, played by Haley Joel Osment as a preternaturally thoughtful and sensitive boy who lives alone with his mother (Toni Collette), gets bullied by his peers and carries a secret that he’s unwilling to share with Malcolm. (Or the audience.)

The Sixth Sense does a lot of cheating with information, barely bothering to establish the physical and psychological limitations of those who are dead and not loving it. (“They only see what they want to see,” Cole later explains to Malcolm, a line that patches over the plot holes like Silly Putty on drywall.) Nevertheless, it’s fun to return to the film after knowing the twist and seeing how Shyamalan makes us see only what he wants us to see, like wordless scenes between Malcolm and his wife (Olivia Williams) that are made to seem like marital discord or catching Malcolm in what looks like the end of a conversation with another character. Once doctor and patient figure out that it’s Cole’s purpose to help these ghosts, the film becomes awash in New Age bromides and neatly tied loose ends: he helps a girl (played by a pre-OC Mischa Barton) communicate to her father that her mother was poisoning her, he assures his own mother that his grandmother saw her big dance recital as a child and was very proud, and he advises Malcolm to say his piece to his wife while she sleeps. (“Good night, Malcolm,” she says, like a storybook closing.)

The virtues and faults of The Sixth Sense would carry over to the rest of Shyamalan’s career, suggesting that he had found his own formula for success and he wasn’t inclined to deviate from it. He has occasionally bowed to the trends of the moment – The Last Airbender was his disastrous run at a YA-style Chosen One fantasy, and The Visit jumped more successfully on the found-footage bandwagon – but he’s remained a self-styled auteur who keeps drawing from the same well. He’s the Twist Guy, forever chasing the same sleight-of-hand that dazzled audiences here, but has mostly made them groan since. But he’s also a sophisticated formalist, with a talent for slowing the metabolism of traditional genre films and creeping into the darker corners of the human psyche. Even when he’s bad, he’s singular.