I emerged from the fourth season of the BBC’s once awesome Sherlock in a kind of incoherent rage at what successful writers get away with when they are, apparently, deemed too big to fail. I’m not the only one, of course. There was a nice skewering of the show’s degeneration from cerebral mystery to James Bond-lite action film in the Guardian and the program’s principal show runner, Steven Moffat, has been drawing feminist flak since season two, so rather than go after elements of the show itself (and spoiling it for those who haven’t seen it in the process) I want to step back from Sherlock and focus on a troubling element I’ve seen in a lot of recent storytelling: the disastrous pursuit of surprise.

I’m talking about plot twists, and I’ll start by saying yes, I love them. There are few more compelling feelings than reading a book or watching a TV show and suddenly thinking “Wait! This isn’t what I thought it was at all! Everything I thought I knew about this story was wrong! The good guys are the bad guys (or vice versa). Up is down and black is white and I can’t wait to see how this works out!!!”

If it works out.

And there’s the rub. There’s nothing more satisfying than being taken off guard by a plot twist only to find that the story now actually makes more sense. Things I had half noticed but not processed suddenly become telling—they might even have been clues I might have grasped if I’d known how to read them, and as we move to the end of the story everything seems clearer, sharper and more intense because it has morphed unexpectedly but coherently into something I hadn’t seen coming.

And then there’s Sherlock. Or Doctor Who. Or any number of other non-Moffat books and TV shows where the delight in twists seems an end in itself. “They won’t see this coming!” you can sense the writers crowing gleefully as they draft in assassin wives and maximum security prisons (which somehow aren’t) and characters returning from the dead, all justified by a scattering of faux science, a little psychosis, and (most importantly) some swift transitions which go by so fast that you aren’t supposed to have time to sit up and say “excuse me?” Lately it seems that I find myself looking up in the final pages or minutes of a show with David Byrne singing in my head “Well, how did I get here?”

How indeed?

It’s not new, of course, this flagrant use of smoke-and-mirrors plotting and nonsense resolutions. Think of that great study in audience abuse, Lost, which began with a plane crash and then added twist upon twist, surprise after surprise, always dangling the possibility of everything coming together and making sense in next week’s episode. It never did. The script heaped up implausibilities and non sequiturs until nothing could finally account for what the show had actually been about. Lost was an object lesson in the financial reality of television whose job is to keep viewers hooked for as long as possible, and then, when they (and the advertisers) have lost interest, vanish, whether the story is wrapped up or not.

So we get extended and increasingly incoherent narrative arcs that leave fans scratching their heads (Battlestar Galactica, anyone?) because we are doing what readers are hardwired to do. We try to find coherence, unity, and meaning whether there is any or not. We assume that the ending was somehow planned from the beginning, though we should know by now that that is not how television is made. TV—unless it’s conceived as a self-contained mini-season—doesn’t begin with a macro idea which they then break into as many episodes or seasons as they have to fill. Generally, they start small and add to the end, extending and extending with no final end game in sight. We shouldn’t be surprised that it doesn’t finally make sense. All those plot twists and surprises we thought were complex revelations of some master plan were just new bits tacked on, each one taking the story in a direction no one (including the writers) had foreseen when they penned Episode 1.

The grand example of all this misdirection might be the original Twin Peaks, a surreal masterpiece masquerading as a detective story. It was lush and strange and unlike anything I had seen on television before but it seemed to work like a conventional murder mystery and the burning question—Who killed Laura Palmer?—seemed, for a while, to be on everyone’s lips. And then we got into Season 2 and gradually we lost faith in the idea that that question would ever be answered in a way that was satisfactory, that all the twists and revelations were a kind of postmodern collage and not an unconventionally told but ultimately linear narrative with an answer at the end. Still, the journey was almost worth it.

And let’s be honest, it’s hard to write plots which surprise and redirect but still deliver the solution or resolution which the genre demands in a way which feels both plausible and satisfying. As unconventional TV mysteries go, Dennis Potter’s The Singing Detective is a good example of one that did, all its meta constructs finally falling away in a Freudian reveal about the writer at the heart of the story. But it’s rare to pull off such a feat, and writers don’t get enough credit for it. They are praised for character, for sentence-level phrasing, but not so much for building the kind of intricately clever plotting where all those twists and reveals lock together like the wheels of a great German clock.

I’m not sure if the problem is the much touted demise of critical thinking, of big picture analysis, or of our shortened attention spans, but too much narrative art seems to think that it doesn’t matter if the whole doesn’t make sense so long as the moment-to-moment stuff keeps us on the edge of our seats. It does matter, if only because if we realize that the solutions and revelations don’t really stand up to scrutiny, then what’s the point of watching at all? The twist in a tale can be potent when it’s earned and part of a larger narrative design, but when it’s just a flash bomb, a distraction from the lack of substance in the story, it derails the whole plot, setting everything off down some new track like a hastily thrown point on a railway line. As a model railway enthusiast I know all too well that twists in the track, turn outs, and sudden shifts in direction might make for an interesting-looking layout—but unless they are done extremely well, they tend to leave you with a derailed locomotive and a string of cars smashing on the floor.

A.J. Hartley is the bestselling author of a dozen novels including Sekret Machines: Chasing Shadows and the upcoming Cathedrals of Glass: A Planet of Blood and Ice (To The Stars Media) and the YA fantasy adventures Steeplejack and Firebrand from Tor Teen. As Andrew James Hartley, he is also UNC Charlotte’s Robinson Distinguished Professor of Shakespeare, specializing in performance theory and practice, and is the author of various scholarly books and articles from the world’s best academic publishers including Palgrave and Cambridge University Press. He is an honorary fellow of the University of Central Lancashire, UK.