Television is a locomotive barreling down unfinished tracks. A novice showrunner may fancy himself the conductor; after all, it feels good to sit up high, push a few buttons, rewrite a few scripts, and imagine all the wonderful places you’re headed. But after a few weeks of production, the most organized writer will find himself back down on the ground with his sleeves rolled up, hammering in planks and fastening ties as quickly as possible and hoping for the best. The journeys of even the most successful TV shows are never smooth, and their eventual destinations are rarely the ones they initially set out to reach.

The path of Boardwalk Empire, which returns to HBO for its fourth season this Sunday at 9 p.m., has been particularly fraught and meandering. Green-lit with much fanfare in 2009, the series was intended to replace The Sopranos as the shiniest diamond in HBO’s prestige lineup. But the glittery talent assembled behind the camera — executive producers Martin Scorsese and Mark Wahlberg; showrunner, and Sopranos vet, Terence Winter — wasn’t enough to free its frustrating first season from what amounted to a gorgeously appointed, totally claustrophobic jewel box. Boardwalk‘s sets were exquisite, its production design beyond compare, and its performances, particularly those by supporting players Michael Shannon and Stephen Graham, superb. Yet the particulars of Nucky Thompson’s Prohibition-fueled metamorphosis from dirty pol to king of the bootleggers were at once too polite and too sober to gin up much enthusiasm. Savvily constructed out of the same quality threads that had previously been woven into Golden Age of TV glory — morally ambiguous antihero, lascivious period setting, Michael K. Williams — Boardwalk‘s assemblage of expensive talent and extravagant detail ultimately felt less like a story that needed to be told and more like one that only HBO could afford to tell.

Still, in the early going one had the sense that Winter knew where we were headed, even if his preferred pace had more in common with a New Jersey local than an Acela Express. The second season moved the show’s central conflict to the fore: a collision between Nucky’s old-fashioned, white-glove glad-handing and the bare-knuckles striving of a new generation of crooks, personified by Michael Pitt’s Jimmy Darmody. Darmody, a sort-of surrogate son to the childless Nucky, was schooled in war, not politics, and his approach to crime traded the gentlemanly backroom bribe for the much more direct punch in the face. At times in Boardwalk‘s feverish second season, this existential turf war was puffed up to delirious, Grand Guignol heights: Richard Harrow, the haunted, half-faced sniper, wandering the woods like a wounded animal; Jimmy, zooted on heroin, sleeping with his mother and stabbing his father in the heart. In moments like these it was possible to feel Boardwalk sloughing off the weight and expectation of its heavy pedigree and giving in to the giggle juice its characters fought so hard to control. Boardwalk Empire was a far better show when it was drunk.

But then it jumped the rails. We may never know exactly the reasons why Pitt was hastily written off at the end of the second season. Was it truly because, as Winter has argued, Nucky had to kill his protégé in order to fully break bad? Or was it traceable to the whispers of unprofessionalism and untoward behavior on set that had dogged Pitt for months? The latter still seems more likely to me, if only because the showrunner code is more inviolable than that of the physician: Do no harm. Most series couldn’t survive the loss of a co-lead, let alone one who seemed so central to the long-term plot. So while it’s impressive that Jimmy’s death didn’t kill Boardwalk Empire, it certainly robbed the show of its purpose and murdered its momentum.

So what happens when a show’s engine breaks down? To his credit, Winter immediately adopted a strategy familiar to anyone who’s ever sat on a tarmac or been stuck between stations: distraction. Boardwalk Empire‘s supporting cast was already bursting with celebrated character actors, wily NYC theater vets, and assorted dependable hams. In its first post-Pitt year, Winter larded his stockroom even further, adding Stephen Root, James Cromwell, and, in a charismatic performance as a bug-eyed cartoon, the great Bobby Cannavale. The resulting season had its moments — many of them involving the aforementioned Michael Kenneth Williams’s smoldering Chalky White, a self-made gangster with a chip the size of Baltimore resting on his immaculately tailored shoulder — but was, as a whole, strikingly aimless, continually either falling back into blood-streaked Scorsese homages or falling asleep whenever the camera focused on its lead character.

About that: Steve Buscemi is a tremendous actor and his skinny shoulders are certainly capable of supporting a big-budget TV drama. But in acting terms, supporting is a far different thing from carrying. Putting a bullet in Jimmy helped solidify Nucky’s organization onscreen, but off of it, the act laid bare the structural problems inherent in the character. As played by the wry Buscemi, Nucky is an unpalatable dash of bitters in what ought to be an effervescent cocktail. He has no foil, he doesn’t mix. His transformation into a full-fledged crime lord robbed Boardwalk of any nuance and pushed an already creaky protagonist further into cliché. Yes, Buscemi can play cold-blooded and serious. But why, oh why would you want him to? Worse, the arrival of Dark Nucky — a listing tuxedo of chain-smoking, pistol-shooting, and showgirl-shtupping — left all his associated characters marooned at sea: the solid Shea Whigham swallowing his pride and his potential as the disrespected Eli, the sparkling Kelly Macdonald reduced to a pallid scold as the perpetually betrayed Margaret.

Never great and now no longer interesting, Boardwalk Empire settled into a lavish middle age doomed to be merely fine. The original reason for the party had been forgotten, but the table was still so sumptuously set. And look at the decorations!

I wish I could say that, in its fourth season, Boardwalk Empire has found a compelling reason for its continued existence — other than keeping the tri-state area’s antique cloche dealers flush. But after viewing the first five episodes, that’s simply not the case. Instead of getting better, Winter seems married to the idea of getting bigger. Like a Gilded Age robber baron, he just can’t help himself; to an already groaning cast list he’s added Ron Livingston (as a weaselly Piggly Wiggly man), The Hurt Locker‘s Brian Geraghty (as a duplicitous probie), and The Killing‘s Eric Ladin (as J. Edgar Hoover). Patricia Arquette washes ashore — and makes a strong impression — as the blowsy proprietress of a speakeasy. And, in Week 2, the criminally underemployed Jeffrey Wright, who’s been committing larceny on inferior projects for close to two decades now, sidles up and pockets the entire show as Dr. Valentin Narcisse, a Harlem power broker who mixes black liberation theory with heroin sales and grift. Wright’s icy precision quiets Boardwalk‘s busyness; he elevates every scene he’s in, even when he’s forced to dirty his nails in the same muck as everyone else.

The season turns on the collision between Chalky and Dr. Narcisse, two men brought together by a particularly ugly scene in the season premiere that pours sex, violence, and racism into a broken whiskey bottle and then stabs someone in the neck with it. This increased focus on the African American side of 1920s American life is one of the lone bright spots in what has become a wildly unfocused show. Atlantic City was initially chosen as Boardwalk‘s setting because of its role as a key hub in the illegal liquor trade. But “key hubs” aren’t always so interesting. Brussels is a key hub. So is the Denver airport. As Nucky’s power and influence waxed and our interest waned, the show has cast a wider and wider net in search of a story, from Al Capone and Nelson Van Alden busting skulls in Illinois to Lucky Luciano and Meyer Lansky cracking wise in New York. This year the canvas stretches further to include Harrow’s stark Wisconsin home and even sunny Florida, where Arquette is pouring drinks and Nucky is looking to expand his business.

Expansion is a good idea when times are flush, but it’s a strategy often deployed in a downturn as well — all the better to mask problems rotting at the core. And ever since Jimmy’s demise, Boardwalk has never stopped its frantic Charleston, adding fresh blood while spilling buckets of the old. It’s kind of fascinating to see a show throw up its hands and marginalize its own protagonist, but, in doing so, it has also tossed any sense of stability and balance off the edge of a short pier. Surviving characters have now swapped jobs and precedence so many times they’re almost unrecognizable — Eli, once a cop, is now a thug; Chalky runs a nightclub; Cousin Oliver is a rumrunner. This year the camera swings around seemingly powered by fan service, elevating Anatol Yusef’s sly Meyer Lansky and Anthony Laciura’s noble Eddie into far more prominent positions. The fresh-faced Ben Rosenfield gets plenty of burn as Eli’s eldest son, struggling with girls, bullies, and his family legacy while attending Temple University — though his plot often seems like a mulligan on Jimmy Darmody’s long fall from the Ivory Tower. As for old favorites Capone, Van Alden, and Harrow — each of whom could support a series of their own — they’re all still accounted for, but the crowding does them no favors. (Shannon in particular seems to have left his interest in the part somewhere on Krypton.) With an average running time of 57 minutes, an episode of Boardwalk Empire feels at once overstuffed and underbaked, its attention flitting around like a wasp but never staying in one place long enough to sting.

In a world filled with terrible TV, being just OK is hardly a sin. And indeed, when watched in bulk, there’s something reliably entertaining about Boardwalk Empire. I’m not sure it’s satisfying enough to qualify as a binge watch, but it certainly goes down easier when chugged. Still, there’s a jarring dissonance between the show Boardwalk Empire‘s creators think they’re making and the show they’ve actually made. With its astronomical budgets and craft-conscious actors and directors, the series continually strains for artistic importance, but it’s impossible to ignore the pulp floating to the surface.

Four years in, Boardwalk Empire has become a decidedly lowbrow show dressed up in silken finery. Though one can never be sure which supporting player will be featured in a given episode, you can set your pocket watch on the presence of thick wads of money being flashed, some lovingly curated jazz being tootled, and someone’s head being explicitly ventilated by period-appropriate bullets. Boardwalk Empire has never introduced a throat it couldn’t fetishistically slash or a gambling metaphor it couldn’t double down on. Its treatment of women, never stellar, now seems less a criticism of period sexism and more a celebration of it; the show appears to have given up all pretense of interest in female characters who wear shirts. Violence on Boardwalk Empire has grown increasingly horrific, used less as a caution and more as a cure: for scenes that aren’t going anywhere, for a show that frequently swaps stakes for knives. Dr. Narcisse belittles Chalky for not being worthy of his station, but he might want to turn his judgment on the series itself.

Because the thing is, Chalky White at least knows who he is — he doesn’t need a fancy, season-long guest star telling him so. And more than that, he’s comfortable with it. When Boardwalk Empire wallows in the seamier side of life, it does so as a tourist, no better than a thrill-seeking weekend warrior visiting Atlantic City and never venturing off the casino floor. It’s admirable the way Terence Winter and his writers, despite all sorts of unplanned obstacles, have never run short of track. Their Pullman sleeper is still beautifully appointed; the bar car never runs dry. But just because the train is moving doesn’t mean it’s headed anywhere in particular. And it doesn’t mean it’s worth going along for the ride.