There are bad movies that make you laugh, bad movies that make you feel like you wasted a few hours, and bad movies that leave you baffled and weirdly fascinated. Then there are bad movies that actually hurt. They hurt because they feel like opportunities squandered, where you can see an interesting film squirming to climb out of the muck. Live By Night, Ben Affleck’s adaptation of Dennis Lehane’s novel, is one of those movies. The seeds are there for something great — several of them, in fact — but they’re smothered by a film that seems overwhelmed by itself.

We start off with Joe Coughlin (Affleck) in Boston during the 1920s. He’s a small-time criminal who is son to a police captain (Brendan Gleeson) and is, thanks to his experiences in the War to End All Wars, averse to authority and death. Those aversions extend to mobsters, but apparently not to their girlfriends. He spends his off-hours snuggling with Emma (Sienna Miller), the mistress of the notorious Irish mobster, Albert White. That leads to complications.

This stretch in Boston is the weakest in a weak film. I have not read the novel, but this part reeks of overly faithful adaptation. It’s rushed, with stabs at character development strung together through montage. It’s so montage-heavy that it plays at times like an extended “previously on” opener for a TV show, complicated by a pointless Alias-style flash forward at the very beginning. The film just can’t breathe.

Once things inevitably go wrong for Joe and he leaves Boston, the movie actually starts to feel like a real movie. A deeply flawed mess of a movie, but a movie nonetheless. He signs on with the Italian mob to help with their bootlegging business in Ybor City in Florida, his motivation being taking on White, the mobster who screwed him over. His reasoning is a bit hazy, but the move to Ybor City is worth it. Ybor is vibrant and interesting in a way that the film’s version of Boston, perhaps due to overfamiliarity, can’t compete with.

More than that, the film starts to enter into thematic territory that the opening only skims. Ybor is a fascinating tangle of competing nationalities and class conflict, barely kept in check by a hands-off but incorruptible Sheriff (Chris Cooper). As Joe is shown around town by a former partner in crime, learning all the dynamics at play, you start to see the possibilities.

But Affleck doesn’t seem to know what to do with these possibilities. The closest the film comes is when Joe and the Italian mob come into conflict with the Tampa Ku Klux Klan. This stretch of the film, with the comparatively rich and powerful Klan getting pissy and murderous over a bunch of Catholic mobsters serving booze to brown-skinned immigrants, could have been a movie on its own. A crime film about mobsters versus the Klan in Prohibition-era Florida? Are you kidding me? I would kill for that. Add in Joe’s romantic relationship with a powerful local Cuban woman (Zoe Saldana) and you have the potential for a thematically rich little genre flick.

Unfortunately, that’s just an episode — one of several — in a much longer and poorly paced film, and nothing ever comes together in thematically or dramatically satisfying ways. Other than a painfully dull lead performance from Affleck, the major problem is that Live By Night is blinded by its own goals. It clearly wants to be a sprawling and complex crime drama and character portrait, but it doesn’t have the attention span, coherence or narrative propulsion to make it work. This extends to the cinematography, with is often punctuated with arbitrary stylistic choices. Just as any scene in the film seems fine on its own, any single shot is quite lovely. But the whole construction? I suspect limitations and a tighter focus could have done it some good.

As Affleck steers the film to a close you can sense him struggling to bring it all together, like he’s realized he’s screwed up. The last few sequences are a cluster of tonal shifts and unearned emotional payoffs, culminating in a moment that surprised me in just how off it feels. It’s so off that I started thinking back, reevaluating how I had been interpreting the film, but I’ll be damned if I can make it all add up.

But again, there’s some great stuff in there. Let’s slice this thing open and save my mob versus the Klan film.