About halfway through the very funny, very disturbing, very ethnic new film A Serious Man, the modern-day Job who is the serious man in ­ question climbs up on to the roof of his ghastly 1960s Minneapolis suburban home and tries to adjust the antenna to improve his TV reception. Beleaguered on all fronts – conjugally, professionally, medically – Larry Gopnik, a dorky physics professor who may be about to lose his job and is very likely to lose his family, is a bright, principled Jewish man whose children have begged him to fix the antenna so they can watch F Troop, an idiotic 1960s comedy. Many of Larry's travails unfold as songs from Jefferson Airplane's seminal 1967 LP Surrealistic Pillow play in the background. With his khakis, his short-sleeved shirts, his nerdy glasses, and his stereotypically inconsequential name, Gopnik is clearly not a Surrealistic Pillow kind of guy.

Once on the roof, Gopnik spies a sultry, mysterious, not terribly classy next-door neighbour sunbathing nude in the privacy of her fenced-in backyard. She is a stoner and looks nothing like his wife, who is about to divorce him. The architecture, the locale, the attire, and the way the scene is shot capture majestically the mood of spiritual desolation and endemic weirdness that defined suburban US life in the 60s. It evokes the opening scenes from The Straight Story, the 1999 film about a 73-year-old man who drives a lawn mower 300 miles across Iowa to see his dying brother, with whom he fell out decades earlier. As I took in the amazing rooftop sequence in this troubling but rewarding movie, I forgot who had really made it and thought: "Gee, I never knew David Lynch was Jewish."

Lynch, the half-Finnish, all Presbyterian maker of The Straight Story, is not Jewish; the Coen brothers, who made A Serious Man, most emphatically are. Yet despite all the trademark humour, quirkiness and sarcasm that characterises their film, A Serious Man does not feel like a typical Coen brothers movie. It lacks the tongue-in-cheek goofiness of Fargo, Raising Arizona and The Big Lebowski, the stately depression-era charm of Miller's Crossing, the overwrought noir of Blood Simple, the unrelenting darkness of No Country for Old Men.

If it is possible to imagine a Woody Allen script with all the schtick exfoliated, and then filmed by Lynch, that master of conveying the under-the-skin bizarreness of small-town America, you have A Serious Man. Although perplexing and unnerving, with a finale that will not satisfy all tastes, the Coen brothers' latest film is the most daring project they have ever undertaken. It is mordant. It is philosophical. It addresses all the big questions. It is frequently hilarious. And it feels like somewhere along the line David Lynch took over.

A Serious Man falls into that category of films that, for whatever reason, do not have the same texture or mood as a director's other films. It may be a decision the film-maker has made deliberately, or it may be entirely inadvertent, but these films stand apart from the other movies in a director's body of work. It is as if the film-maker abruptly decided to take a holiday from his own personality and make a film in somebody else's style.

These films are not always successful, and not always memorable, but there is definitely something appealing about the genre itself: included among the successes are Martin Scorsese's The Age of Innocence, a high-society, drawing-room melodrama with nary a gangster in sight; Peter Weir's Green Card, a lighthearted comedy starring Gerard Depardieu that Weir filmed between the lugubrious Dead Poet's Society and the even more depressing Fearless; The Bridges of Madison County, one of the very few Clint Eastwood movies in which nobody dies, and the only truly romantic film he has ever made. By stripping away Robert James Waller's appalling prose and chopping the story down to its essentials, Eastwood made a touching motion picture that has absolutely nothing in common with Unforgiven, Pale Rider, The Outlaw Josey Wales, White Hunter Black Heart or even Bird. Immediately after finishing The Bridges of Madison County, Eastwood directed and starred in Absolute Power, a thriller about a murder involving the US president. Eastwood's sabbatical from his own career was over.

Not every film in this genre is as successful as The Age of Innocence and The Bridges of Madison County. To this day, no one knows what Ang Lee was thinking of when he made Hulk, one of those catastrophes so bad that its sequel seems like the industry's personal apology to the movie-going public for what has gone before. Windtalkers, though reliably sadistic, was a tremendous disappointment to John Woo fans, in part because it takes place on assorted beaches in the south Pacific during the second world war, and is therefore devoid of the ominous gangsterland charm that characterises Woo's Hong Kong films. For my money, Woody Allen's strangest project ever is Cassandra's Dream, his benighted attempt to make a suspense film. After the failure of this London-based drama, Allen directed Vicky Cristina Barcelona, the same film he has been making for the last 20 years. His fans adored it.

In certain cases, a film does not bear a director's stamp because he is renting himself out for a project he did not originate, or because he is at the tail end of his career. Inside Man is a passable caper film, with serviceable performances by Denzel Washington and Clive Owen – but, with its complete lack of social commentary, it certainly doesn't have the feel of any other Spike Lee film. It is work for hire. This is an apt, if unfortunate, description of The Next Best Thing, John Schlesinger's disastrous 2000 incursion into Madonna territory, and one of the worst films of the millennium. It is a sad commentary on our times that the man who started out making Billy Liar, Midnight Cowboy and Sunday Bloody Sunday should end up in Mondo Ciccone.

Another curiosity is Werner Herzog's Rescue Dawn, essentially The Great Escape remade during a different war, this time Vietnam. Entirely lacking in the all-encompassing strangeness of The Enigma of Kasper Hauser, or Aguirre, The Wrath of God, or Fitzcarraldo, or just about any other Herzog film including Little Dieter Needs to Fly (the 1997 Herzog documentary on which it is actually based), Rescue Dawn is a well-crafted action picture. And nothing more. Presumably, Herzog made the film to raise money for other, more interesting projects like Grizzly Man. And obviously, with Rescue Dawn, Herzog was at a disadvantage from the start: it is hard to make a distinctive, intellectually engaging motion picture when your leading man is Christian Bale.

Small, perfect jewels

When I was at university I had a professor who said we could learn as much from a writer's minor works as we could from their major ones. He also believed that, in some cases, an author's best book was the one that stood apart from the bulk of their oeuvre – because, when they stepped out of character, they were not seduced into doing the same old routine they had done a dozen times before. Examples included: James Baldwin's Giovanni's Room, which is not about the African-American experience in the US; The Colossus of Maroussi, a beautiful travel book Henry Miller wrote about Greece that does not even slightly extol fleshly pleasures in Paris; and Thomas Mann's Confessions of Felix Krull, Confidence Man, an oddity that is infinitely less portentous – and far more readable – than the novels that won Mann the Nobel prize. All of these books are small, perfect jewels that bear little or no resemblance to the works that made the authors famous.

Films like A Serious Man and The Age of Innocence, and even Quentin Tarantino's sublimely goofy Inglourious Basterds, all fit into this category. The nicest thing about these strange interludes is that they hold out the possibility that other directors might follow suit. I for one would love it if Michael Bay took a break from movies like Armageddon and Transformers and tried a musical instead, perhaps a long-overdue remake of My Fair Lady, starring Jean-Claude van Damme and Penélope Cruz in the roles originally made famous by Rex Harrison and Audrey Hepburn.

I would also be thrilled to see a Merchant-and-Ivory-type affair directed by Guy Ritchie, with the rustle of crinoline and the echoes of pianofortes everywhere; perhaps something along the lines of Henry James's The Wings of the Dove. Last but certainly not least, I think the public would react quite enthusiastically to a Ridley Scott movie starring Maggie Smith and Joan Plowright as zany spinsters working as tour guides in Vichy in 1942 before striking out alone. Tea with Monsieur Pétain, that sort of thing. Or better yet, a Michael Moore documentary extolling the virtues of free trade and paying tribute to Ronald Reagan and Margaret Thatcher.

That's one I'd pay to see. So would Margaret Thatcher.