Every great American film-maker struggles to create their own peculiar vision, just as the studio men struggle to stop them doing so. Yet few visions are quite so peculiar as Whit Stillman's, and few have seemed so marginal to the industry of which they are a part. It's hard to say how much impact his films have had; there have been, for reasons beyond his own control, too few of them. He has succeeded in getting four films made: a comic trilogy set in the 1980s, Metropolitan (1990), Barcelona (1994) and The Last Days of Disco (1998), and now the about-to-be released "campus comedy" Damsels in Distress. On one level it may seem a rather meagre body of work. However, for some, myself included, they are four of the very best American films made in the past 25 years.

The presence of many of the same actors in each of the films adds to the sense that we are entering a very particular world. Each film in the trilogy explores some central concept: if Metropolitan is about virtue, then Barcelona is about beauty, and The Last Days of Disco about social life as a kind of performance. It's harder to say what Damsels in Distress is about, it being so odd and so various. In it, some of the characters are taking a class on "Dandy Literature" – Peacock, Pater, Wilde, Firbank and Waugh. Like these writers, Stillman does one thing and does it supremely well. He makes Whit Stillman films. It's impossible to imagine him making some other kind of work – a historical romance on the House of Saxe-Coburg for example, or an episode of Homicide: Life on the Street.

For all their muted sense of doom, his films play out in a better world. Stillman's preference is for 30s cinema; for him the Hollywood golden age ended in 1942. His favourite film used to be Kind Hearts and Coronets; now it's The Gay Divorcee. You can see the influence of these films in his own work, a preference for the sophisticated, for the artificial Eden of Wodehousian musical comedy. The Astaire musicals inhabit an art deco universe, a place of artifice. Those directed by George Stevens may venture outside, but when they do so the world they find looks as charmed as a garden painted by Watteau. Stillman's movies have something of the same spirit, but take place on location, not in the studio but on the street. Yet still the films' locales present a world as we'd love it to be. His characters might long for plain, even homely partners, but in fact everyone is gorgeous. In Damsels in Distress, even Frank, a supposedly unattractive doofus, resembles a young Bill Clinton. Sevenoaks University is awash with an autumnal lustre. Leaving a pre-Christmas party, Metropolitan's heroine remarks how the night-time Manhattan streets remind her of War and Peace; it's a romantic city, and the subway at the close of The Last Days of Disco itself transforms into a disco, the travellers dancing with each other. Stillman captures disco's vacant shimmer, its beautiful melancholy, but he is also alert to the music's vibrancy, even, at that movie's end, as the O'Jays' "Love Train" plays, attaining a kind of joy.

For some, what compromises the Whitman idyll is that nearly all his people are prep-school-educated Ivy Leaguers. In Metropolitan, when the hero's divorced father moves without warning to Santa Fe, it shocks us, not just because he has so sneakily removed himself, but because it is hard to picture a town like Santa Fe co-existing with the world of the movie. In this film, at least, America has only one coast. These are movies by, for and about people who are no longer central to cinema, or, the films themselves argue, to life itself.

It is hard, even frightening, to imagine the contemporary British equivalents of Stillman's movies. The films would be about boarding-school survivors, trustafarians, sloane rangers, a milieu ignored by most native film-makers. Memories of the appalling Oxbridge film Privileged (1982) may arise, but are best repressed. Was there any British film that portrayed the "yuppie" sympathetically? In the past 25 years, a British drama would have to balance the picture by bringing in other classes as parallel or counterpoint, as in Mike Leigh, or retreat to the past like Merchant Ivory, or both, as in Downtown Abbey. The tone could neither be so fond nor so contemporary as Stillman manages, things would be sharper, more obviously satirical. It's Dickens versus Henry James.

Even in their American (and Spanish) context, Stillman's films include the sense that no one beyond their own clique will be endeared to their protagonists: in Barcelona they're scapegoated, in America they come across graffiti blazoning "yuppie scum", disco records are burned at a baseball game, the word "preppy" bears its taint of mild derision.

The trilogy's tone was in part formed by the abrasive screen persona of Whitman's lead actor, Chris Eigeman. But that abrasiveness is not Eigeman's alone; Stillman's people are earnest, indeed pompous, constantly "pissed" (in the American sense); in the early films, his people rarely smile. (It's interesting that the aggressive campus-editor Rick De Wolf, the nearest that Damsels in Distress gets to such a character, is so marginal to the plot.) The films present parodies of the judgmental, and yet offer a softness, a capacity for tenderness that is rare in contemporary cinema. The first half-hour of Stillman's films are sometimes taken up with the thought: will we like these people? With Damsels in Distress, at first we rather look down on Violet (the wonderful Greta Gerwig) and her companions – their equable, careful sympathy just looks a bit Stepford Wives; some will continue to dislike them, others are won over. In The Last Days of Disco, Charlotte (Kate Beckinsale) acts as a skilful saboteur of her prettier friend's ego ("You know, Alice, I'm not so much of a bitch as I might seem"). We enjoy despising her, but then the film finds not just her vulnerability, but her appeal; Stillman's films may honour virtue, but they're alive to the fact that people are loveable with, and even because of, their faults. With a couple of exceptions (the pony-tailed Rick Von Sloneker in Metropolitan, perhaps the pony-tailed Bernie in Last Days of Disco), no one in these films is loathsome.

Indeed, Stillman excludes evil and all truly dark passions. In Damsels in Distress, hearing of a potential suicide in progress, Violet and her cronies amble to the scene at a decorous trot. In Metropolitan, two preppy young men launch a raid to retrieve Audrey, the heroine they both love, who is lost, perhaps desperate, at sleazy Rick Von Sloneker's beach house; anything might be happening to her. The good guys' feckless quest plays as a cross between The Searchers and a chivalric romance. Too maladapted to drive, they take a yellow cab out to the country. When I saw the film in a cinema back in 1990, somehow this inept chase enchanted all the audience in a spell of danger. The heroes burst into a scene of decadent abandon that looks very lame indeed: one of the two women lays on the floor reading a magazine with her bra unclasped; the portlier of the two villains plods about the room wearing a pair of knee-length khaki shorts; the apparently compromised heroine is covered up in a pink polo neck jumper, reading Louis Auchincloss's earnest novel The Rector of Justin. It seems that Audrey didn't need rescuing after all, though she's glad to have an excuse to leave the house with her two incompetent knights errant.

In Barcelona, there is some attempt to draw in the larger world, showing anti-American terrorism: bombs explode, a man is killed, another is shot in the eye. This intrusion of the politically serious is one reason perhaps why, for all its merits, this is the least satisfying of all Stillman's films; the collision between his characters' idle perplexities and the cold war is just too abrupt. The moments jar, as if characters from The Battle of Algiers had strayed into A Room with a View. In Damsels in Distress, suicide is a constant presence, and the film therefore draws in a potentially unbearable despair. Stillman treads a fine line here, and some viewers will feel that the film cannot accommodate such passions without perpetually hitting the wrong note. And yet to me at least, the film does succeed without condescension in responding to thoughts of suicide with comedy, acknowledging pain and vulnerability but answering them with intimations of joy.

His are courteous comedies; no one is mocked in his films. In Damsels in Distress, when we first come upon the apparent knucklehead, Thor, who does not know the names of the colours, we may wonder if we're being invited to laugh at his stupidity. When we next hear of him, his ignorance is touchingly, and satirically explained: his parents were so hooked on the idea that precocity is a form of social triumph that they put him in school before he was ready, making him miss out on the last year of nursery, when his peers learned the colours. When Thor finally gets it, and ecstatically points out the colours in the rainbow, the laugh is, I think, on us. After all, when were we last delighted at colour itself?

Yet for all his taste for the frivolous, Stillman's worlds are not Edens – choices must be made, and they matter. He doesn't advocate goodness; he has fallen for it. Metropolitan's heroine is akin to Jane Austen's Fanny Price: unregarded but virtuous, but by the end of the film everyone is in love with her. His people are often instinctive puritans, people edging through the sexual revolution. In Barcelona, Ted Boynton (Taylor Nichols) reads the Bible in secret, sometimes while dancing to Glenn Miller. When his cousin comes back unexpectedly and surprises him at it ("What is this? Some strange Glenn Miller-based religious ceremony?"), the oddity is apparent, and yet the moment is a key to all Stillman's films, blending as it does spiritual engagement, moral consciousness, nostalgia and a cheerful embrace of human ludicrousness. The films celebrate modern life, the bright possibilities of parties or nightclubs; yet their heroes' moral lives seem to belong to some bypassed tradition. They would adhere to absolutes that everyone else around them perceives as relative.

Stillman's films favour the out-moded and antique. His characters are adherents of exploded idealisms, followers of forgotten heroes; one character is a Fourierist, another a Cathar. At least in this there's a recognition that the past once existed: if Austen's world looks silly to us, it's nothing to how silly we would look to her. In Stillman's films things teeter on the edge of ending. Metropolitan portrays what its participants claim will be "the last real deb season"; at a hotel function someone declares how they love the St Regis. "They'll probably knock it down soon," their companion replies. We're inside the self-dramatising melancholy of youth. One of his heroes has Spengler's The Decline of the West for his bedtime reading; civilisation decays, the "preppy" (or "Urban Haute Bourgeois") class meets its doom, and yet somehow the talk goes on.

Stillman's people are endlessly reflective, analysing themselves, each other, life, their class, their society; they have theories on everything from suicide to soap; relationships are conducted according to formulas; pop songs are looked to for romantic advice. In Damsels in Distress, Violet is something of a female Samuel Johnson (Stillman is known to be a fan), sharing his decided opinions, his moral questioning, even his neurotic compulsive habits. She declares her faith in clichés and hackneyed expressions, seeing these as repositories of the wisdom of the ages. It's a paradox that unsettles cliché itself, its conservatism in fact disquieting the stale response. There are many similar contrarian moments in the films, from the character who in watching The Graduate prefers Carl Smith, the blond make-out king jilted at the altar, to the seedy stalker, Dustin Hoffman, or another who finds Scottie the true loyal hero of The Lady and the Tramp, as opposed to the philandering Sinatra-figure that's Tramp. This inversion of expectations even extends to his characters' religious sympathies. At a screening of Damsels in Distress at the Rotterdam Film Festival, Violet's declaration that "We're all Christians, or Judaeo-Christians" evoked a collective gasp of horror – the only one I heard at a festival that prided itself on displaying the taboo-breaking.

The characters stand in bored hostility to the 60s. In The Last Days of Disco, Charlotte affirms: "Before disco this country really was a dancing wasteland. You know the Woodstock generation of the 1960s, that were so full of themselves and conceited? None of those people could dance." (Having seen Woodstock, I think the evidence for this comment may be there.) Yet, of course, they have more in common with the Woodstock generation than they'd like to think. Both endeavour to form community, in Stillman's case the "Sally Fowler Rat Pack", the world of the nightclub, the life of a sub-Ivy League university. Such small societies are both homogeneous and yet images of human variety, a sense of plenitude and difference that enters into the form of the films themselves. What other films could take in the sexual practices of Cathars, the therapeutic benefits of tap-dancing, the pleasures of Ronald Firbank, the uplifting properties of soap, and make it all somehow, more or less, cohere? In Damsels in Distress, Violet and Lily – linked by their floral names – suggest two contrasting ways of being: bright-eyed eccentricity versus the normally cool. Neither is preferable to the other. Having so far watched the film twice, I found myself preferring Lily the first time around, and siding with Violet on the second viewing.

Yet in plot terms, these little worlds cannot hold. Both Metropolitan and The Last Days of Disco end in separation, the disintegration of a social oasis. Above all, friendships prove transient; a circle forms, but then dissolves. For all Stillman's preference for the group over the couple, it's perhaps couples who are shown making the most permanent connections. The mood lapses into melancholy, though these movies close with affirmation.

Everything in Stillman ascends to sweetness. Twice his films end with dancing. In Metropolitan, someone is invited to dance the "Cha-cha-cha", and retorts that the dance is ridiculous. "It's no more ridiculous than life itself," her suitor replies. Stillman is a connoisseur of bad dancing – the disco moves in his first three films look refreshingly amateur. Yet partly because of their unskilled enthusiasm, the dances at the ends of his films also present a closing image of communal togetherness, the affirmation of harmony. Let's hope we do not have to wait 14 years more for another such vision.

• Damsels in Distress opens on 27 April.