In As You Like It, melancholy Jacques asserts: "I am for other than dancing measures." It's bewildering to discover that every Shakespeare play, including King Lear, ended with a jig, moving seamlessly from gouged-out eyeballs to cutting a caper. That closing dance expressed concord, delight, the measured order of things. Yet Jacques seeks to exile himself from pleasure and community; he won't let dance help him step towards the sweetness of life.

In Jean-Luc Godard's Bande à Part, two guys and a woman get up in a cafe and dance the madison. It lasts about three and a half minutes, and all that time, without one cut, the camera regards them, moving with them, just taking in the sheer purposeless charm, how handsome the men are, how beautiful the woman. Gesture assumes a pattern; watching them hypnotises you. It is not a dance film, but there they are, nonetheless, dancing, and every measured step leads us to freedom.

Reading on a mobile? Click here

In American movies especially, dance wants to assert the place of the individual, to find a space for their freedom, their spontaneity, and their capacity for improvised joy. It makes its stand against automation and pompous authority. Above all, it does so by being deliciously silly. By avoiding dancing, Shakespeare's Jacques refuses the embrace of his own foolishness. Near the beginning of even the best musical, there is a point where it may all seem just sublimely daft. From the first 10 seconds, Jacques Demy's Les Demoiselles de Rochefort strikes you as preposterous – it's so vivid, there's just so much dancing, it's all so pink – but, by the time Gene Kelly arrives, the absurdity has won you over.

Dance on film could only emerge properly with sound, and the synchronisation of the human body to music. In Fritz Lang's Metropolis, the robotic workers jerk in rhythm with the machines they service. It looks like dance, but one forced through the sensibility of totalitarianism. The men who shift are turned away, faceless, their movements a mechanised parody of the liberty found in dance. In the early 1930s, Busby Berkeley's musicals, despite their all-American sheen, present a similarly anonymising view. His amazing dance sequences swallow up individuals into a chorus; the figures are just an organised crowd. Seen from above, people lose themselves in the making of a design, absorbed into stars or ephemeral flowers. This is dance at its most abstract, an almost mathematical form – Berkeley himself referred to these sequences as his "homogeneous quadrangle equations" – closer to cubism than to the traditional Hollywood movie. Spectacle dissolves the story and rejects individual character.

Always the battle was between the self and that which would deny it. In Modern Times, when Charlie Chaplin cavorts and pirouettes around the production line, he offers an anarchic, individualistic riposte to the Henry Ford, Metropolis world, deftly switching factory work into ballet. (Nijinsky once told Chaplin that essentially he was a dancer.) Similarly, in Shall We Dance, Fred Astaire furthers the revolt against mechanism, syncopating his body to the pistons, wheels and turning-rods of a steam-ship's engine room, a gleefully human impersonation and transcendence of the machine. What was abject in Metropolis becomes pure enchantment.

When not matching himself against the machine and winning, Astaire was often inviting the censure of the stuffy – a classless American guy in Top Hat up against the po-faced upper-class Englishmen obeying the Thackeray Club's rule of silence. There's disapproval too in Singin' in the Rain, with the policeman who moves Gene Kelly along, and in Baz Luhrmann's wonderful Strictly Ballroom, with the old guard who resist the young hero's new dance moves. In such moments, the dancer embodies a natural pleasure of which authority disapproves.

Reading on a mobile? Click here

Kelly affirmed that "dancing is a compulsion from within, more authentic than the forms imposed from without". He had begun his career as a dance teacher, and something of the teacher always remained with him. In Anchors Aweigh, when he instructs the cartoon Jerry Mouse how to dance and gives up being a grouch, the movie taps into childlikeness, the capacity to achieve unselfconsciousness through playful imitation.

In relation to dance, two tensions held the Hollywood musical together. The first involved the relation between the white movie stars and the largely anonymous African-American dancers that they in part imitated. This was a boundary that could rarely be mentioned, though the "Slap That Bass" sequence in Shall We Dance and the shoeshine dance in The Band Wagon did their best to allude to it, however problematically. On the other border lay ballet. The American celebration of the natural entailed a suspicion of European classical styles, seen as French or, worse, Russian. Such foreign highbrow imports were suspect, especially if compared to the native vigour of the cakewalk, vaudeville and tap. The demotic dance world, vulgar and free, squares up against elitist pretensions. Yet dance in the American musical was always also in love with the ballet world that it affected to despise. In particular, the long "Broadway" ballet sequence in Singin' in the Rain shows American popular art aspiring to European achievements.

Ballet may come into Hollywood musicals, but the ballet film follows rules all its own. Dancing should be carefree, but such films portray the dance world as the home of suffering, whether through work and anxiety as in Robert Altman's The Company, or as pure gothic in Dario Argento's Suspiria. Behind every adaptation of Ballet Shoes there looms a Black Swan. These are largely films about dedication to art, where dancing means compulsion, pain or a shimmering illusion.

In ballet, exaggeration exceeds itself; close-up, the makeup looks grotesque; the plots are melodramatic, the gestures heightened. It is both vibrantly physical and oddly unreal, allowing us to imagine an unlimited freedom for the limited body.

The greatest of all ballet films must be Michael Powell and Emeric Pressburger's The Red Shoes (1948). This movie conducts us into an alien, backstage world, and then guides us to a narrative within, the ballet of The Red Shoes, a place where stories are told differently, through gesture related to music. Weak dance films make us a second-hand audience; others place us within the dancer's moment. The Red Shoes achieves both; we both watch the ballet and are inside it. The ballet pitches that film's tendency to stylised excess one notch higher, as a kind of enraptured surrealism takes over; the music metamorphoses the dancer, a knife becomes a rose, the audience turns into an ocean.

Watching Pina, Wim Wenders's glorious homage to the German choreographer Pina Bausch, you wonder how so much angst can be so joyful. Elsewhere, too, dancing can become poignant, simply by being so different from the rest of life. For all The Full Monty's zest, wit and generosity, there's an element of They Shoot Horses, Don't They? about it. When everything fails, all these men have left are their bodies. In their way, the best 50s Bollywood films similarly draw the glorious folly of dance into contact with poverty and genuine suffering. Raj Kapoor's Shree 420 possesses a spirit and a social conscience that shows up West Side Story for the act of stylised sentimentality it is. Dance here is the thought of hope amid the hopelessness.

Reading on a mobile? Click here

Few films can balance the contrast between pathos and energy so tactfully as Kapoor's. Perhaps the dance film is at its very best when most removed from life. The dance critic Lincoln Kirstein suggested that the technology of film would open up the fullest possibilities for dance, by which he meant "a treatment of human bodies comparable to the way Disney treats his puppets". There is something persuasive in that, not least because one of the greatest filmed dance sequences remains that of the hippopotami and the crocodiles in Fantasia; in "I Got No Strings", Pinocchio becomes one of cinema's greatest dancers, transcending – as only an animated puppet could, yet all the while also remaining an awkward child – all the limitations and constraints of the body. (Just as, for The Red Shoes, the cinematographer Jack Cardiff sped up Moira Shearer's pirouettes, making the impossible happen.)

Animation can seem the rightful home of dance, because for all the insistence on naturalness, the dance movie is as busy celebrating artifice. After all there is nothing more unnatural than suddenly breaking into a dance, an activity in the same relation to everyday movement as poetry is to prose. The Astaire-Rogers films take place in a studio-enclosed world, an unreal realm of chic, upper-class leisure, an ordinary gal in a ballgown, a man of the people in top hat and tails. In the past, film was far more welcoming to dance, because it already made room for the theatrical, the unnatural and make-believe. At a time when songs could be sung in dramas such as The Big Sleep or Rio Bravo, dance was similarly part of the vocabulary of film. Later, as naturalism and authenticity became key values, the dance movie somewhat dropped away – Saturday Night Fever being one attempt to weld together dance and the new taste for genuine urban grit.

"Writing about music is like dancing about architecture" – this celebrated warning has been in my mind. I am in many ways disqualified to write this article: my ankles are rheumatic, my dancing days are done; I cannot tell a plié from a pas de bourée; I must have seen Dirty Dancing, but have forgotten it. Yet both dance and film, and dance films have always meant much to me. Moreover dance films come across as magnificently welcoming, a club without bouncers or dress code.

Dance is as close as the human gets to being like music, the body converted into the instrument that plays. It is pure expression and, like music, it rejects no one. It is at its best in the single take, where you see the performance as a whole performance, well away from Flashdance-style trickery. Film gives us a trace of dance's magnificent physicality, just as dance brings something essentially theatrical to cinema. So it is that dance films work best in the cinema, with conditions as close as possible to the size and vividness of the stage.

But it's more than performance that the dance film hails; it commends and applauds play. When Gene Kelly sings "Singin' in the Rain", he's clearly on a Hollywood set. Yet in our imaginings, he's fooling around on a Los Angeles street, just like the real streets that Kelly and Stanley Donen had shot in while making On the Town. The dance film was moving out on location, and the street was becoming the stage. As Kelly dances, and the rain falls, he shows that to dance on the street is to renew it, to claim back the urban space as one briefly dedicated to frivolity and fun. And when in Pina, the dancers splash across a flooded stage, the moment clearly descends from Kelly kicking through those puddles. Both films – and all the great dance films – ultimately point towards this great transformation, an alchemy that with a touch of rhythm, and a gesture of style, changes the world to delight.

• Gotta Dance, Gotta Dance!, a two-month season of dance on film, begins at the BFI, London, on 3 July. bfi.org.uk.