Michael Powell once said about Peeping Tom, the movie that effectively wrecked his career: "It's full of sympathy. It just happens to be sympathy for a diabolical murderer." Fritz Lang's masterpiece, M depends on the same paradox; its coup de théâtre tests the limits of our pity, inviting us to share, if we can, compassion with a crazed serial killer. Played by a youthful Peter Lorre in 1931, this would be the first, the greatest and most representative major role in a career that, in its own fashion, goes to the heart of what makes the movies so intriguing and so potent.

M concerns the pursuit of a child murderer loose in Berlin; what's unusual about the film is that – frustrated by the police's lack of success – gangsters and career criminals unite to track down the predator themselves. In part, moral revulsion motivates these underworld gumshoes, though it's also clear that killing children is bad for illicit business. There are too many raids and too many police on the streets. The madman must be stopped so that everyday larceny can continue.

M still amazes. It influenced dozens of later movies, yet it is unlike anything else. It is hard to think of many other mainstream films that are so radically decentred, so reluctant to grant us a hero. Yet it's an unexpectedly methodical, measured picture; the only dreaminess attaches, just off-centre, to the spaces where the killer himself moves. The film's subject-matter could not be grimmer, yet Lang enlivens grey emptiness with an Otto Dix-like burlesque energy. Around the random murders, the film hones in on Berlin's melancholy, a landscape of puddles and prostitutes, a silent city – the hush broken by bells or shrieks, the tap-tap of a tool being sharpened, and haunted by the killer's eerie whistling of Grieg's "In the Hall of The Mountain King". Yet against this frosty atmosphere, the film revels in its satire, its playful equation of cops and robbers. The criminals so much enjoy hunting Lorre down that we tend to forget why they are doing so. These hunt scenes are oddly relaxed, even frivolous in a chilly way; the criminals are simply getting down to work. The only vital passion belongs to Lorre; while the criminals are rational figures, he releases despair and fury, a rage that finds its echo in the crowd that faces and judges him.

It's a film about murder that pays close attention to the business of living. For those who know Berlin, there's something beguilingly moving about the glimpses of the city's life – the Mickey Mouse figurines in the confiserie, the list of prices in a cafe, the faces of invalids and beggars. It is a "city film" at the end of the grand era of city films, though the city it depicts is a dark, troubled, paranoid one. Above all there are the glances into a child's metropolis of toy shops and school gates, of hurdy-gurdy men and balloon sellers. Lang looks on at a childhood world, seen from above and at a distance, not with the children but observing them. M's child victims are strangely on the edge of things, unknown by the film, hardly ever seen close up. The film's final spoken moral, "we must take more care of the children", is one that remains relevant, and with which nearly all will agree. Yet as an ending it feels somehow off-key, a heartfelt plea that is also a platitude, and that does little to resolve the weird energies of the movie itself.

The first time we really see Lorre, he's at the mirror, darkly enjoying acting weirdly, pulling faces, indeed literally pulling a face, turning himself into another. It might be said that he would act out that craziness, professionally, for the next 30-odd years. He had not come from nowhere; M was his fourth film. But those other roles had been slight affairs, and in M, he was the closest thing to being the movie's star. Here his relative obscurity enabled him to personify anonymity; practically unknown on screen, he could also be frighteningly invisible on the streets. One message of Lang's film is that anyone might be guilty.

It is one of M's unintended ironies that in the end the killer should have one of the most distinctive and caricaturable faces in movie history, as identifiable encapsulating the macabre as a drawing in a Bugs Bunny cartoon as he was in peering around Sydney Greenstreet's substantial shoulder. Far from appearing anonymous, Lorre presented one more way in which cinema discovered uniqueness. There was only ever one of him.

Lorre looked like a sleazy baby, his face registering every passing petulance, ready to drop from a hopeful grin down to a sulk. When acting, he seems haunted, shiftless; he moves between an uncanny calm and fits of restless mania. This instability might be traced back to the desperate rhythms of his life as a morphine addict. His style depends on rapid transitions, his talent for letting an emotion leap towards its opposite. Dramatically, this proves vivid, but it also leaves us with a sense of the characters he plays as purely reactive, puppets of expression.

He was born in 1904, in Rosenberg, Hungary, as László Löwenstein. He fled the prospect of a life as a bank clerk for the stage, training in Vienna and making his debut in Zurich, before being discovered by Bertolt Brecht. Lang's film made him a star, but stardom was no defence against the Nazis, and in 1933 he fled to Paris. All but broke, he was rescued by a meeting with Alfred Hitchcock. That evening, all the English that Lorre then knew was "yes" and "no", but he impressed Hitchcock largely by using as often as possible the first of these two words, and by falling about with laughter every time he guessed that yet another of the English director's anecdotes had reached its punchline.

So it was that a few years after M, in his first English-language film, Lorre was endangering children again, kidnapping a teenage Nova Pilbeam in the first version of Alfred Hitchcock's The Man Who Knew Too Much (1934). Remarkably he learned to speak English even as he was playing the part. He's outstanding, a precursor of Graham Greene's Pinkie, a wicked boy with a blonde (or grey) streak in his hair. He offers the threat produced by the apparently unthreatening; small, plump, frail, he nonetheless conjures up a strangely devious menace. Quite rightly, he stole the show, transforming what had begun as a bit part into the movie's dynamic centre; when the film was advertised, it was Lorre's face that dominated the poster. Curled and moustachioed, he's almost as memorable in another Hitchcock film, Secret Agent (1936), based on Somerset Maugham's Ashenden stories, and a movie of marvellous moments. From Hitchcock, he would pass to Hollywood, where he would become one of the era's defining stars.

Two versions of Lorre face each other in uneasy connection: there's the respected Brechtian actor Peter Lorre (rhymes with "Gomorrah") versus Peter Lorre (rhymes with "sorry"), Hollywood's simulacrum of everything foreign, alien and fascinatingly toad-like. Lorre could stand in for any kind of suspicious foreigner – for instance The Maltese Falcon's (1941) Joel Cairo, a man with many passports, all equally unconvincing. In Secret Agent, John Gielgud hunts for a concealed German spy, though that movie's real secret German-speaker is Lorre himself. The East End shoot-out at the end of The Man Who Knew Too Much inevitably recalls the Sidney Street siege, and memories of murderous continental anarchists; here Lorre was an émigré actor touching on the nation's darkest fantasies of an immigrant intrusion. In the late 1930s, he achieved great success churning out a series of mediocre if charming crime films, as the inscrutable Japanese detective, Mr Moto. He's prone to murmuruing sagely, "Ah, so!" and offering up implausible lines such as (describing a drawing) "What harmony and colour. Truly this is a voiceless poem." At the time, this kind of thing went down well. After all, Lorre was inescapably alien, a man to whom the usual rules and social conventions may not apply. In Secret Agent, moral scruples perturb the decent English while, killer and clown, Hispanic Peter Lorre finds the murder business ludicrously comic. A dozen national stereotypes had coalesced in one man.

Mr Moto was an archetypally chaste detective, an oriental Father Brown. Though he could be lascivious in Secret Agent and Strange Cargo (1940) or erotically obsessed in Karl Freund's Mad Love (1935), onscreen sexlessness suited Lorre, who was rarely (if ever) allowed a successful romantic relationship. On set with Hitchcock, he was nicknamed "The Walking Overcoat" because the long coat he wore all but trailed on the floor so that it seemed to be wearing him. (Though it may not have been the coat that was long, but rather Lorre who was short.)

Some have felt that the Mr Moto films wrecked Lorre, turning him from a highly inventive character actor into a Hollywood hack, even though some of his most memorable roles (including Casablanca) came after them. He played best with a great foil, his ridiculous relish for excess up against John Gielgud's elegant Home Counties restraint, his penchant for improvisation unnerving the stage actor's disciplined regard for the script. In The Man Who Knew Too Much, matched by Leslie Banks, Lorre cannot help but steal each scene; he's a physically present actor, often, you feel, surrounded as he is by the pallid English, the only one in the room with a body. Paired by Warner Brothers in a recurring double-act with the imposingly corpulent Greenstreet (they have been memorably described as "the Laurel and Hardy of crime"), Lorre found a home in film noir, a genre receptive to grotesque vigour and the eccentric variety of life. Though both are excellent in The Maltese Falcon, their first outing together, the prize goes to Greenstreet, who (almost literally) fills the screen. Elsewhere honours are more even, though it's precisely the contrast between the two men – the secret connection present in the contradiction of types – that was the key to their achievement.

At his best, Lorre occupied that uneasy territory between the silly and the sinister. He was a great star, a quintessence of the enchanting variety offered up by mid-century movies. If an air of disappointment hangs around him then that too is somehow intrinsic to the vision of the world summed up by his image. It is unfortunate, but appropriate, given his world-weary, hangdog demeanour, that one of his last films was titled The Sad Sack. He was successful at embodying human unsuccess. None of his characters get what they want, unless it is to kill and be caught like Lang's urban murderer. He died in 1964; given his chaotic and driven life, it is miraculous that he survived so long. It is our luck that he left behind so many versions of himself, doubled, distinct, and yet all instantly recognisable as Peter Lorre.

• The Peter Lorre season continues at the BFI Southbank, London SE1, until 7 October. bfi.org.uk