In Edward Norton’s new film, Motherless Brooklyn, a keening ballad blows in and out, affecting the narrative and painting the prevailing mood with a deep shade of blue. It’s unmistakably the work of Radiohead’s frontman, Thom Yorke.

Some film songs (Stayin’ Alive, The Harder They Come) sit so snugly with the tale that one can barely see the join. Others (Mrs Robinson, Raindrops Keep Falling on My Head) drop like gaudy visitors from another world. It’s an inexact science, a curious alchemy. Norton and Yorke are still figuring it out.

“As a director, your job is to take a lot of talented people and pull them into a frequency alignment,” Norton says. “You’re trying to create a coherent aesthetic. But that’s exactly the wrong headspace to be in with regard to the music. The worst thing you can do is to get in the way. You don’t want to over-manage a whole separate poetry.”

If the film and the song should be standalone entities, it therefore follows that the collaborators must be, too. Here they are, perched side-by-side in a posh hotel; the crisply respectable film-maker and the rumpled, elfin musician, like a Norman Rockwell illustration next to a cartoon by Dr Seuss. Norton orders a pot of tea. Yorke, for his part, opts for a tequila. He smiles. “What can I say? I’m starting early.”

Norton has wanted to make a film of Motherless Brooklyn ever since he read Jonathan Lethem’s source novel in 1999. The result is rich, gamey, abundant; a labour of love. Norton wrote the script, called the shots and takes the lead role of Lionel Essrog, a whip-smart private eye who uncovers institutionalised corruption in 1950s New York. Lionel has Tourette’s, which has made him a pariah. He jerks and yelps and puts his hand on random bystanders as though to anchor himself. The condition is a drag but it’s his rocket fuel, too.

Shortly before shooting, Norton decided that Lionel needed a song. Something fractured and beautiful; something to reflect the inside of the man’s head. He found himself thinking of Billie Holiday’s Strange Fruit. “It’s the most mournful of songs but it’s also political, it’s about living in dark times. And I thought: ‘Well, you know, Thom is my Billie Holiday.’”

“I’ll take that as a compliment,” Yorke says, as if there’s any other way to take it. “I mean yes, Strange Fruit is like the ultimate song for me.”

Daily Battles is not Strange Fruit any more than Motherless Brooklyn is Chinatown, the film’s most obvious influence. But it’s an excellent ballad all the same – something soft, raw and gorgeous tucked amid all that hard-boiled noir aesthetic. Just as Lionel explains that he sometimes feels that he is two people living inside the same body, so the film and the song somehow conspire to share the story between them.

“Yeah, but I had no idea whether it worked or not,” Yorke insists. “It’s a weird experience, because you’re working outside of the film. But you still have to try to make it personal. Maybe that’s the mark of every good song. If you make it personal it carries everything else with it.” All told, he reckons he just about got away with it. He puts down his shot glass and mimes mopping at his brow.

In the end I think I liked Motherless Brooklyn for the same reasons others didn’t (the reviews have been mixed and the box office grim): because it’s lengthy and knotty and longingly romantic as only a dyed-in-the-wool cynic can be. Norton is aware of the problems. He admits that the tale’s clashing ingredients can be both a blessing and a curse. “I’d describe this film to people and say: ‘Well, it’s a big period epic about dark things in the vein of the best noir. But by the way it’s also got Rain Man at the centre of it.’ People go crosseyed. It’s like you’ve offered them an olive and a piece of chocolate. They think: ‘I might like them both, but they don’t go together.’”

Gugu Mbatha-Raw and Norton in Motherless Brooklyn. Photograph: Allstar/Warner Bros

Norton and Yorke, it transpires, have known each other for years. They rose up at around the same time, in the mid-1990s, with each man running along a parallel track. Norton was the upstart bright spark of American acting, Oscar-nominated for his screen debut in Primal Fear, playing alongside Brad Pitt in David Fincher’s Fight Club and clashing with the director Tony Kaye on the set of American History X. Yorke, in the meantime, was steering Radiohead towards global stardom and sell-out stadium tours, before buckling under the strain of it all.

I’ve heard a rumour that they almost worked together last century. Wasn’t Yorke originally tapped to write the soundtrack for Fight Club?

The singer nods. “Got the email. Got the script. And I was just too fucked up in the head to do it.”

Norton takes up the story. “It was probably my fault. We were listening to The Bends and OK Computer constantly in the makeup trailer. Brad and I were obsessed – we had those albums on all the time. Then we started leaning on David [Fincher] a little. ‘Wouldn’t it be cool if Radiohead could do the score?’”

Yorke, belatedly, is all apologies. “I’d just come off tour. I was mentally incapable of even tying my shoes.”

Instead it was Jonny Greenwood – Radiohead’s guitarist – who first made the move towards film soundtracks; he has scored every Paul Thomas Anderson picture since 2007’s There Will Be Blood. Two years ago, Yorke followed suit, rustling up a haunting electronic arrangement for Luca Guadagnino’s remake of Suspiria. He explains that he spent 18 months on the project; wandered down all kinds of rabbit-holes; dredged up lots of treasure. “But then it becomes like a weird piece of art criticism where the director listens to all the material and goes: ‘Oh, I’ll just take this bit and this bit.’”

Norton drizzles a spoonful of honey into his tea. “Not to say that we don’t love Luca,” he says. “But Thom came off Suspiria feeling so burned.”

Yorke and Norton at an event in London. Photograph: David M Benett/Getty Images

The initial plan was to have Yorke write and record the entire Motherless Brooklyn soundtrack; steeping himself in 1950s Harlem jazz; fitting the film with something like 75-minutes of music. But it was too big an ask; the two men would have fallen out. “The last thing you want with somebody you know is to get into a bloody situation,” Norton says. “That’s the nightmare of all nightmares.”

“That and the fact that I don’t do jazz,” Yorke adds.

“Well yeah, it would have been different. Lots of egos. And then I talked to Paul [Thomas Anderson] about it, about maybe working with Jonny [Greenwood]. And he said: ‘If you expect any iterative back-and-forth flow with Jonny, this is a mirage. He just sends me what he’s got and says: ‘Good luck to you.’” Norton laughs, a little mirthlessly. “And, well, no. That wouldn’t work for me.”

As it is, the biggest ego in Motherless Brooklyn is a man named Moses Randolph. He’s loosely based on Robert Moses, the notorious “master builder’ of mid-century New York whose creation of the suburbs amounted to a form of social cleansing. But the fact that Moses is played by a bullish Alec Baldwin, who impersonates Donald Trump on Saturday Night Live, suggests a more obvious contemporary parallel. Norton insists that this was largely unintentional.

“Look,” he says. “I finished writing the script in 2012, when the insane clown was still a gameshow host. No one was paying any attention to him then. But then obviously, later on, the resonance increased. So yeah, there are a few signposts there, a few little arrows. But only up to a point.”

Sitting at his side, Yorke heaves a heavy sigh. “Trump’s going to be gone soon anyway,” he says. “He’s going to be history. The real question is why we keep producing people like Trump.”

“I’d go further,” chips in Norton. “The really dangerous people are the ones you can’t see. Trump is someone else’s puppet. It’s the people in the shadows you have to worry about.”

“Robert Mercer,” says Yorke.

“The Koch brothers,” breathes Norton. The men stare down at the carpet as though in search of dark patterns, two contrasting talents in harmony at last.

• Motherless Brooklyn is out on 6 December.