All thoughts of conducting a straightforward interview with Tom Waits turn to steam within seconds of his arrival in the Soho hotel suite. I come in through one door, carrying a notepad and a tape recorder. He comes through the other, carrying the exact same equipment. "Now OK," he says, arranging his effects on the table. "You have your questions for me, and then I have some questions for you." Introductions complete, he whips off his porkpie hat to let the hair stream up. He could be a conjurer unveiling a rabbit.

One does not so much interrogate Waits as be granted an audience, a private performance. Talking to the press, he once confessed, is like talking to the cops. You only do it when you have to, and it is always better to bear false witness. So he will claim he was raised by a pair of circus acrobats, or that he met his wife after busting her out of a convent, or that he trained as a doctor and still occasionally practises on the kids. "Most of the time I just tell 'em stories," he allows. "And if the stories are entertaining, who cares whether or not they're true?"

Waits is in town to discuss his role as the devil in Terry Gilliam's film The Imaginarium of Doctor Parnassus. I tell him Gilliam has said this is the role he was born to play, and he chuckles and says he can't think why; he was raised in the church. I mention this is actually his 25th screen role and he shrugs and says that Oh Lord, he wasn't keeping track. In person, Waits looks much the same as he always did: the same hunched, simian posture and weathered Dustbowl features; the same wispy, rust-coloured bouffant. This, perhaps, is the benefit of a life spent play-acting the wily, disreputable old puck. At the age of 59, he has finally grown into the costume.

Now look, he says. He figured I would probably ask him about some movies he likes, so he has done his homework and written them down. He crouches over his notepad and reads out the names. Putney Swope. The Pawnbroker. The Ox-Bow Incident. "Did you ever see a movie called Dirty Little Billy?" he asks, squinting up at me. "Starred Michael J Pollard. Made right around the same time as McCabe & Mrs Miller."

The jump-rope test

Acting, he points out, is merely a sidebar. He has been fortunate enough to work with the likes of Coppola and Altman, Jarmusch and Gilliam, and yet this has always been secondary to the music. Music was his first love – but then everybody loves music. "What you want is for music to love you back. That's why you pay your dues. You want to feel like you belong and are part of this symbiosis, metamorphosis, whatever you want to call it. That one day … " He coughs and regroups. "I used to imagine that making it in music – really making it in music – is if you're an old man going by a schoolyard and you hear children singing your songs, playing jump-rope, or on the swings. That's the ultimate. You're in the culture."

Is that what motivated him? The quest for immortality? "I guess, to a certain degree. Whether you say it or not, that's what you're thinking. You want to feel ongoing, because it's like getting extra time." He glances at the page. "Now, what do you think of Zatoichi? The blind samurai."

'We're all insects crawling on the shiny hood of a Cadillac'

Waits was born not to circus performers but to a pair of teachers in Pomona, California. He was, by his own account, a strange little boy: bookish, overwound and with a tendency to be spooked by untoward noises. He did not thrive at school, he says, because he did not like the little holes they drilled in the cork-board ceiling, or the hooked stick they used to open the windows. He did not like being young, and took to shuffling around with his granddad's hat and cane.

Later, he fell under the spell of Charles Bukowski and the beat generation, and took to hanging out amid the flotsam of downtown LA. He was fascinated, he said, by "the great American loneliness", a loneliness that stretched from coast to coast and was as elusive and mysterious as ground fog. "Yeah, that all came from Bukowski and Kerouac," he recalls. "I always liked the idea that America is a big facade. We are all insects crawling across on the shiny hood of a Cadillac. We're all looking at the wrapping. But we won't tear the wrapping to see what lies beneath."

Throughout the 70s, Waits viewed the fog at eye-level. He lived semi-rough at the Tropicana motel, where he would set his piano up in the kitchen and fish his songs from the old men who sat in the lobby. "You know how it is," he says. "If you're a writer you know that the stories don't come to you, you have to go looking for them. The old men in the lobby: that's where the stories were. And then when the record label would send me on tour, I always resisted checking into the usual places. I'd step off the bus and look for the hotels named after presidents." Hotels named after presidents, he argues, guarantee a certain grubby authenticity. "The Taft!" Waits says with relish. "You could usually rely on finding a Taft in every town. Take me to the Taft! You walk in and there they are: the old men in the lobby."

Bourbon and rumba

Waits's musical output falls into two distinct categories. The songs on those early albums arranged themselves like the patrons of a seedy lounge bar. They were woozy, jazzy offerings, marinated in bourbon, and spinning tales of loss and longing and half-chances that never quite came good.

Then, with 1983's Swordfishtrombones, a curious transformation occurred, and these barflies grew wings. They began hammering on the lamp-shades and rattling the optics. They learned rumba, gospel and delta blues. They became wilder, richer, more radical.

I confess that I like the early songs as well. But it's too late, they're gone, disowned like bad relations. "I'm embarrassed by them," he admits. "It was a time when I was trying to find my place within the business. I was figuring out who I was and where that person intersected with the world of commerce. It was like I was sitting there with a ventriloquist's dummy on my knee. And the dummy is made out of wood. And after a while you start to hate each other."

Then whoops, it's back to the notebook. "Cantinflas," he says, bent low over the page. "You know Cantinflas?" Cantinflas was Mexico's answer to Charlie Chaplin and a comic beloved by Waits's own father. "Oh Cantinflas, he was something else. He had the hair and the walk. Looking back, I now see that I lifted a lot of my act from Cantinflas."

Who's steering the ship?

The catalyst, however, came courtesy of his wife. Waits met Kathleen Brennan on the cusp of the 1980s, when she was working as a script consultant on Coppola's One from the Heart. It was Brennan who broadened his range, knocked him out of his rut. Without her, he says, he would probably be playing in a steakhouse today.

"She rescued me. Maybe I rescued her too; that's often how it works. Upshot is that we both got into the same leaky boat. Maybe the weight drags it down, because now you've two people sitting in it. Sorry, baby! But on the other hand you've also got two peoples' imagination to patch it up again."

The fact is, women are just that bit smarter than men. "Everybody knows she's the brains behind Pa, as Dylan might have said. I'm just the figurehead. She's the one who's steering the ship."

Specifically, she has steered it all the way from downtown LA to a home amid the hills of northern California. Brennan helped Waits to clean up his act. He quit smoking, embraced sobriety and went on to raise three children who are now all but grown, except the singer argues no one ever really grows, they just become different. In the meantime, the albums have kept coming, even if they wash in at a slower rate these days. Real Gone, in 2004, was his last collection of original material. Since then, he has put out Orphans, a collection of offcuts and offshoots, and has a live album, Glitter and Doom, set for release next month. "I'm almost 60," he marvels. "An honest-to-God old man. I'll write you from there and tell you what it's like."

Do we have time for one last trip to the notebook? It transpires that we do. Waits loves Toby Damnit, a short film by Fellini, and the opening scene from Once Upon a Time in the West, when the tin windmill turns around and says "Weaargh! Weaargh!" He loves Central Station and City of God; all those fresh films out of Latin America. I tell him about Tony Manero, a Chilean black comedy that came out earlier this year. He likes the sound of that one and duly jots it down.

"You know what one of my favourite movies of all time is? And if I'm at home with my kids and say, 'What do you want to see?', the big joke is, 'Aw Dad! Not Pig in the City!' But I love that movie. I'd see that any time."

I tell him I like the Babe sequel myself, but am now struggling to recall the details. Wasn't there some scene in a vivisectionist lab? A tragic orangutan who won't leave his cage because he is not properly dressed? "Oh, I know," groans Waits, raising a hand as if to ward off evil spirits. "Oh God," he says. "Don't." Nothing pierces his heart so keenly, it seems, as a monkey that has spent too long in the world of men.

'When you're in hell, keep going'

I ask whether he ever feels nostalgic for the wild years, when he lived at the Tropicana and laid his head at the Taft. "I can't say that I miss it," he says with a shrug. "We're all eating our way through the potato. Like they say: when you're in hell, keep going. Don't look back, because someone might be gaining on you."

In any case, he says, so what if he no longer holds court at the Tropicana? The world is different, but it is not entirely different. There are still dark pockets to explore, so long as you know where to look. If he wants to spend a night out on the railroad tracks he still can: he just has to plan it in advance. And if he cares to check in at the Taft then hey, he can do that too. "I kept hold of the room key," he confides. "I can go back anytime I want." And at this point, Waits dissolves into an emphysemic cackle. "That's the key," he says. "The key is the key."