"I used to think that all great recordings happened at about 3am," Tom Waits is telling me, in the conspiratorial, wasted and wounded voice that still seems made for those early hours. "So my first studio experiences, I wanted to be recording after the bars closed. I just thought that's when it all happened. And it worked for me for a while, I guess. But I don't believe that so much any more. I realise now there's more than one way to sneak up on a herd of cattle…"

Waits is sitting in the back room of a roadhouse near his home town of Santa Rosa, where the industrialised farmscape north of San Francisco starts becoming wine country. Like him, the Washoe House is something of a revered and ramshackle institution. Just about every visitor since Ulysses S Grant – who reportedly made a speech from the balcony in a state of undress after an amorous encounter on the way upstairs – has pinned a dollar bill to the ceiling, and no one has ever been desperate enough to take one down. Country music is playing on a jukebox to which Waits, ever alert, has one ear cocked, like a dog in front of a fire. A couple of times he will break off from talking to grunt a snatch of some cowboy melody.

On the table in front of him is the book he is reading, Crow Planet, about how corvids are "very much smarter than you might imagine", a cup of black coffee (he gave up booze a long while ago, having got his share in early) and a notebook in which he keeps his ideas for songs (He occasionally reads from it at random: "Here's one: 'It's good to be 40 feet tall on a billboard or something but not when your wife's dying of cancer and you just knocked up the babysitter.' Another 'I wish I was a component of water and I could go off in the sun and just dry out…'). He gives the impression of being in a state both of constant startled awareness, and vague puzzlement at the world. Some of this has to do with his hair, which seems to have led a long and interesting life of its own. Unusually, he is not wearing a hat.

We are talking about his latest record, Bad As Me, his 17th, and his newfangled habit, at 61, of getting to the studio early in the morning. "These days I want to be there before anyone has had a musical thought," he says. "If you are making a record you are the one saying 'action', and you are the one saying 'cut' and you have to be sure that the most interesting thing is not going on outside the frame. I try to pay attention to what people are doing the moment they come into the room. If they are just goofing around before we begin that may be the best thing they do all day. I have to be waiting."

Since the musicians on Bad As Me include such legends as Keith Richards, the bass player Flea (of the Red Hot Chili Peppers), and the extravagantly gifted Marc Ribot you can see why Waits might want to be on his toes. One of the tracks on the album finds him duetting with Richards. "I'm the last leaf on the tree" they croak fabulously in unison and you are half convinced that the end times might be upon us.

In this sense you could be tempted to think that this is a glimpse of late-period Tom Waits, the home straight. The fact is, though, he has always liked the idea of being last man standing. If there has been a constant theme of his career – which has taken in two Grammy awards, scene-stealing acting roles for the likes of Robert Altman and Terry Gilliam, an Oscar nomination and a unique gift for performance that made his rare live shows just about the hottest ticket in any town – it is the constant sense of imminent dereliction. His first album, Closing Time, made when he was 24, already saw him adopting the broken-down voice of a survivor of all that life and love might throw at him. The hats he has taken on and off since as a performer, late-night barfly, all American hobo, fairground huckster, have all suggested several lifetimes of hard-won experience. Now he is finally approaching the age he has always imagined himself to be, I wonder if it feels like he hoped it would?

He gives his guttural half-laugh. "It's the usual story. When I was younger I wanted to be older," he says, "now I am older I am not quite so sure."

The album is the latest in his heroic one-man attempt to include the whole history of American song in his own voice – now bellowing like a deranged Louis Armstrong, now essaying a Marvin Gaye falsetto, now groaning like Lead Belly, always very much himself. Waits likes to divide his repertoire into "grand weepers and grim reapers" or "bawlers and brawlers"; Bad As Me is no exception to this. It starts with the most convincing runaway train you've ever heard, Waits shovelling coal insanely, and then shifts gears between desperate, defeated ballads and the kind of "rumpus" you imagine from the party scene of Where the Wild Things Are. ("Anyone who has ever played a piano," Waits likes to say, "would really like to hear how it sounds when dropped from a 12th-floor window." His music satisfies that curiosity.) The band are required to do more than keep up. "It's like Charlemagne or one of those old guys said," he notes. "You want soldiers who, when they get to a river after a long march, don't start rooting for their canteen in their pack, but just dive right in."

It seems beside the point to ask Waits whether he feels a keener sense of mortality. Death has never been far from his thoughts as a songwriter ("I guess it's a pretty popular subject among those of us still breathing," he says.) But he signs off Bad As Me with a chilling little coda – "What's it like when you die?" – imagining the eternal hangover in increasingly baroque similes. Is that the kind of thing that keeps him awake at nights?

"Well, I guess," he says. "But that song started out just as a riff, taking triple rhymes, batting them back and forth. 'Like a jail door closing, like a male whore dozing…' And so on." The free association lets him access some strange truths he believes. The other place that Waits has always gone to look for those surprises is in his own voice. Does its scalded range still shock even him from time to time?

"I'd like to think so," he says. "Jimmy Stewart said he stopped making movies because he didn't like the way he looked on screen anymore. I'm more the guy who says I look like hell but I'm going to see where it gets me."

What started out for Waits as a stage act – the ultimate raddled crooner – became a compulsive caricature and is now a kind of larger-than-life persona that he can slip in and out of as he likes. Looking back though, I wonder why, as a kid growing up in southern California, he was so keen to advance his years?

"Probably because I knew I was going to have to be my own father," he says. "There was a need for maturity and guidance from somewhere and I was going to have to provide it for myself – even if it meant putting on an overcoat and a hat and looking in a mirror and squinting a bit. It's pretty usual for a kid from a broken home."

That latter fact has always seemed the most salient scrap of knowledge about Tom Waits's much mythologised background ("I was conceived one night in April 1949 at the Crossroads Motel in La Verne, amidst the broken bottle of Four Roses, the smouldering Lucky Strike, half a tuna salad sandwich…" and so on). When he was 10 or 11, his father – a Spanish teacher who used to drive his son from suburban Los Angeles over the Mexican border for haircuts and to listen to mariachi bands – walked out and didn't come back. Waits became, he told one interviewer, immediately fixated with dads. He would go around to his friends' houses not to see them but so he could hang out with their old man. At 12, he carried his grandfather's walking stick and wore a trilby (the hats have never gone away) and he would clear his throat and ask his friends' fathers stuff like: "So how long you been at Aetna, Bob?"

It was a natural progression to extend his search for adult wisdom to music, he suggests. I recall how he once described his childhood as "kneeling by a jukebox, praying to Ray Charles". Did they ever meet? "I never got to know Ray, no," he says. "But I did shake his hand at an airport once, which was a very profound moment for me. He was surrounded by a cadre of very scary-looking people. It was like a president was in town, everyone on walkie-talkies, but for some reason I just walked though and grabbed Ray's hand. It felt like the biggest thing I ever held, just this huge hand. And I squeezed it and just said, 'Thanks, Ray, thanks for everything.'" He pauses, looks aslant. "It wasn't so profound for him, I guess, but it wasn't meant to be."

Though Waits has never been a mainstream star like Charles, few performers attract more ardent disciples. Do people come up and grab his hand for the same reason now?

"There are a few people who want to let you know," he says. "And that's maybe profound for them, I guess. But the odd thing about this life is always that you spend half your time trying to get people to listen to you and the rest of the time trying to get them to leave you the fuck alone."

Over the years, Waits has developed more effective strategies for deflecting invasive attention than most. His attitude to autobiography is summed up in the lines from "Tango Till They're Sore" on his seminal album Rain Dogs: "I'll tell you all my secrets but I lie about my past…" He has always appeared to work on the basis that if you make the yarns entertaining enough, why would anyone want to know the truth? There have been a couple of attempts at biography, notably Barney Hoskyns's indefatigable Lowside of the Road, but Waits guards his privacy with his life. Interviewers seeking facts have occasionally crashed and burned. In response to the bland questions of an Australian TV host, Don Lane, Waits infamously spent the first few minutes of live TV distractedly looking for an ashtray. When he looked up he noted the interviewer's discomfort. "What's up, Don, you are starting to sweat?"

There are one or two moments in our conversation when my attempts at cod psychology are greeted with an amused sneer. "Ah I get it, you're observing me, you're the Observer," but they don't last too long. (I also work for the Guardian, I point out. "Ah, so you're looking out for me, that's OK then.")

Waits's original stage persona was born out of a similar sense of self-protective mythology. He suggests he feels a kinship with clowns, citing the Mexican hero Cantinflas as the performer he feels most sympathy with. Though he is capable of great pathos, nothing is ever quite in earnest in his music. Even his love songs carry the gruff awareness of their mechanics with them; he was compared with Kurt Weill before he knew Weill's music or Brecht's philosophy, but he had stumbled into the same theatrical territory.

To begin with, Waits had set about living the life his songs tended to describe. While performing in clubs, he dossed for a long while in cheap motels and flop houses in LA and occasionally lived out of the back of his '55 Buick. In some of this, he acknowledges, he was again trying on his dad's wayward life for size. He exorcised that ghost pretty much in his two-act musical Franks Wild Years in 1987 in which the eponymous hero torches the family home (Waits's dad was named Jesse Frank Waits, after the outlaw James brothers). The hard-living had reached its inevitable dead end. "There ain't nothing funny about being a drunk," he observed, looking back. "You know, I was really starting to believe there was something amusing and wonderfully American about a drunk. I ended up telling myself to cut that shit out."

The opportunity to do so came when he met his wife, Kathleen Brennan, in 1980 on the set of Francis Ford Coppola's One from the Heart. He was writing the music, she was helping to edit the script. They were married at 1am a couple of months later at the 24-hour Always Forever Yours Wedding Chapel in LA. His life was immediately transformed. They moved out up to the farm in Santa Rosa; he changed record labels in order to protect his musical freedom and to avoid the pressure to have hits; they began writing songs together and they raised three children.

Brennan is even more private than her husband, but he leaves you with little doubt she has been his soulmate as well as his muse. "My wife doesn't want me to say how long we've been married," he suggests. "Or, better, she likes me to say that she was basically a child bride. You know, 'They said it was wrong, but we didn't listen'; 'I told her I'd wait until she was old enough', that kind of thing."

Brennan encouraged Waits to express the more dislocated sound he had wanted and which has given his career its adventurous longevity. "I think everyone has irreconcilable musical differences," he says. "You know when you throw a party, you think people will show up and no one will like each other. It's like that with music – parts of your musical psyche have never met other parts. You wonder if you should get them together. I used to think it was good to keep them apart. Now I kind of throw them in and see what happens."

The first collaboration with his wife, the critically acclaimed, triumphantly discordant, Swordfishtrombones, was Waits's turning point and he has never stopped reinventing. To begin with, on a very basic level, Brennan opened him up to new influences, he suggests.

"Her record collection and her library were both impressive compared to mine. When I met her most of my records were kind of stuck together with cheese and hair and oil and stuff. She had hers not only still in the cases but still in the little paper sleeve too. That in itself was something of a revelation."

Since then, they have shared pretty much everything, ("She washes, I dry," is how Waits describes their songwriting technique). The family firm has been extended by the presence of their middle son, Casey, playing drums on the last couple of albums. In one maudlin lament on the new record, Waits sings in time-honoured fashion: "I gave it all up for the stage." The great thing about this heartfelt sob, though, is that it is delivered in the full knowledge that he successfully avoided the fate he describes. Has contentment ever felt an enemy to his creativity?

He pauses, laughs. "Right when you asked that the jukebox played 'I'd like to settle down but they won't let me.' Merle Haggard. It's always a bit like that. People don't want you to be satisfied if you are a musician. And the scope for songs isn't that great. Most songs that aren't jump-rope songs, or lullabies, are cautionary tales or goodbye songs and road songs…"

Given the insistently improvised and homemade texture of his music, it is tempting to think Waits spends a good deal of his time out here in the sticks tinkering with farm machinery in his yard or fixing his truck. One inspiration for his music is Harry Partch, an American pioneer of the 40s and 50s, who not only invented his own instruments but created his own notation too. Waits's music still allows the possibility of "hitting a cabinet hard with a two by four" just for the hell of it. He distrusts digitalisation.

"Music has generally involved a lot of awkward contraptions, a certain amount of heavy lifting," he says. "The idea that it will just be a sort of vapour that you listen to out of speakers the size of a dime alarms me. It's like injecting yourself. Or eating alone."

He is, he says, equally wary of the ease of search and shuffle. "They have removed the struggle to find anything. And therefore there is no genuine sense of discovery. Struggle is the first thing we know getting along the birth canal, out in the world. It's pretty basic. Book store owners and record store owners used to be oracles, in that way; you'd go in this dusty old place and they might point you toward something that would change your life. All that's gone."

Does he ever stray online?

"No," he says. "But then I'm one of those guys that is still a bit afraid of the telephone, its implications for conversation. I still wonder if the jukebox might be the death of live music."

If he feels out of kilter, he says, it's maybe in his genes. He was leafing through a musical dictionary the other day and he came across a definition for "waits" – "They were these guys who went door to door, carolling, bringing you the news, telling you the time. I was strangely reassured by that."

He takes comfort, too, in the fact that the future is not uniformly distributed. "If I want to walk out in the desert and heat up a can of beans on a fire, I still can. In those movies like Gattaca or whatever, the space age stuff is always all there is. But in the world there is never just one way of living. It's more like a big junkyard. Put it this way: I'm not afraid I'm going to end up on a space station in aluminium-foil underwear."

Music itself is the closest he gets to time travel. He gestures toward the jukebox: "The studio is torn down, all the people who played on it are dead, the instruments have been sold off. But you are listening to a moment that happened in time 60 years ago and you are hearing it just as sharp as when it was made. That remains an amazing thing to me."

Photographs have the same ghostly quality for him. He pulls out a little camera to prove the point. "You used to be able to see the pictures of America's 10 most wanted men in the post office, but they stopped that. I keep them on here, instead." He flips the pictures of fugitive murderers and terrorists. "Just in case, you know."

Does he wake up paranoid or grateful?

He gathers up his crow book, finishes his coffee, pockets his song ideas. "I don't know. As Bob Dylan says, 'Fear and Hope: always sounds like a comedy team to me…'"