For plenty of artists, getting that seal of approval from a megastar is the make-or-break moment; the all-important co-sign that'll put you in the lap of the gods, and introduce you to a previously unconquered demographic. But for Billy Childish, who's been namechecked by everyone from Jack White to Kylie Minogue, it's not praise he has any interest in reciprocating. "When I've met some of these people, I don't live up to their expectations because I'm not a fan of their work," he says. "I mean, I've met Jack a few times and he seems like a very nice chap, but he was upset that I didn't like his music. Millions of people think he's great, I'm very happy for him, but he doesn't need one more fan. I've got things to be getting on with."

And get on with it he does. Over the last 35 years, Childish has produced a catalogue of work that encompasses music, art, poetry and prose, yet almost never strays from his favoured formulae. He's released music at an astonishing rate (it would take a more patient man than me to count all his albums and EPs, but four albums in 2013 alone will give you some idea) yet it all stays within the narrow garage rock genre. His art plants itself firmly in the deeply unfashionable genre of abstract figurative painting and his written work is published unedited, despite the fact that Childish is dyslexic. I ask him if the range of his work is designed to create a kind of exclusivity for his fans, like being part of a secret club. "Maybe," he smiles, "but I don't know how many members it would have."

Childish refers to himself as a "radical traditionalist" and appreciates the methodology of old (he won't play live any more because he doesn't like modern PA systems). "Tradition is the platform to freedom," he says quietly. "It's often seen as enslaving, but it's actually the way to go. The problem with fashion-based stuff, is that it's… kind of a blancmange." A striking 54-year-old in dirty overalls, he paints every Monday in his studio, located in the same Chatham dockyards where he first worked as a 16-year-old apprentice stone mason. He's lived and worked in the picturesque yet brutal town all his life (he says a bit of gentrification is OK, "because it means you get good sausages"). There seems to be a kind of parallel between his work, which sits just outside of London's cultural scene, and Chatham's geographical placing, also just outside the capital. "I never needed to validate myself by moving to London," he says. "In my mind, it's actually a very provincial place because it's full of people from the provinces trying not to seem provincial. I always found it very limiting."

The legend of the man born Steven Hamper is as of much interest to his followers as his work. He was expelled from St Martins, he became an alcoholic, he beat up his own father, he co-founded the stuckism movement (which took a tongue-in-cheek stand against conceptual art), he's made the same album for 35 years and he looks like a boozy 40s cad straight out of a Patrick Hamilton novel. I ask him about the famous moustache, now so prevalent in certain districts of the London he's never musch cared for. "I'm an interesting person with a moustache, rather than a person who's grown a moustache to be interesting. But it's a good trick to do that, and good luck to them if they can pull it off."

Childish's fans see him as the last artist out there with any integrity, a Medway renaissance man who you'll never see on the front of Mojo. But just looking around his studio, you realise a strange paradox: you are in the presence of a man who is endlessly creative and somewhat subversive, yet only seems to want to create the same thing over and over. A gentleman monomaniac, if you will. He's fronted a number of bands down the years – the Pop Rivets, the Milkshakes, Thee Mighty Caesars, Thee Headcoats, the Buff Medways, etc – but I put it to him that he might as well have called all of his bands "Billy Childish" as the form differs so little. He is cheekily defensive. "Well that's not true, because we just did four albums where I played bass and my pal sang and played guitar."

I wonder if Childish's output is, in the best possible way, secondary to the man himself and his story. Perhaps Steven Hamper from Chatham, as much as he would hate the suggestion, is a kind of performance artist, the main feature of a 35-year-long conceptual satire of the London art world. But then again, tenacity alone suggests this is no send-up. And if so, he is surely the ultimate outsider: not just an artist who is ignored by the mainstream, or one who sticks two fingers up at it, but one who respectfully says it's not for him. A radical traditionalist, I guess.

I ask if he feels like an outsider. "It's an interesting one, this outsider term," he says. "Because I feel like an outsider, but I consider my work mainstream. I think the mainstream are the outsiders and I'm the way it should be." I'm not sure I entirely agree, but listening to Childish tell me how he paints twice a week, writes books in the morning and makes music in the evening, you can't help but think he's got it all figured out. Without ever having to be in a red-carpet selfie.