Bryan Saunders has created 50 self-portraits under the influence of everything from Valium to lighter fluid. Does that make him an outsider artist, or a troubled man in need of help?

Johnson City, Tennessee, is a long way from bustling, well-to-do Nashville, a five-hour trip by car. The man I've come to visit, Bryan Saunders, lives on the fourth floor of a housing project called the John Sevier Centre. It was once a fancy hotel, back in the 1920s, then a home for the elderly. Sixteen people died in a fire here on Christmas Eve 1989. Nowadays it's so insalubrious that each apartment is fitted with a loudspeaker. Every few hours they burst into life: "In five minutes we will be testing the fire alarm system…"

"Jesus!" I say the first time it happens. "It's like being in prison."

"It's intrusive," Bryan smiles. He's happy to see me startled. "They say stuff all day sometimes."

"Who lives here?" I ask.

"Mentally and physically disabled veterans," he says, "unemployed people. You can live here for $18 a month."

"Can it be a dangerous building?" I ask him.

"Not from fire any more," he says. "Nowadays they have sprinklers everywhere. If someone burns their toast, there's three fire trucks here in 10 minutes. But people die here and it's mysterious. The police don't advertise why and it's just creepy. That's why I picked an apartment that's not so high up. I can throw my sketchbooks out of the window if I have to."

"How many journalists have come to interview you here over the years?" I ask.

"You're the first," Bryan says.

"The first ever?" I say.

"Yeah," Bryan says. He looks a little melancholy. "Nobody," he says.

Bryan is an artist. For the past 17 years he's been sitting in this room – or somewhere like it – drawing a self-portrait or two every day. "I've done 8,700," he says. "Every day is different. Like snowflakes and DNA and fingerprints, no two are the same."

The thing is, 50 of these 8,700 self-portraits have lately become very famous – celebrated all over the world, with millions of Google hits and a forthcoming exhibition alongside Damien Hirst at the influential Maison Rouge gallery in Paris. They're the 50 he drew while he was on drugs. Each was created under the influence of a different substance, from marijuana and cocaine through lighter fluid and "bath salts" – "They're what everybody says are causing people to eat each other's faces" – to prescription pills with names like Cephalexin and Risperdal. In fact, most of the 50 were prescription pharmaceuticals. "That's the popular thing today," Bryan says. He says he hates drugs but feels obliged to try new ones, "just for the drawing".

Bryan Saunders' self-portraits: 'Drugs make you look ugly.'

Downstairs in the communal living room, a man in a tie-dye T-shirt rocks frantically on a battered leather chair. I saw him as I walked in and I'll see him doing the same as we leave for lunch a few hours later. In the elevator up to Bryan's apartment a boyfriend and girlfriend were staring outwards, their eyes glazed. I ask Bryan if his neighbours know about the drug project. "They're who give me all the drugs," he says.

"Oh, really?" I say. "Where do they get them?"

"Doctors," he shrugs. "People on the streets."

"Do they sit with you while you take them?" I ask.

"No, no, no," he laughs. "I don't even know who they are."

He knows almost nobody, he says. He'll categorise as a friend a man he goes to the movies with once a year. Instead, he spends his life in his room, drawing his self-portraits. "I'd rather be working on art projects than wasting my time standing around telling jokes and drinking beer and stuff."

So the Trazodone, for instance, came from a neighbour – a Gulf war veteran who had been prescribed them in an attempt to get him off Valium. Bryan hated Trazodone. In the sketch, his face is crumpled and his body transparent. His internal organs look like doll faces. "Trazodone gave me this real intense sensitivity to light," he says. "I turned all the lights off. But being in the dark was terrifying. I ended up calling the hospital. The woman said, 'Don't take medicine from strangers.'"

His only regular visitor is his girlfriend, Nicole. He says he'll introduce me to her before I drive back to Nashville.

"If you don't know your neighbours," I ask, "how do you get drugs from them?"

"They hear I'm doing the project," he explains. "They knock on the door and say, 'Here's some lighter fluid.' Always for free."

"That's nice of them," I say.

"I've never paid for any drug," says Bryan. "I would never pay."

But there's another way Bryan acquires his drugs – a somewhat more disturbing method, as I'll learn later.

Bryan Saunders: 'I would rather live here in squalor for next to nothing. It gives me all kinds of freedom’ Photograph: Troy Stains for the Guardian

Bryan's room is small and square. On the wall is a half-finished commission: an acquaintance is paying him $100 to paint his face on the body of a condor flying above two rabbits having sex while wearing ties. "He says every time he sees a guy in a tie, they're always rude to him, so he wanted me to make a painting for him where he was some kind of bird, and he wanted me to emasculate the people wearing the ties."

There are other paintings, including two by a friend called Don Morgan. One is an intricately pastel-shaded drawing of a beautiful woman. The other is a black scribble so intense you can no longer see the white paper. "He did this one," Bryan says, pointing at the pastel drawing, "and then he shot himself in the head and survived and did the other one. It's the first thing he drew after he came out of the coma. He's in a nursing home now and he has permanent confusion."

We leaf through Bryan's drug sketchbook. The drawings are beautiful and compelling. "That's lighter fluid," he says. "I used metallic crayons to give it that lighter fluid feeling. That's computer duster. If you don't do it right, it can freeze and shatter your teeth. That's crystal meth. That painting's probably the seventh or eighth on the same piece of paper, because I kept doing it over and over again. I couldn't stop. That's marijuana…"

"Who is the little boy sticking out of your head in the marijuana one?" I ask.

"That's me," he says. "I was thinking back to when I was a child."

Bryan grew up in Washington DC. His mother was "most proper", but she married a member of a biker gang called the Pagans. "My dad was a bad person. One of his friends was found guilty of having seven bodies buried in his back garden."

Bryan barely knew his father. He'd vanish for years at a time. "He gave me a horse once," he says. "He said, 'This horse is yours. Have this horse as a gift.' So I fed the horse every day." Bryan pauses. "It wasn't his horse. He gave me somebody else's horse. He was a terrible man."

The anti-anxiety pills seem to have been Bryan's most pleasant drug experiences, judging by the primary colours that bathe him in those drawings, the childlike crayon outlines, their babyish innocence. "I liked the Valium the most," he nods. "It made me feel sociable, down to earth. Real earthy. Real comfortable in my own skin. I could say hi to people I didn't know." The same goes for the Xanax, which has him looking dreamy and wistful and wearing a My Little Pony T-shirt.

"I drew that one in mental hospital," he says.

"How did you end up there?"

"Someone was drugging girls in my local bar," he explains. "They were putting something in the beer. One girl woke up in a church parking lot. She didn't know if she'd been raped or not. I was going to burn the place down. So instead of getting violent and going crazy, I checked myself into a mental hospital and they got me all doped up."

We turn to the next one. "Whoa," I say. This one could not be less Xanax-like. The drawing is spindly and paranoid, and the page is patterned with real-life bullet holes. They pepper Bryan's stomach and neck. I ask Bryan how they got there and he explains that he used a gun borrowed from a friend. He propped up the page from the sketchbook and repeatedly shot it. "I remember bouncing into the walls like a fly going bong, bong, bong," he says. The drug that elicited this reaction was called Geodon.

"Geodon?" I say.

Bryan Googles it. "It's for symptoms of schizophrenia," he reads, "so it's an anti-psychotic agent, I guess."

"Did you get it from somebody with schizophrenia?" I ask.

"No, I got it from a doctor," Bryan says. And this is when Bryan tells me the other way he acquires many of his drugs. He sometimes visits psychiatrists, tells them about the art project, and asks them for "samples of some pain pill or sedative I've never tried. I say, 'Can you write me a prescription for just one so I can do my drawing?' And I take my book with me and show them my art project. And they always give me some crazy, crazy anti-psychotic pill instead."

"They never give you what you ask for?"

"Never,. Always something way worse."

"Even though they know you aren't psychotic," I say, confused.

"They think I am," he says.

"And are you?" I say.

"I don't think I am," he says.

The worst anti-psychotic of all, he says, was called Seroquil. The psychiatrist prescribed him 100. "I was sitting in front of that mirror waiting for it to kick in, so I could draw myself, and all of a sudden I heard this inner voice saying, 'Don't look in the mirror.' So I looked down at my book. 'Don't look at your book.' I thought, maybe I need to lie down. So I looked over at my bed and the voice said, 'Don't look at your bed.' "

The Seroquil drawing is a simple pencil one, but Bryan's forehead is bulging as if it can't contain his engorged brain, and his mouth is clenched like that of a man in battle. "For two hours I was fighting it," says Bryan. "It scared me to death."

'I’d rather be working on art projects than wasting my time standing around telling jokes and drinking beer.' Photograph: Troy Stains

The next drawing is, by contrast, lovely. In fact, it's the most famous of all his drug self-portraits – the one for psilocybin mushrooms. His grin is goofy. Tiny, pretty bubbles emanate from his head and beard. It looks like how everybody wishes drugs could make them feel, although they so rarely do. "Everyone wants to buy this one," Bryan says. "Every week someone emails, 'How much for this one?' "

"And what do you tell them?"

"That it's not for sale. I don't even respond to those people. They think they can buy everything."

Bryan is broke. He's so broke he qualifies for a free mobile phone. He's so broke, he says, he can't even afford to change the outgoing message, recorded by the previous owner: "Hi, leave a message for Jason Bishop…"

"It costs money to change the message," he says.

"Have you no interest in making money?" I ask. "You could be like Robert Crumb. You could be a rich man."

Bryan says he has no idea how to achieve that. "People want everything to be free these days," he says. "It isn't the 60s any more."

He's had jobs. He's been a janitor and an electrician and a private investigator. He says he was terrible at the latter. The people he followed – adulterous husbands, usually – would forever spot him hiding in bushes. They'd stalk over to ask him why he was secretly filming them. After that, he became a ditch digger.

"I dug a lot of ditches," he says. "I liked that job."

"It gets you out," I say.

Bryan looks around his room. "Yeah," he says.

And throughout all this were the art projects. The drug pictures are just one of many. It makes him rueful that nobody cares much about the others. Nobody wants to know about, say, the 28 days during which he blocked up his ears and attached a copper funnel to his mouth "in an attempt to hear the world through the inside of my mouth". Nobody wants to know about the thousands of discarded photographs he's spent years retrieving from bins around Johnson City. He points to a portrait of a sweet little girl. "Her family threw all her possessions out because she went to jail," he says.

Bryan has been in jail, too. He served eight months for "a knife fight with some punk rockers". This surprises me. He seems so nerdy and introverted, I can't imagine him knife-fighting anyone.

We go downstairs for lunch. The elevator is filled with spaced-out and damaged-looking people. One man murmurs a hello to Bryan. He smiles politely and looks away. When we get outside, he tells me about the last time someone died in the building. "It was four or five months ago. I was asleep and I heard all these police radios. I opened my door. The door across the hall was open and a woman was lying dead on the floor."

"What did she die of?" I ask.

Bryan says he doesn't know. "They used to do autopsies, but they don't care any more."

It was probably drugs, though.

It makes him miserable that his drug self-portraits have become the famous ones. He feels typecast by them and resentful – as if his whole life has been reduced to these 50 drawings. I tell him he should feel proud of them because they're so interesting and unique. But he gives me plenty of reasons as to why their popularity is a bad thing. For a start, he says, Europeans like them for condescending reasons: "You like to see America's problems. Some guy on drugs or some obese woman undressing goes down well in France or Germany."

Then there's the misplaced admiration: "People write to me thinking I'm a Hunter S Thompson. They'll say, 'Oh, you opened up new doors for me.' I tell them, 'If you look at my portraits, what do you see? Drugs make you look ugly. Drugs are not flattering.' "

Then there are the trolls: "I get so much hate mail," he says, lighting a cigarette on the street corner. " 'I'm going to put you down like an animal, you and your sad weak chin having to hide behind a beard.' That stuff freaks me out."

"How many of those have you got?" I ask.

"About 20," he says. "I quit smoking, but I started again when the drug pictures went viral."

We have lunch in a soul food cafe on the corner. Like Bryan, Johnson City is having money troubles. It infuriates him that they're trying to attract tourists by renaming bars the Al Capone. "There's no proof whatsoever that Al Capone was ever in Johnson City." The cafe was once owned by a group of transvestites, Bryan says. "They sold coupons to everyone in the John Sevier. They said for $60 you could eat every day for free. As soon as they got everyone's money they loaded up all the mayonnaise and took off in their skirts and high heels. Ha ha ha!"

After lunch, he takes me to meet his girlfriend, Nicole. She works in a seamstress shop on the main street. The shop, and the street, look untouched since the 1950s. I think I was expecting someone odder, but Nicole seems utterly lovely and normal, standing behind the counter, surrounded by brightly coloured thread. She looks delighted that, after all these years, a journalist has finally travelled to meet Bryan.

"He could be a big star if…" I say.

"If I wasn't so intense," Bryan says.

"Maybe it'll happen to you after you're dead," I say.

"How did you like the John Sevier?" Nicole asks.

"I found it frightening," I say.

"I go over there!" she says. "Can you believe that? There used to be a guy who followed me around really fast. He raced. One time, a guy was running around with a butcher knife. The police were chasing him for three hours."

"We were all quarantined downstairs in case he stabbed us," Bryan says. "Finally, a lady came over the loudspeaker: 'Daryl, will you please come down to the office.' Soon as he heard that, he calmed down and turned himself in."

"The power of the loudspeaker," I say.

"But I like it there," Bryan says. "It's my favourite place I've ever lived. People I know from Johnson City think I'm insane for living there and liking it. But if I lived in New York I'd have to pay $1,000 or $2,000 dollars a month, I'd need a job that would take all my energy, I'd be forced to socialise, to go to all those art openings, and all that junk, and I would never get any of my art projects done. So I would rather live here in squalor for next to nothing. It gives me all kinds of freedom."

Nicole's colleagues evidently aren't aware of the content of Bryan's self-portraits, so I have to be careful what I say.

"What do you think of his… art?" I ask her.

"When I first met him I couldn't work out if he was really creepy or really smart," she replies. Then she pauses and says, "I still can't."