“People constantly compare me to Diane Arbus,” says Roger Ballen, wearily. “But I think Samuel Beckett is the key influence on my work. My photographs evoke the absurdity of the human condition, but they are also records of a personal psychological journey. For me, photography is a way of looking in the mirror.”

If all portraits are to some degree self-portraits, the photographs Ballen has made over the last 15 years would appear to be evidence of an intensely imaginative but deeply troubled psyche. An American who has lived in Johannesburg since the early 1980s, where he initially worked as a geologist, Ballen is an artistic outsider whose subjects tend to be actual outsiders: South Africa’s poor, many of whom are disabled or mentally unstable. But, where once he went looking for his mainly white subjects in the outlying “dorps” or small villages, shooting them against the harsh landscape in which they lived, he now creates ever more elaborate sets in the derelict houses of South African cities in which the poor and damaged have been exiled.

The results are never less than intriguing: ornate, visceral, meticulously created and often nightmarish. Certain symbols recur: animals, makeshift sculptures, masks, dolls, blankets, scrawls and primitive paintings. Someone, somewhere, is probably writing an academic paper on the symbolism of snaking cables and wires in his work. Ballen can spend years on a single series. “I try to remove anything extraneous,” he says, “so that every single element adds up to the whole. I’m a believer in simple form and complex meaning. My images have presence. They tend to stay with you long after you have seen them.”

Phaidon is just about to publish an expanded edition of Ballen’s third book, Outland, which originally appeared to much critical acclaim in 2001. In it, he shed the stark documentary style of his earlier works, Dorps: Small Towns of South Africa and Platteland: Images from Rural South Africa. Outland was the pivotal moment in Ballen’s journey from photographer to artist: the even more startling images that appeared in subsequent books (Shadow Chamber in 2005, Boarding House in 2009 and Asylum of the Birds last year, each as eerily evocative as their titles suggest) amount to one of the most instantly recognisable visual signatures in contemporary photography. “The images I have made since about 1997 are much more metaphorical,” he says. “That is the essential difference.”

Ballen still shoots on film and in black and white, the relentless monochrome accentuating the difference between the viscerally real (the strange faces, bodies and poses of the people he finds) and the fabricated (the elaborate settings). Everything in Ballenworld is out-of-kilter, makeshift, shadowy and unreal, including the human beings: an old man in a metal tank in a derelict room; a disturbed-looking man cradling a boar in his arms; an emaciated figure lying prone on a bed below a curling electrical cable and a bare lightbulb.

Why is he so drawn to the extreme and the grotesque? “I really don’t see my work in that way,” he says. “I don’t really believe in traditional definitions of beauty. In my brain, for instance, what we would call traditionally beautiful people have their own disturbance. As a photographer, you see the mask in front of everyone’s face. You have to go beyond that. With me, it’s a case of being compelled to journey deeper psychologically in order to make art.”

Facebook Twitter Pinterest Captor and captured, 1997

Alongside Beckett, Ballen cites Carl Jung and the radical 1960s psychoanalyst RD Laing, author of The Divided Self, as touchstones for these journeys. “Jung’s idea of the shadow self is in there, for sure,” he says. “The darkness in all of us that we suppress. I often think that when people react to my pictures, the darkness they see is a reflection of their own repression.”

Having once struggled to sell his documentary photographs, Ballen is now represented by the art powerhouse Gagosian. In 2012, he went further “overground” in dramatic fashion when the South African rap-rave group Die Antwoord persuaded him to direct the characteristically disturbing video for their single I Fink U Freeky. It was a viral hit, receiving over 30m hits on YouTube. Yolandi Visser, the band’s singer, described Ballen as “the weirdest person I have ever met in my life”.

One aspect of that weirdness may be the way Ballen represents humans in his work – as either actors or props. It has always intrigued me that, while Ballen’s early documentary work caused a storm of negative criticism in his adopted homeland for its stark depiction of the plight of marginalised poor whites, his more constructed images have received near-universal praise from critics and academics alike. It seems almost quaintly old-fashioned to suggest that the metaphorical aspect of his work cannot mask a distinct lack of empathy for the marginalised souls whose perceived freakishness is, more often than not, exaggerated by the theatricality of the settings and their abject or plaintive poses.

Is Ballen reflecting, however extremely, a broken society? Or indulging in a kind of elaborately staged artistic voyeurism? Or is it neither? Is he simply following his own psychological journey into a constructed shadowland where empathy or indeed humanity have no place? “My critics have no idea about the reality of what I do or of the relationship I have with my subjects,” he says. “The fact is I have a deep relationship with these people. I would go as far as to say, without wanting to exaggerate, that they love me. And for me to make these photographs, I have to look deep within myself and ask, ‘Can I live with myself?’ I can.”

Facebook Twitter Pinterest Brian with pet pig, 1998

Even if Ballen’s essential subject is, in fact, himself and the darkest recess of his psyche, the problems of representation – voyeurism, vicariousness, objectification of the other – persist, not least because he is using the poor, the mentally ill, the marginalised as characters, or metaphors, in his ongoing artistic psychodrama. His work certainly echoes Arbus’s: it makes us, as viewers, collude as fascinated or appalled onlookers. As with Arbus, that’s a crucial part of its unsettling power.

Increasingly, though, humans are disappearing from Ballen’s photographs, as they become ever more sculptural and painterly. In Asylum of the Birds, humans have given way to masks, mannequins and dolls. “From about 2002,” he says, “faces started to disappear, and drawings entered the photographs. The work became even more abstract and more to do with theatre and installation, with symbols and signs.” He is currently working on two new projects, one featuring rats and mice, the other “abstract, almost Rorschach drawings”.

Just as Beckett moved towards total silence, is Ballen moving towards complete abstraction? “Maybe,” he says. “I am always following my instinct and trying to deal with what is real and what is imagined. I go where the journey takes me. One thing is important in this: the photography is way ahead of my conscious mind. It can be years before I figure out what a series is actually about.” He sighs, as if weary of all these questions and explanations. “You always know inside when an individual work is strong though,” he says. “When a picture has real power, it feels like it might walk right off the wall.”

• Outland by Roger Ballen is published today.