An attractive middle-aged woman, naked bar a pair of red high heels, poses as if for a porn shoot, her legs spread wide and a defiant look in her eyes. In another photograph, the woman is naked and laughing, entangled with a younger male lover. Elsewhere, in furtive black-and-white shots, she is pictured having sex with various – mostly younger – men. These are some of the more provocative photographs from Leigh Ledare's series Pretend You're Actually Alive – and, even without knowing their full context, they are not for the easily offended.

What shifts them from the provocative to the shocking is the single word in many of the captions: mum. We may have grown used to photographers like Nan Goldin and Larry Clark mapping out their often hardcore personal lives. But Ledare, a soft-spoken American, has taken this to a new extreme, capturing the exhibitionist sexuality of his own mother.

The work, which I came across when it was published in book form in 2008, comprises images of his mother Tina; his own adolescent diary entries; ads she placed in the Seattle Weekly for "a generous, wealthy husband (not someone else's) who wants his own private dancer"; plus moving descriptions of her troubled relationship with her sons and former husband; and, most intriguingly, ageing.

Although initially taken aback, I soon became fascinated by Ledare's no-holds-barred delineation of an extreme mother-son relationship. Goldin curated a show of Ledare's work at the 2009 Arles photography festival, which was met with shock. Now, extracts from Pretend You're Actually Alive will feature in a group show – Home Truths: Photography, Motherhood and Identity – at the Photographers' Gallery in London. The exhibition includes various series by female photographers that brutally depict the emotional and physical strain of motherhood. But even viewed alongside Tierney Gearon's disturbing images of the mother-child relationship, Ledare's photographs are by far the most transgressive. Their titles alone – Mom Spread with Lamp, Mom Fucking in Mirror – prompt a swift intake of breath, and their inclusion in the show will doubtless offend some visitors and appal others.

The obvious question is why did he – and she – do it? When asked, Ledare can retreat into a mixture of conceptual art speak, as in "the extremely open and intimate relationship I have with my mother ... was developed through the work. (It) comments on the confusion around these sexual boundaries … through imposing herself on me as a subject, she was asking me to be complicit in her sexualisation. I saw her sexuality as a means of antagonising her father and refuting expectations he had for how she should behave as a mother, daughter, and woman of her age."

A look at the arc of Tina Peterson's life explains things further. At 16, she appeared as a beautiful, precociously talented ballerina in Seventeen magazine. Later, she danced at the Joffrey and the New York City Ballet before working briefly as a model. She then married, had two children, and – when the marriage failed – worked on and off as an exotic dancer.

Born in 1976, Leigh left home at 15 and attended Rhode Island School of Design. He was a skateboarder of some repute and worked as an assistant to Larry Clark, the controversial photographer and film-maker. Clark, who in 1971 made the seminal confessional photobook Tulsa about his life as a heroin addict, and later the Harmony Korine-scripted film Kids, is clearly a huge influence on Ledare.

His decision to chronicle his troubled relationship with his mother, he says, started when he returned home one Christmas. "I arrived home not having seen her for a year and a half," he recalls. "She knew I was coming and opened the door naked." When Leigh walked in past the bedroom, "a young man, almost exactly my age, was sprawled out naked. He rolled over to see me, saying hello, before rolling back over and returning to sleep." Ledare interpreted this welcome as "her way of announcing to me what she was up to, at this period in her life – almost as though to say, 'Take it or leave it.' I had a camera and began making photos of her then. She was the catalyst."

Leigh Ledare, Mother and Catch 22, 2002 Photograph: © Leigh Ledare Courtesy of the artist and Pilar Corrias, London © Leigh Ledare Courtesy of the artist and Pilar Corrias, London Copyright Leigh Ledare

So began their collaboration which, it must be stressed, is performance-based in the manner of much conceptual photography. Tina is performing an exaggerated version of herself for her son's camera. "I do find it difficult that the model often doesn't get any credit for her participation," she once said. "She's at least 50% of the picture." If the camera makes her even more of an exhibitionist, it also allows her son to play the role of detached observer. The process offers them both a distance – morally, psychologically and creatively. But still, he is a son who photographs his mum exposing herself and having sex.

How did he feel about it? "The work spanned eight years," he says. "I moved between different feelings – uncomfortable, absurd, funny. She would present herself to me, and through me, in a very confrontational way." Did he feel she was controlling the situation, manipulating him? "My mom was a strong figure in my life. My grandfather retired to homeschool my brother and me, and gave me a number of tools that helped me step back from my relationship with her, such as discussing with us the work of [sociologist] Erving Goffman, who was a friend of his."

Ultimately, it is hard to write about Pretend You're Actually Alive without resorting to cod psychoanalysis or pretentious art speak. For me, the work has become more fascinating, and no less shocking, the more I have tried to get to grips with it. Lurking beneath the shock factor are dark subtexts, not least a brief reference to Tina's bad deeds: she once ran up debts of $50,000 on her son's credit cards, then persistently claimed the purchases were gifts from men who adored her. In certain photos, mother and son kiss and canoodle; in a teenage diary entry, he names her alongside models and film stars in a list of "girls I wanted to do".

Did he ever get turned on when taking the photographs? "I don't know what to say …" he laughs. "There are many ways to be excited. Towards a sexual object, towards direct honesty and openess. I think already in the background there were some foggy boundary issues. What people talk about as being Oedipal – there's a flirtation with that, but the boundaries were never actually crossed." Ledare sighs at the idea of the work being taken so literally. "There's a lot of emphasis on me being her son, but the work looks at archetypal relationships; also fantasy life and social conventionseverybody who looks at it brings their own understanding of their own relationships. We end up displacing what is culturally taboo. The work is trying to look at those blind spots."

Ledare says he became friends with the young men pictured in his mother's bed. "They all saw the work and approved it. But like any nuclear-family situation, in some ways they were on the periphery. They weren't freaked out: they understood the work in terms of the relationship between a man and a woman, which is all very acceptable. But it also shows my mother as a woman turning 50, still being very sexually active. That is a situation we tend not to share publicly, but something that resonates with the majority of real, lived experience."

If the photographs of his mother having sex are wilful provocation, as I suspect, there are other, more sombre images that broach her fear of ageing and death, including one in which she poses as a corpse.

There's something fascinating about the balance of power between the two – does he feel he exploited her at all, or is his mother in fact more radical than him? "I think our agendas are different. She's maybe more iconoclastic. There is a masochistic position that she was acting out, as a way of taking control, achieving an upper hand in the dynamic of the family. She was a ballet dancer so she is very aware of her body and performance. She was publicly doing what she was doing already. You know, she was working as a stripper, and very upfront about it."

Ledare has made some challenging work since Pretend You're Actually Alive, including a series called Double Bind, in which he intimately photographed his ex-wife, Meghan, in a country cabin over several days. Her new husband – photographer Adam Fedderley, who had magnanimously agreed to the project – then followed Ledare's request to photograph her later in the exact same setting. Fedderley subsequently gave the undeveloped photographs to Ledare, who made the series from both sets of shots.

But Double Bind never quite compares to the transgressive jolt of Pretend You're Actually Alive, its outright assault on propriety as well as its inherent bigger question: does the semblance of art make it acceptable to photograph your mother having sex?

So looking back, what impact did this unholy collaboration have on its willing participants? Ledare has no regrets. What about his mother? "She sees the importance in putting herself forward in a way that provokes people to think. She's an extremely brave and articulate person."

There is a revealing piece of text in Pretend You're Actually Alive. It reads: "One day I told my Mom jokingly, 'As long as you regard your life as fiction, in the very least you'll have some interesting experiences.' She replied, 'Finally somebody who understands me.'"