In the late 1980s, when AM Homes was studying at the renowned Iowa Writers' Workshop, finishing a series of short stories, she kept a Barbie doll in her apartment. She thought of it as a particularly innocent-looking Barbie, dressed in white, aimed specifically at younger children. There was a note on the box that declared: "'Now with bigger buttons!'" she says, "so it was easier for girls to undress, for little kids who don't have good hand control." It sat on a counter, and she started imagining a story around it, something innocent, to fit the doll's image. Yet everyone who came to visit her almost immediately stripped Barbie's clothes off. Homes felt a little confused and affronted by this. "Then they would tell me strange things. It became like a Barbie therapy session." She ended up writing a story about a teenage boy who feels attracted to his sister's Barbie, a doll that is a walking, talking, sentient being, albeit only 11.5 inches tall. He drugs the doll with Valium, and has sex with her. Then, one day, when Ken, Barbie's boyfriend, has had his head removed, the boy ejaculates into the hole. This makes him wonder if he might be gay, which would seem the least of his worries.

Now in her early 50s, Homes has spent the last 30 years chronicling the surreal undercurrents of everyday life, the darkness behind closed doors, the absurdity beneath the perky, affluent, suburban version of the American dream. What she's interested in, she says, is "the gap between who people are publicly and privately ... What I'm doing, which sometimes makes people uncomfortable, is saying the things we don't want to say out loud."

That's as true as ever of her most recent novel, May We Be Forgiven, which includes, in the first few pages, a man imagining his brother and brother's wife having sex, and thereafter charts a path of destruction and severely hard-won redemption. This week it won the Women's prize for fiction (formerly the Orange, soon to be the Baileys), in a year with a glitteringly strong shortlist, with nominees including Zadie Smith, Kate Atkinson, Barbara Kingsolver and the woman who has scooped most of the country's other major literary awards, Hilary Mantel.

AM Homes beat a glitteringly strong shortlist including Hilary Mantel. Photograph: Linda Nylind for the Guardian

When I catch up with Homes, the morning after the awards – she was up until 2am, awoke at 4am, but still rushes breakneck through ideas – she says she had no expectation of winning. She wrote a few notes for a speech "in a grudging way, thinking, God, why am I even doing this?" Did it feel good to beat Mantel? "No. I mean, come on! To win those three in a row [Booker, Costa, Women's prize] would have been incredible for her, and hearing her read the night before, I just thought, my God, I could listen to this for ever." Homes has a reputation for transgression, yet she is also a girl scout leader with a young daughter, and a highly developed moral sense. She writes about secrets and incongruities, yet comes across as bracingly forthright.

She started the book about seven years ago, as the result of fellow nominee, Smith, asking her to contribute to a book of short stories. Homes started one that she didn't have time to finish back then, but later picked it up and just kept on writing. The novel's short-story roots are clear in the pace of its opening pages, which take in a Thanksgiving meal, the turkey-greased start of an affair, a fatal car crash and domestic violence. We're introduced to the central character, Harry, who is a hapless, helpless Nixon scholar, along with his brother, George, an incredibly angry, successful executive, who has changed the face of TV with the shows Your Life Sucks and Refrigerator Wars. By page nine, George is in a psychiatric ward, and when Harry ponders whether this is the right place for him, George's wife Jane replies: "It's the suburbs. How dangerous could a suburban psych ward be?"

It took the Women's prize judges four hours to settle on May We Be Forgiven, but it's easy to see why. The chair, Miranda Richardson, praised its darkness and wit, saying it had made them all laugh out loud on the train. The novel is a mix of the smallest, funniest observations, essaying the many ways people succeed in saying exactly the wrong thing at exactly the wrong time, while also exploring sweeping themes, including the growth of technology, the rise of China, the dangers of individualism and instant gratification.

The book has been called a contender for that mythic construct, the great American novel. Is that what she set out to write? Homes says she's thought about this, and did once look up the way that that phrase was first intended. "It was not meant as a critical judgment – 'Oh, this is a great American novel' – but great as in expansive, and far reaching, and so in that sense, absolutely, very much, I think of myself, in positive and negative ways, as a very American writer." Her fellow citizens don't always appreciate the lens she turns on them. Her six novels have tended to receive highly divided reviews, with some notable stinkers from the New York Times's most famous critic, Michiko Kakutani.

"In America, people get confused and sort of pissed off with me, because I think they feel like they're being criticised," she says. "And I think, from other countries, with a little bit of distance, it's a clearer view."

A key paragraph in May We Be Forgiven addresses the growth of technology, stating that "there is a world out there, so new, so random and dissociated that it puts us all in danger". It's a world where kids play on computer games while travelling to their mother's funeral, where our protagonist contacts women on the internet for sex – one of whom turns out, in fact, not to be a woman, but a young brother and sister who feel abandoned by their parents, and who summon Harry through subterfuge and handcuff him. "Basically, our life sucks," says the boy, "Our parents pay no attention to us, Dad works all the time, Mom's entirely electronic."

Homes says she usually avoids references to technology, because it dates the writing, but this time she wanted to chart how it's changing us. "I think what's interesting about the internet is that people now present themselves in chatrooms – and not always dating or sex sites – as different versions of themselves. There is a fracturing of identity, and the question becomes: 'Well, who am I really? When I go out into the world and meet somebody I've met on this site, and present myself in a different way, which version of myself am I?' I think there's a lot of dissonance in there, in how we describe ourselves to ourselves."

The use of technology arguably adds to Harry's mid-life meltdown – he's been described as a Job character, swiftly losing his wife, job, house and credit card – but also helps him form and maintain the relationships that save him. May We Be Forgiven has been accused of sentimentality, and there's no doubt that, as in her previous novel, This Book Will Save Your Life, she's interested in the ways human contact can mend her fractured male protagonists, an interest that sits uneasily beside her satire.

The tone of her novels changed after 9/11, she says, an event she witnessed and recorded on camera from her window in downtown Manhattan. That was the first time "that I felt the world around me was unstable, and that it was difficult to then ... go into my imagination and make something else. I was also thinking a lot about our responsibilities to each other, and how to be optimistic at a time that's not optimistic. That really prompted me, with both of these two novels, to try, as organically as possible, to figure out what would need to happen for these characters to feel some sense of hope by the end, as opposed to what I think is the traditional end of a novel: death."

Homes grew up in Washington DC, the adopted daughter of an artist father, and a mother who worked as a school guidance counsellor. She was contacted by her birth mother in her early 30s, and wrote about the experience of meeting her and her birth father in the 2007 memoir The Mistress's Daughter. She was no less stringent with herself in that work than she is with her characters; when she met her birth father, she wrote, she imagined having sex with him. Overall, the reunion didn't go that well – the more she spoke to her birth mother, the happier she was she'd been given up – but she is glad it happened.

Did being adopted create a space, as a child, to fill with stories about her roots? She was in no doubt about who her parents were. "It was Jack Kerouac and Susan Sontag, and it was a one-night thing. Of course! I've had interesting talks with Edward Albee, the playwright, who's also adopted, about the sense of distance, the dissonance, of being in a family that you're not related to. Plenty of people say 'I don't fit in in my family,' but the true biological dissonance [of adoption] is a very real and palpable thing, but incomprehensible to those that haven't felt it. It's like being George Washington, but being told you're Abraham Lincoln."

She didn't enjoy school. In her teens, when a teacher told her she had an attitude problem, she refused to go back for a year, and stayed at home writing a book of poetry she called An Introduction to Death, with Excerpts from Life. While studying at Sarah Lawrence college in New York State, aged 19, she wrote her first novel, Jack, and a play that was partly about JD Salinger. The notoriously private author was not happy, and threatened to sue her. "Can you imagine how you'd feel? Your hero says: 'You're in trouble now!' My hair started falling out. I was a wreck. And we changed the name of Catcher in the Rye to Life in the Outfield in the hope that would fix it. It was performed, and I was so shy I couldn't go to the opening. I spent the night driving around Washington DC in circles."

Salinger dropped the threat, and while this experience – "stressful beyond belief" – might have made some authors play it safe, Homes has always chosen to take the most difficult path. Her 1996 novel, The End of Alice, which focuses on a paedophile in prison, and his correspondence with a woman who also has a sexual interest in children, was described as "repugnant" by the NSPCC, which called for a boycott. WHSmith refused to stock it. I ask if this shocked her, and she says the biggest surprise was that the book didn't receive a similar response in her country. Instead, in the US, "there was a very passionate range of reviews, from positive to wildly negative, and I remember the publisher saying, 'What do we do?' And I said, use them together, because that's the thing. It is not meant as a book that you [love]. When someone says to me 'I love that book,' I'm thinking: 'That's worrisome.'" Artistically and intellectually, it's the hardest story she's ever written, but she wanted to push people to talk about the subject. "Until we're willing to deal with it, it will continue to happen."

Homes is private about some aspects of her life. Her friend Jeanette Winterson has written that Homes "is gay but describes herself more honestly as bisexual," and she's understandably protective of her daughter, Juliet. But she is willing to go further than almost anyone in exploring the human psyche. Is there anything she's wanted to write, that she's thought was going too far? "I think I've taken care of that already!" she says. "When people say they're shocked by something, I think it just means I hit a nerve."