The first of the three fathers that Nicky Loutit remembers is Robert Kee. He had been a fighter pilot and prisoner of war, and later became a writer, broadcaster and member of the “famous five” who founded TV-am. He was handsome, charismatic and angry.

Aged two-and-a-half, Nicky was put in a children’s home by her mother, Janetta, who is now 95. “She said to me recently that she knew I didn’t like that home, but she was so mad about Robert that she put me there for my own protection,” says Nicky. “She felt she was saving me. I remember crying and crying and crying. Still now, it makes me cry. If I’d talked about it 10 years ago, I probably wouldn’t have cried.” Getting older, says Nicky, now 73, she “feels more tender”.

Nicky is a painter who has produced a unique and haunting memoir, in pictures and words, of her extraordinary childhood. It reveals a trio of fathers who were highly regarded figures in public, but who, in private, variously inflicted on her mental, physical or sexual abuse.

She is upset when we meet because an event to promote her book has been marketed as, “Come and meet Nicky Loutit, the sexually abused child from the London elite.” Her publisher points out that this is all true. “I don’t remotely feel I am an ‘abused child of the cultural elite,’” says Nicky. “The book is more about an old person remembering ghastly things and the emergence of exquisite beauty.”

Then again, she recognises the possibility that her distress about the event is a form of denial. “I’m astonished at the power of denial. Even when I was little, teachers might say, ‘I understand you’re from a broken home.’ I was like, ‘That’s fine.’ ‘No self-indulgence’ and ‘nobody is special’ was what I picked up from my mother and her friends. I was in constant pain, but I didn’t recognise it. I had this wonderful act, acting cool, and it was gross denial. Gross denial gave me the strength to get through.”

An image from Nicky Loutit’s book New Year’s Day Is Black.

Nicky was the first child of Janetta Slater, a beautiful young bohemian woman who mixed with leftwing artists and writers of 1940s and 50s London: Kee, George Orwell and the literary critic Cyril Connolly. Unsurprisingly, given Connolly’s famous dictum about “the pram in the hall” being an enemy of good art, Nicky felt unwanted, an impediment to the creative high-achievers around her. “That’s what made me so vulnerable to being abused,” she says. “I was desperate to be some use, in some way.”

When Janetta fell in love with Kee in the late 1940s, Nicky, her daughter by another man, was “in the way”. Nicky “genuinely can’t remember” if Kee pushed her down the stairs, but she had a terror of stairs as child. “He used to get hold of me and violently shake me, shouting, with his face very close to me. It was very frightening. He was jealous, resentful, maddened. For anyone else, he was charming, amusing and very popular.”

Nicky wasn’t in the care home for long and was reunited with her mother before being sent to boarding school. Nicky loved her new school uniform and tie but was subjected to a subtler form of mental abuse after her mother left Robert for Derek Jackson, a brilliant, wealthy physicist who split from Pamela Mitford to marry Janetta in 1951. Derek drove eight-year-old Nicky to her new school in his flash car. “He had this great Buick and I was embarrassed. He sensed my embarrassment and drove round and round on the gravel in front of the school. He loved it.”

Derek called her Knickers. “Oh, God, he was ghastly. I never liked him. He was quite happy to pay for me to go to school, he wanted to please my mother, but he didn’t care a fiddle about me.” He would test Nicky on maths and delight in her failures. She often confuses her numbers, even now.

Derek had a daughter with Janetta. “When my sister was born, Derek went off with my mother’s sister. My mother was devastated and miserable. It was all very frightening. The big question of my childhood was to help her be happy – it was the wrong way round, but that’s how it was.”

All the while, Nicky secretly idolised her biological father, Kenneth Sinclair-Loutit. He was a doctor who had treated casualties in the Spanish civil war and enjoyed an illustrious career with the World Health Organisation (“His life is an inspiration to all of us,” said a newsletter for former WHO staff after his death). Janetta never married Kenneth but Nicky bore his surname, looked a bit like him and “thought he must be wonderful”. She only met him a couple of times but her fantasy was sustained when he posted her “a really exciting American Indian leather jacket. I was thrilled with that present but it would be months, years between contact.”

When she was nine, the one reliable man in her childhood helped to reunite father and daughter. James MacGibbon was a publisher and friend of Nicky’s father Kenneth; Nicky played with James’s son, Robert, and adored them both. James, Robert and her father planned to sail across the Channel and on to Paris together and James, thoughtfully, suggested they invite Nicky along.

In her book, Nicky’s paintings recall her excitement about the trip, the boat setting sail and then the horror: her father forcing himself on her in a cramped cabin. He sexually abused her on their week-long journey across the Channel and up the Seine to Paris. She couldn’t tell anyone: “He specifically said it was something special between us.”

The MacGibbons slept in a separate cabin and didn’t know. They were appalled when Nicky told them 30 years later.

Boarding school was a release. “I felt saved by boarding school. I loved it but had to slightly pretend that I didn’t, because my mother and her bohemian friends were scornful of school,” she says. She was beaten by a maths teacher and then found friendship with one particular teacher. “He was wonderful, very inspiring,” she says. “He loved me and I was very important to him.” He took her on little excursions in his car and read Under Milk Wood and the Bible. He continued lessons in his bedroom, where several pupils joined him in bed. As his favourite, Nicky had to put her head on his chest. “Looking back, it was sexual, but it wasn’t overtly sexual like my father’s abuse,” says Nicky.

One day, she was swimming with other pupils, naked, in the lily pond in the school grounds (which, bizarrely, they were allowed to do). She had reached puberty and her friends were envious of her breasts, but he saw that she had “got too big” and “I wasn’t this sweet little girl snuggled up to him”. He ended their special relationship. “It was devastating. His rejection was like everything in my childhood. It wasn’t said, so I couldn’t reply, ‘Please don’t.’ It just happened. Suddenly, I was never invited into his room again.” She recalls his next favourite: a younger, skinnier pupil. “It was terrible,” she says.

“Everyone’s life is like that isn’t it?” says Nicky at one point. “This isn’t brute horror, it’s just normal life.” Doesn’t she feel angry about her abuse? “Oh. Anger,” she ponders. “A lot of the abuse wasn’t violent. I’ve had a very privileged life.” But she is angry with her father. “I can rage at the image of him in his grave. I can’t really speak to my father’s [other] children, who don’t really believe my father did anything, so there’s anger there.”

When she was 15, she met him again, in Paris. “He took me out to dinner and wanted the fantasy that we were lovers. I was terribly confused. He didn’t lie on top of me and do terrible things like he did when I was younger, but I was certainly in an abusive relationship.” A decade-and-a-half later, she confronted him about the abuse. “He cried and said, ‘Yes, I was a terrible bastard’ and he was so sorry. I even managed to feel sorry for him, but he tried to continue his sick relationship with me. He was never really sorry and he never really changed.”

Nicky’s adulthood has been shaped by this abuse. Most disastrously, when she was a young mother, with three young boys, she and her ex-husband joined an ashram in India led by a charismatic, free-loving, abusive guru. Nicky was beaten up in group “therapy” yet remained in the ashram for four years. Nicky knew she must escape and eventually returned to Britain, where she raised her three boys. “Was it being a painter?” she wonders. “Somehow I knew there was beauty in life too and I had to get out.”

Nicky can now talk about her three abusive fathers but is less comfortable discussing her mother: “Her own mother died while she was pregnant with me and she hated her father. She was a lost, beautiful young thing in the war. She talks about me in rosy, romantic terms, what a wonderful, independent child I was – ‘independent’ is the biggest praise she can give anyone. That was just how posh parenting was in those days: children were seen and not heard.”

In contrast, Nicky’s three grown-up sons are “absolutely central” in her life. “They are all terrific, which is amazing because I’ve not been a terrific mother,” she says. “The only thing I’ve done is consistently loved them.” Despite her traumatic childhood, her life, she says, has been full of beauty. “Painting isn’t an escape, but through creating I’ve realised the depth of existence. Even if I’m agonised, there’s another side full of joy and light. I’ve had such a rich life and I’m grateful. I suppose I was enabled to explore, even when things were very painful.”

An image from Nicky Loutit’s book New Year’s Day Is Black.

She does not see herself as a victim. “Funnily enough, being a victim is one of my mother’s positions,” she says. “My mother was quite clear, ‘All men are bastards’, but because of the abuse, if men wanted me that was wonderful.” When she met her current husband, the author Jonathan Gathorne-Hardy, she was “startled that he could be so kind,” she says. “The major thing he’s done is allow me to be myself and we’ve really, really loved each other ever since, so that’s all right.”

I wonder at her achievement of enjoying loving relationships when her childhood was cursed by terrible men. “How the hell have I done it?” she laughs. She points out that as well as her two marriages she’s had great support from James MacGibbon, his wife and their son, Robert, who is still a close friend. “I did feel loved,” she says. “There’s always been someone in my life who has sort of loved me.”

• New Year’s Day Is Black – An artist’s journey through memory by Nicky Loutit is published by Propolis