"I remember my mother dying, quite vividly, and afterwards sitting in the car – a big old Armstrong Siddeley I think – and I was in the passenger seat and Daddy just cried and cried. After that, we moved forward – that was it." Fay Ballard, daughter of the writer JG Ballard, is describing what happened when her mother, Mary, died of pneumonia, aged 34, on a family holiday in Alicante, Spain. After the burial, her father drove Fay, her sister, Bea, and brother, Jim, home to England. Fay was seven years old.

She recalls a doctor visiting their holiday apartment and her mother's fight to breathe with the aid of an oxygen cylinder; and, finally, her father emerging from the bedroom, holding his children tightly and saying, "She's dead."

"My father was the most wonderful, loving, brilliant father," she says, "but we never talked about our mother. Not once. We couldn't discuss her with the wider family either. I never discussed her with Bea or Jim. I felt very awkward if her name ever came up. I buried the trauma deep inside of me."

After her father died of cancer in 2009, Fay, who is an artist, came across some family photographs of her mother that she had never seen before when she was cleaning the family house in Shepperton, Surrey. "I wanted to bring her back to life," she says. "The only way to do that was to start drawing her. I found the process very cathartic. I put these photos up in my studio at home and started living with them, getting used to them. I started talking to her, as if she were there in the room with me."

Her mother has been a blank canvas to her and now she is starting to sketch in some details. "There was a collection of small black-and-white snaps of my mother playing with my brother, Jim, and me on Prestatyn beach in north Wales. My mother is fooling around with us on the sand, playing on all fours and pulling funny faces – my first direct insight into her personality.

"There was also a box of colour slides of the holiday in Alicante in 1964, taken before she died. When I opened the box, I cried. Here she is with us three children, standing in front of a cracked, dried-up river bed, smiling at the camera, arms round us."

I meet Fay, 57, at the Eleven Spitalfields gallery in east London where her exhibition of artwork relating to her parents is on show. There is a heart-breaking yearning in these pencil and watercolour drawings of objects that evoke memories of her parents, Fay somehow trying to conjure her mother through Mary's powder compact, the Welsh blanket she had slept in or the Staffordshire tea cup that Fay's maternal grandmother gave her and which she feels Mary had used too.

The first object that she started to draw was her mother's watch, which she'd stolen from the room in the hotel in which she'd died. She'd kept it ever since. "I hid it in a drawer at home – my father must have known."

In her art, Fay, who is also the joint executor with her sister Bea, of Ballard's literary estate, is echoing the work of her father, whose protagonists are often engaged in a desperate search for meaning following some catastrophic event, and who have to adapt to a harsh new environment.

None of the pictures in the gallery are of her parents together; they are isolated and compartmentalised. Fay has called the collection House Clearance after the auctions she remembers buying furniture from when the Ballards didn't have much money. "I saw her as fragmented," she says of her mother. "I remember her Playtex girdle, her breasts, but I don't think of her as a whole person. I've still got to work that through." Perhaps the most powerful images are of her mother and father holding her as a baby. Behind is a statue of the sphinx, a menacing portent of what was to come. She has entitled them Omen (i) and (ii).

Omen, Fay's drawing of her mother, holding her as a baby in front of a sphinx.

"It was only after my father's death in 2009 that I found a sense of release but also urgency, wanting to learn more about my mother." She had read Ballard's autobiography, Miracles of Life, when it was published in 2008, which contained a chapter devoted to her mother. The information, including a photograph she had never seen, was almost all new to Fay. She learned that her parents had met in a London hotel and that Mary had worked as a secretary at the Daily Express for Charles Wintour. She had always thought that her mother had been a nurse and met her father at Cambridge where he studied medicine. But Fay felt awkward and embarrassed; it was a subject she couldn't discuss with him.

The drawings of her brother Jim's flipper, which he'd worn on holiday in Spain – and used from the early 1970s as a doorstop at Shepperton – appear also to be of a hooded woman whose face is a void. This absence is also apparent in other drawings, such as one of her mother's swimming cap. Then there is a lemon that was kept on the living room mantelpiece – for 40 years.

There's something oddly surreal about some of these images. Given that her father wrote novels such as Crash, which dealt with the sexuality of car crashes, and The Atrocity Exhibition, growing up, Fay had access to books other than Enid Blyton. "I liked these strange crash injuries or plastic surgery books because the images were so vivid and arresting. And the surrealist art books, including Max Ernst and René Magritte." She says that the flipper "is a sort of Dada joke. Maybe it references swimming pools in his work ..."

She didn't read any of her father's novels until she went to university, where she studied history of art. "I felt that it would be like opening a drawer in his bedroom and reading his love letters. Too personal, too intimate."

Fay is aware of the slippery nature of some of her memories. For example, there is a drawing of an origami elephant that she is convinced she and her dad made together but when she looked through the origami book that they used, there was no elephant in it. "This is the thing about memory: it's part fiction, part reality."

All through her 30s, Fay had a recurring nightmare of a large black cloud that got bigger and bigger and started to press down on her with such force that she would wake up in a cold sweat. She saw a therapist about the dream and it eventually went away. But after her father's death, she experienced a different recurring nightmare. "I started to have these very strange swimming pool dreams in which I was in a pool full of excrement and all this horrible stuff, unable to swim freely." It's hard not to see this as being connected to one of her father's signature images – that of a drained swimming pool.

Did she ever feel angry with him for not being able to talk about her mother? "No, I was more ... sympathetic. I can understand that as a man left with three children to bring up, you have to make these choices, and it was partly a convention of the time and he felt he had to move on."

There were also her father's brutal war-time experiences in Shanghai – he was interned by the Japanese as a boy from 1943 to 1945 – to take into account. Though it's clear that this was a traumatic experience for Ballard, Fay says that he actually quite liked being in Lunghua camp and based his novel Empire of the Sun on that time. "It was the first time he'd been so close to his parents, living in the same room. His parents before that were quite remote and he was looked after by his White Russian nanny, Vera. He had a very different upbringing to us. But I remember in the summer of 1976, in Shepperton, he was naked the whole time, walking around the house. It was very funny. He used to go into the garden and the neighbours would complain. The complete opposite to how he grew up – this cathedral school boy living a very formal upper-middle class life. He really broke away from that."

In 2008, when Ballard became too ill to look after himself, he moved in with his long-term partner Claire Walsh. Fay helped him move, and when she went to pick him up from the family home in Shepperton, it was the first time she had been there in 15 years.

"He always preferred to meet in town," she says, "usually with Claire and often in a Chinese restaurant, or he'd visit me at home."

One of only two artworks on display to feature any colour is Farewell, based on a picture she took of her father looking out from his study into his wild, overgrown back garden for the last time. "I'm really fascinated by the house and these objects, and what they mean. Was he trying to arrest time, was he creating this timeless zone in this house? Or were these objects just not important to him? To me they unlock so much memory.

Fay Ballard in her garden with a sculpture JG Ballard made in the early 1960s. Photograph: Graham Turner for the Guardian

"There were certain things – like the flipper and the lemon – that he did allow to be covered in dirt or dust because they weren't part of the house. They were something else."

Fay visited Shanghai a couple of years ago and the house her father grew up in. A waitress who showed her round the building, now a restaurant, also took her next door to see where she lived. Fay was amazed to find how uncannily similar her kitchen was to their kitchen at Shepperton – yet more evidence, she feels, of her father recreating Shanghai in Shepperton. "It had the same black and white chequered line, the same white tiles, the same wooden cupboards with mottled glass," she says, incredulously.

In Miracles of Life, Ballard argued "At least my girls were spared the stress that I noticed between many mothers and their daughters as puberty approached" and there are similar rationalisations such as "I probably needed them more than they needed me." Clearly, Ballard tried to blot out many of his experiences. "I remember taking it for granted that there would be whisky on his study table at 9am," says Fay. "Whisky was a helping friend all day long."

Even in the last few months of his life, when his physician Dr Jonathan Waxman tried to broach the subject of his wife's death (they were writing a book together), it wasn't something he could talk about. It was a wound that never healed.

Ballard wrote that he often thought his children brought him up in the "chaotic, friendly brawl" at Shepperton, and his fiction is the great counter-example to Cyril Connolly's enemy in the hallway. Bea has written that Ballard was "our mother and father combined".

"When I reached the age of 34, that was a milestone," says Fay. "I was overtaking my mother – a huge moment for me. It was quite healing to have my own children actually [Isabella and Matthew are now 18 and 16 respectively], to feel that I'd become a mother myself. But then I got worried about dying young, of leaving them."

Fay says it has been very hard for her and her siblings to not have a focus for their grief and "perhaps the drawings are the gravestones, where I can lay the flowers". She has never been to Alicante to visit her mother's grave and is afraid to go even now. "I'm a bit frightened about how I'd respond," she admits. "I feel I might break down. I'd have to be feeling in a fairly strong place to go there. But I must go. I have to go."

• House Clearance is at Eleven Spitalfields Gallery, London E1 until 29 June. To visit, call 020-7247 1816. More information, elevenspitalfields.com, Fayballard.com