It is a friendship as remarkable as it is unlikely: Robi Damelin is a 65-year-old Jewish grandmother whose son, David, served in the Israeli army; Ali Abu Awwad, 37, is a former Palestinian revolutionary who joined the first intifada as a teenager and was later sentenced to 10 years in an Israeli prison – he served four. David Damelin was killed by a Palestinian sniper seven years ago; two years before that, Awwad's brother, Youssef, 32, was shot and killed by an Israeli soldier.

These were two deaths that might have been expected to contribute to the cycle of violence and hatred that characterises the Israeli-Palestinian conflict. Instead, for the past five years Damelin and Awwad have travelled the world, speaking together at mosques, synagogues, parliaments and public meetings to spread the message that there will only be peace in the Middle East when there is reconciliation. Last week they brought that message to Britain, attending an Amnesty International event and speaking to an audience of Muslims at the London Central Mosque.

In the back of an east London coffee shop, the two reflected on a relationship that both hope may prefigure a different future for Israel and Palestine. "If someone had said to me when I was 15 that I would have someone like Robi as a friend," said Awwad, "in my wildest dream I would not have imagined it. But when I met her she began to tell me about her relationship to her sons and how the killing of David affected her relationship with her other son.

"I felt this very deeply because my mother didn't pay much attention to me after the death of my brother, because she was closer to him than me. We made a connection."

That connection was made through the Parents Circle, a remarkable organisation that brings together Israelis and Palestinians who have lost immediate members of their family in the conflict. The circle, also known as the Bereaved Family Forum, was set up by Yitzhak Frankenthal, an Orthodox Jew whose 19-year-old son had been killed by Hamas activists. Today it has more than 500 members who have all rejected violence, hatred or revenge as a means to solve the conflict.

"At first I didn't want to mix with bereaved parents," admitted Damelin, "but I gradually realised that the only hope for progress was to recognise the face of the conflict – that was why when they told me that David had been killed my first instinct was to say I did not want anyone killed in the name of my child. I did not want revenge."

It was through the Parents Circle that she met Awwad, whose brother had been shot in the head. "When I learned about my brother's death I was just so full of anger," he recalled. "But I realised that for me to survive I needed to meet someone from the other side who could understand me – so when I met Robi I met someone who gave me hope."

It is a hope which both Damelin and Awwad are keen to stress is not based on pretending there are no disagreements, but is instead rooted in the idea that the first step towards reconciliation is to recognise thesuffering of the other side. "If you are pro-Palestinian or pro-Israel you are not helping us," said Damelin. "You are just feeling good about yourself."

"For some Muslims, the situation in Palestine is theoretical," said Awwad. "Maybe that is why there is more militancy among British Muslims than there is in Palestine, and it's the same with American Jews who seem less likely to compromise than Israelis. Why? Because they are not the ones who have to stand at roadblocks every day, it isn't their children, their sons and brothers who are dying, so it is easy for them to say they do not want to compromise."

Compromise, Damelin and Awwad believe, is possible; they are both hopeful for the future. Damelin – who was born in South Africa – suggested that the Bereaved Families Forum could inspire a future Truth and Reconciliation Commission in Israel and Palestine.

"I remain an optimist," she said. "After all, if you had told me in 1967 when I left South Africa that one day whites and blacks would sit together in the same room I would have thought you were mad." As mad perhaps as if she had been told that a Palestinian ex-revolutionary would become one of her closest friends. "It is very difficult to give away things that belong to your child," she said in a quiet voice. "It is like admitting that they will not come back.

"But I had this lovely jacket that a relation had brought David – a beautiful jacket – and Ali and David are the same height and build, so I chose to give that jacket to Ali, and I am happy I did. It was the right thing to have done."

"Whatever you do, you cannot replace the child a mother has lost," said Awwad. "Youssef and David will never come back again, but at least you find something to live for. Wherever I go I carry a picture of David with me, it's in my bag with me right now."

The tea has been drunk and it is almost time to leave. It must be a strange experience, continually and publicly reopening the wounds of personal loss in the hope that it may eventually inspire peace. "This is my life, I don't have a choice any more," said Damelin, when asked how long she imagined she would continue travelling. "I have a motherly relationship with Ali. I worry about him, I care about what happens; maybe I am a little overprotective about him."

There is a palpable sense that she and Awwad complete the other in some way, offering some consoling echo of what has been lost. Theirs is a friendship that only appears unusual when individuals are reduced to labels and tribes; and yet when Ali Abu Awwad and Robi Damelin are viewed not as Israelis and Palestinians, Jews and Muslims, but rather as a son and mother, the relationship begins not to feel unlikely at all.

"I have found in Robi what I didn't get from my own mother," said Awwad. "She knows what kind of clothes I like, the people I like, and she advises me on all these things. She even knows what food I like."

"Shrimps," said Damelin, laughing. "He is addicted to shrimps."