The Labour leader, Ed Miliband, has recalled his first visit to Israel at the age of seven and spoken of his gratitude to the country for providing a sanctuary for his grandmother after her husband died in the camps.

Miliband made the comments to students at the Hebrew University in Jerusalem, where he faced sometimes pointed questioning about perceived hostility in the Labour party to Israel, which one questioner suggested was verging on antisemitism, the boycott movement – which Miliband opposes – and whether, if he became prime minister, he would ever deploy British forces against a nuclear Iran.

He refused to be drawn on whether he considered himself a Zionist, despite the question being asked three times during an hour-long question and answer session.

Miliband, the first Jewish leader of the Labour party, is visiting Israel and the Palestinian territories as part of a three-day visit that will see him meet both family members and officials.

The Labour leader, whose grandfather and other family members died during the Holocaust, has described himself in the past as a "strong supporter of Israel".

He told the students: "For me, Israel is the homeland for the Jewish people, and the reason I put it in those terms is because it is not just a theoretical idea for me, it's my family's experience. That's the way I like to talk about it."

The question of Miliband's relationship to his family's Judaism and to Israel has been increasingly scrutinised since he became Labour leader in 2010. His visit to Israel is his first as Labour leader. After the question and answer session he visited the Yad Vashem Holocaust museum.

Although religion was not a feature of his upbringing, he has revisited the fact that his father, the Marxist academic Ralph Miliband, and mother were Jewish refugees, describing his sense of Jewishness in cultural more than religious terms. After an early foreign policy speech some pro-Israel advocates voiced concern that he was tougher on Israel than on Hamas. Since then he has spoken and written about his family's own experience on a number of occasions.

Describing the current state of the peace process as both sensitive and at a potentially perilous moment, he said he saw no other way forward than a two-state solution. "I do believe the growth in settlements is a serious issue for the peace process and needs to be addressed," he said, adding that they were illegal under international law.

Miliband's trip comes a month after David Cameron's own visit, during which he was briefly heckled during an address to the Knesset.

During his opening remarks to around 200 students in the senate building of the Hebrew University's Mount Scopus campus, Miliband talked of first visiting his grandmother in Israel as a seven-year-old. "The image I have of 37 years ago is seeing a photo and asking who it was and my grandmother being very upset. I was taken out and it was explained that the person in the picture was my grandfather, who had died in the camps.

"So I am grateful for what Israel did for my grandmother, providing a sanctuary for her from this indescribable grief."

On the question of critics of Israeli policy in the Labour party, Miliband said: "You don't have the family history that I've had without feeling that when prejudice and antisemitism come to the fore, it raises a whole set of deep anxieties about where it might lead."Miliband also brushed aside a question whether he planned to visit relatives living on settlements across the Green Line, replying: "I have relatives all over the country, I'm not in touch with everyone."