Palestinian negotiators privately agreed that only 10,000 refugees and their families, out of a total refugee population exceeding 5 million, could return to Israel as part of a peace settlement, leaked confidential documents reveal. PLO leaders also accepted Israel's demand to define itself as an explicitly Jewish state, in sharp contrast to their public position.

The latest disclosures from thousands of pages of secret Palestinian records of more than a decade of failed peace talks, obtained by al-Jazeera TV and shared exclusively with the Guardian, follow a day of shock and protests in the West Bank, where Palestinian Authority (PA) leaders angrily denounced the leaks as a "propaganda game". The documents have already become the focus of controversy among Israelis and Palestinians, revealing the scale of official Palestinian concessions rejected by Israel, but also throwing light on the huge imbalance of power in a peace process widely seen to have run into the sand.

The latest documents to be released reveal:

• The then Israeli foreign minister, Tzipi Livni, repeatedly pressed in 2007-08 for the "transfer" of some of Israel's own Arab citizens into a future Palestinian state as part of a land-swap deal that would exchange Palestinian villages now in Israel for Jewish settlements in the West Bank.

• The US secretary of state, Hillary Clinton, and other American officials refused to accept any Palestinian leadership other than that of Mahmoud Abbas and the prime minister, Salam Fayyad. The US "expects to see the same Palestinian faces", one senior official explained, if it was to continue funding the PA.

• Condoleezza Rice, secretary of state under George Bush, suggested in 2008 Palestinian refugees could be resettled in South America. "Maybe we will be able to find countries that can contribute in kind," she said. "Chile, Argentina, etc."

• Livni told Palestinian negotiators in 2007 that she was against international law and insisted that it could not be included in terms of reference for the talks: "I was the minister of justice", she said. "But I am against law – international law in particular."

The scale of the compromise secretly agreed on refugees will be controversial among Palestinians who see the flight or expulsion of refugees when Israel was created in 1948 as their catastrophe (nakba) – while most Israelis regard the Palestinian right of return as incompatible with a democratic Jewish state.

The PLO's chief negotiator, Saeb Erekat, is recorded telling the US Middle East envoy, George Mitchell, in February 2009: "On refugees, the deal is there." In June 2009, he confirmed what the deal was to his own staff: "Olmert accepted 1,000 refugees annually for the next 10 years."

Abbas, who is himself a refugee, is also recorded arguing privately: "On numbers of refugees, it is illogical to ask Israel to take 5 million, or indeed 1 million. That would mean the end of Israel."

On the issue of accepting Israel as an explicitly Jewish state, Erekat privately told Israeli negotiators: "If you want to call your state the Jewish state of Israel you can call it what you want." He told his staff privately that it was a "non-issue".

But publicly PA leaders reject any ethnic or religious definition of Israel, and it is fiercely opposed by many of Israel's 1.3 million Palestinian citizens, who see it as a threat to their own civil and national rights, particularly since there have been moves in Israel to introduce a loyalty oath along the same lines.

In several areas, Livni pressed for Arab citizens of Israel to be included in a future Palestinian state as part of a land-swap deal, raising the controversial spectre of "transfer". In other words, shifting Palestinians to another state without their consent, a demand backed in its wholesale form by rightwing nationalists.

Livni explained privately that there are "some Palestinian villages located on both sides of the 1967 line about which we need to have an answer, such as Beit Safafa, Barta'a, Baqa al-Sharqiya and Baqa al-Gharbiya". Earlier, she had made clear that such swaps also meant "the swap of the inhabitants". But Palestinian negotiators rejected the proposal.

Tonight Livni's spokesman said she had not discussed population transfers and insisted she had not criticised international law. In Ramallah on the West Bank today, al-Jazeera's offices were taken over by a crowd of 250 security forces and protesters in response to the disclosures. Abbas said they were an intentional "mix-up", while Erekat claimed they had been "taken out of context and contain lies".

But senior PLO sources accepted privately that the documents were genuine.