Formal peace talks between Israel and the Palestinians will begin next week amid rising tensions over Israel's continued settlement building and the European Union's tough stance on funding for Israeli institutions with links to occupied territory.

The US state department announced that the first substantive talks for five years would open in west Jerusalem on Wednesday and then move to the Palestinian desert city of Jericho. The talks will be preceded by Israel's release of the first batch of 104 long-term Palestinian prisoners, expected on Tuesday.

But the approval on Thursday by Israel's defence ministry of plans to build more than 800 new homes in West Bank settlements prompted a warning from Palestinian negotiators that such moves could scupper talks.

Saeb Erekat, the Palestinian chief negotiator, wrote to the US secretary of state, John Kerry, the driving force behind the renewed peace process, to demand he take action to stop continued construction.

Without a halt, it was hard to see how talks could "bring about progress towards a peace agreement", Erekat said. The announcement was evidence of "Israel's bad faith and lack of seriousness", he said. He urged Kerry to "take the necessary action to ensure that Israel does not advance any of its settlements plans and abides by its legal obligations and commitments".

His warning was echoed by Hanan Ashrawi, a member of the Palestine Liberation Organisation executive committee, who said the decision was "consistent with the pro-colonisation approach of [Binyamin] Netanyahu and his coalition partners, who are determined to make secretary Kerry's initiatives fail … With their policies, Israel is effectively annexing the occupied West Bank."

The state department spokeswoman, Jen Psaki, said US officials had expressed its concerns to the Israeli government. "We do not accept the legitimacy of continued settlement activity," she said in Washington.

The interlinked issues of borders, territory and settlements are the top priority for Palestinian negotiators, who insist that the pre-1967 line should be the basis of their future state. The US shares this position but Israel has refused to sign up to it in pre-talks discussions.

The defence ministry approval followed a decision by the Israeli cabinet this week to include a record 90 West Bank settlements on its "national priority" list, making them eligible for generous government subsidies. "It is our duty to encourage the continued settlement in Judea and Samaria," said Naftali Bennett, a key far-rightwing partner in Israel's coalition, using the biblical term for the West Bank.

Meanwhile, Israel has said it will forgo an agreement with the EU worth hundreds of millions of dollars in research grants rather than sign up to stringent new conditions excluding institutions and bodies with a presence in or links to settlements in East Jerusalem, the West Bank or the Golan Heights.

Horizon 2020, a seven-year European scientific research programme, which Israel was invited to join, is the first test of new EU guidelines reflecting a growing willingness in Europe to take action against settlements. Ze'ev Elkin, the deputy foreign minister, told Israel Radio that "we can't sign" under the new terms.

Israel is expected to announce on Sunday the names of 26 Palestinian prisoners who will be released before the renewed peace talks. The men, who have been in jail for more than 20 years for crimes that include violent attacks on Israeli citizens, are among 104 prisoners who will be freed in four stages over the next nine months. The government is obliged to identify the prisoners 48 hours before their release to allow for last-minute legal challenges.

The move is vehemently opposed by many victims' relatives and has been denounced by some politicians. There are likely to be emotional scenes in the West Bank and Gaza as the freed men are reunited with their families and feted as heroes by the public.

The state department said Kerry was not expected to make any public statements next week on the progress of the talks. He told Jewish American leaders in Washington on Thursday that peace was a "strategic imperative" for both Israel and the Palestinians. He was due to meet representatives of the Arab American community on Friday.