The US secretary of state, John Kerry, has warned in a closed-door meeting in Washington that Israel risks becoming an "apartheid state" if US-sponsored efforts to reach an Israeli-Palestinian peace settlement fail.

In an apparent sign of Kerry's deep frustration over the almost certain collapse of the current nine-month round of peace talks – due to conclude on Tuesday – he blamed both sides for the lack of progress and said failure could lead to a resumption of Palestinian violence against Israeli citizens.

The remarks were made on Friday at the Trilateral Commission, a non-governmental organisation of experts and officials from the US, western Europe, Russia and Japan. A recording was acquired by the Daily Beast website.

Kerry also suggested that a change of either Israeli or Palestinian leadership might create more favourable conditions for peace and the final, long-delayed agreement on the shape of a Palestinian state.

Aftewards, as a public storm grew around the remarks, Kerry issued a statement regretting the use of the word apartheid, saying it had opened him to "partisan political" attacks.

"If I could rewind the tape, I would have chosen a different word," he said.

Israeli leaders had made similar points in the past but "apartheid [is] a word best left out of the debate here at home", Kerry said.

Regardless of the apology, Kerry's remarks represent a significant departure, as senior US officials historically have avoided the word "apartheid" relating to Israeli policies. It is believed to be the first time a US official of Kerry's standing has used the contentious term in the context of Israel, even if only as a warning for the future.

The Emergency Committee for Israel, whose chairman is the prominent neo-conservative William Kristol, said: "On Friday secretary of state John Kerry raised the spectre of Israel as an 'apartheid state'. Even Barack Obama condemned the use of this term when running for president in 2008. It is no longer enough for the White House to clean up after the messes John Kerry has made. It is time for John Kerry to step down as secretary of state, or for President Obama to fire him."

Although the danger to Israel of a failure to move towards a two-state solution has been framed by Israeli politicians in terms similar to those used by Kerry, US officials have long been wary of following suit. When the former president Jimmy Carter used it for the tile of his 2006 book Palestine: Peace or Apartheid it caused controversy.

Kerry's comments reflect similar recent warnings to Israel from western diplomats that the collapse of the peace talks might lead to the country's increasing isolation.

Kerry said: "A two-state solution will be clearly underscored as the only real alternative. Because a unitary state winds up either being an apartheid state with second-class citizens – or it ends up being a state that destroys the capacity of Israel to be a Jewish state.

"Once you put that frame in your mind, that reality, which is the bottom line, you understand how imperative it is to get to the two-state solution, which both leaders, even yesterday, said they remain deeply committed to."

Kerry has had a sometimes strained relationship with some senior Israeli officials as the peace talks have become gridlocked. In January Israel's defence minister, Moshe Ya'alon, described Kerry as "obsessive and messianic".

In 2008 in an interview during his election campaign, Barack Obama explicitly rejected "injecting a term like apartheid" into the discussion over Israel and Palestine. "It's emotionally loaded, historically inaccurate, and it's not what I believe," he said.

Attempting to defuse the row, Jen Psaki, spokesperson for the US state department, said: "Secretary Kerry, like justice minister Livni and previous Israeli prime ministers Olmert and Barak, was reiterating why there's no such thing as a one-state solution if you believe, as he does, in the principle of a Jewish state.

"[Kerry] was talking about the kind of future Israel wants and the kind of future both Israelis and Palestinians would want to envision. The only way to have two nations and two peoples living side by side in peace and security is through a two-state solution. And without a two-state solution, the level of prosperity and security the Israeli and Palestinian people deserve isn't possible."