David Miliband fuelled doubts about his intentions to serve on the frontbench under his younger brother tonight after he was heard chastising Harriet Harman for clapping when Ed Miliband described the war against Iraq as "wrong".

Yesterday, the shadow foreign secretary used a speech to the party conference in Manchester to urge his supporters to unite behind the new Labour leader, who defeated him for the post on Saturday.

But today David Miliband appeared to struggle with his own advice. Despite describing Ed's speech as "very strong" and "nerveless" after leaving the conference hall, ITV News claimed to have caught him on tape making a barbed comment to Harman, the deputy Labour leader, while his younger brother was still speaking on the stage.

David Miliband, who has stood by the Blair government's decision to go to war against Iraq, looked tense and showed his displeasure as Harman applauded a key section of his younger brother's speech in which he urged the party to follow America in drawing a line on Iraq.

With his own hands kept firmly apart, he turned to Harman and told her: "You voted for it. Why are you clapping?"

She replied: "I am clapping because, as you know, I am supporting him."

The exchange suggests that the shadow foreign secretary, who was defeated in his leadership bid by his brother by a narrow margin, may choose not to add his name to the list of nominations for the shadow cabinet ahead of tomorrow's deadline..

Ed Miliband professed ignorance of the comment when he was approached by the BBC later. "I haven't seen that," he said. "All I know is that David has been incredibly gracious both since Saturday and in response to my speech, where he sent me a very nice message."

When challenged about the incident, Harman told the BBC: "I can't remember exactly what he said to me … he was clapping at the end of that section as well."

Asked why she had applauded, she said: "Because that was a passage in his speech where he was saying we've got to move on from the distrust ... As far as weapons of mass destruction are concerned, we were wrong."

She added: "Everybody in the shadow cabinet was clapping that portion of the speech."

Ed Miliband cast the decision to invade Iraq in 2003 as one of the many errors New Labour made during its 13 years in office.

He urged the party to be humble about mistakes in the past and draw up an optimistic alternative for the country's future.

Ed Miliband, who was not an elected MP at the time of the Iraq vote in parliament, made his criticisms of the foreign policy decision a dividing line between himself and his older brother, who was serving in government at the time.

The new Labour leader said in his keynote speech today: "I've got to be honest with you about the lessons of Iraq. Iraq was an issue that divided our party and our country. Many sincerely believed that the world faced a real threat. I criticise nobody faced with making the toughest of decisions and I honour our troops who fought and died there.

"But I do believe that we were wrong. Wrong to take Britain to war and we need to be honest about that. Wrong because that war was not a last resort, because we did not build sufficient alliances and because we undermined the United Nations. America has drawn a line under Iraq and so must we."

As David Miliband contemplates his future, he will rue private Tory polling which suggested 53% felt he was fit for the role of Labour leader, compared to a clear majority of people (64%) who said that his younger brother was not up to the job.