Jeremy Corbyn faces a new front of internal dissent over his handling of antisemitism in the party with Labour peers potentially set to hold a vote of no confidence in him next week, it has emerged.

The threat came as Labour staff members of the GMB union voted by a huge majority to pass a motion condemning the party’s response to the BBC Panorama programme on antisemitism – a move illustrating a potentially big rift between the leader’s office and party staff.

Labour peers could hold an emergency meeting on Monday afternoon to consider the vote, and if it is taken forward the ballot would take place over Tuesday and Wednesday with the result soon afterwards.

If the no-confidence motion were passed it would have no formal effect but would be a blow to Corbyn’s authority. It remains to be seen how much support there is among Labour peers for holding or passing such a vote. The plan for the emergency meeting has been put forward by one peer, it is understood.

A Labour source said it would be “both undemocratic and absurd for unelected peers with no mandate to seek to remove an elected leader who twice won the landslide support of Labour’s membership and led Labour to the biggest increase in the party’s vote since 1945”.

This week more than 60 peers took out an advert in the Guardian accusing Corbyn of having “failed the test of leadership” over his handling of antisemitism complaints within the party.

Corbyn’s office sacked Dianne Hayter as a Brexit minister in the Lords after she accused Corbyn of having a “bunker mentality” that she compared to the final days of Hitler.

Lady Hayter, who was sacked on Wednesday after accusing Jeremy Corbyn of having a bunker mentality. Photograph: PA

Lady Hayter had said: “Those of you who haven’t [read the book] will have seen the film Bunker, about the last days of Hitler, where you stop receiving any information into the inner group, which suggests that things are not going the way you want.

“That seems to be where we are at the moment: having the leadership in a bunker so they are not hearing in those views, that evidence. That is in conflict with what they are trying to do, to the extent that even undermines what they are trying to do.”

A Labour spokesman described the comments as “deeply offensive”.

Hayter, who is also Labour’s deputy leader in the Lords, said on Thursday Corbyn’s office had misunderstood what she had said and she had learned of her sacking from the media.

She told Sky News that the comments came in a meeting in which she referred to The March of Folly, a book by the US historian Barbara Tuchman, and to the common fault of leaders and their advisers to screen out critical views, saying Corbyn’s team was guilty of this. “If a leader is not getting the intelligence he needs, which is counter to what he believes, then actually the project can fail. So I was actually being quite supportive,” Hayter said.

She said that during the meeting she referred to Downfall, the 2004 film about the final days of Hitler, as a possible example of this phenomenon. “So everyone said, ‘ah, if you’re referring to the bunker and Hitler’s last days, that means you mean Jeremy Corbyn’s Hitler’.”

Hayter added: “I’ve had 50 years in the party, I’ve been chair of the party and I’ve done pretty good work on Brexit. But I’m not important to that. What’s important is that the Labour party has a problem, and until the leadership starts taking action on that – and only the leadership can – then this is going to niggle away and stop us doing the things we ought to do.”

Earlier in the week Hayter was among four Labour peers, also including the party’s leader in the upper house, Angela Smith, who wrote to Corbyn offering to set up a panel to review the allegations made by former party staffers in Panorama programme.

On Thursday Labour members of staff in the GMB union voted by 124 to four in favour of a motion condemning the Labour press office’s response to the BBC programme, which was published on Twitter. The motion said it was “unacceptable for an employee’s workload or the culture of an organisation to cause staff to have breakdowns or to contemplate suicide, as several whistleblowers claimed on the programme”.

One staff member in the meeting claimed there was a “toxic culture” in the party. “Today Labour staff said we won’t stay quiet on antisemitism or tolerate this pernicious culture any more. Jeremy has to listen. To move forward there has to be strategic changes at the top of the Labour party and that starts in his team,” they said.

The GMB said it would now seek an urgent meeting with party leaders. A spokesman said: “Our members have today expressed a number of serious concerns that must be addressed by Labour party management. GMB’s local branch representatives will be scheduling urgent meetings with Labour party management to ensure the range of concerns tabled today are properly acted upon. GMB will be offering additional support and representation to members asked to give evidence as part of the ongoing EHRC inquiry.”