Prime Minister Kevin Rudd's leadership is under siege tonight from some of the Labor Party's most influential factional warlords.

The ABC has learned that powerful party figures have been secretly canvassing numbers for a move to dump the Prime Minister and replace him with his deputy, Julia Gillard.

Television cameras captured Ms Gillard and Defence Minister John Faulkner going into Mr Rudd's office. They have since been joined by Finance Minister Lindsay Tanner, Anthony Albanese and Senator Joe Ludwig.

Factional support for Mr Rudd has collapsed around the country and the Australian Workers Union (AWU) is now supporting Ms Gillard.

The ABC understands that senior ministers are in shock at how quickly all this has unfolded. One senior minister told ABC "this is madness".

Earlier, Senator Faulkner was being interviewed by the 7.30 Report's Kerry O'Brien when he was told of the news.

"I don't know what's on ABC news all I know is I've been sitting here talking to you. It might be on the ABC news and it's also news to me," Senator Faulkner said.

Ministers and party members have been lining up all week to voice their support for Mr Rudd but behind the scenes, party leaders have been contemplating a leadership change.

Although Mr Rudd looks likely to survive the challenge, news of the attempted coup will undoubtedly weaken him.

It is understood that the only thing holding the Prime Minister up is that his deputy refuses to join in a bid to bring him down.

A series of bad poll results have seen the ALP's primary vote tracking as low as 33 per cent.

The polls dived after a series of policy bungles and backflips made by the Government, including shelving the emissions trading scheme.

Any serious talk of a challenge is the last thing Mr Rudd needs as the Government prepares for the election.

However, if MPs want to act they will have to do so quickly.

Tomorrow will be the last sitting day before Federal Parliament breaks over winter.

This means that if a challenge was to be made against Mr Rudd to install Ms Gillard before the next election, it would have to be done tomorrow before MPs leave to go back to their electorate.

Earlier today it was revealed that the Prime Minister's chief of staff, Alistair Jordan, had been contacting Labor MPs to discuss the Government's performance.

On Friday Mr Rudd will travel to Canada for the Group of 20 leaders' meeting.