Drivers on Australia's busiest train line are voicing their alarm about an unprecedented number of dangerous mud holes along the track that have the potential to cause derailments.

The track's conditions were so bad that last month buses replaced passenger trains between Wodonga and Melbourne because of multiple speed limits.

One driver, speaking anonymously to ABC's 7.30 Report, has described the main Sydney-to-Melbourne line as "'an accident waiting to happen".

"The odds are there for a train to derail. I'd describe it as absolutely atrocious. I've never seen conditions like it in my 30-odd years on the job," he said.

Another driver says he and his colleagues are concerned for their passengers' safety.

"You're just driving your train normally and all of a sudden you hit this bad hole in the road or some track defect and it shakes the whole train around," he said.

Rail, Bus and Tram Union spokesman Bob Hayward says the mud is the result of a bungled re-sleepering program carried out by the government-owned Australian Rail Track Corporation (ARTC).

It replaced wooden rail sleepers with concrete ones, but Mr Hayward describes it as "a massive misuse of taxpayers' money".

"It was supposed to decrease transit times to make rail more efficient. And it's done the opposite, the total opposite," he said.

The union is calling for an independent inquiry and an audit of the infrastructure program.

"The Federal Government needs to act. We are happy to have public money spent on infrastructure - that's what the union's been campaigning on for a long time - but it needs to be spent properly," he said.

Wagga Wagga resident Karen Brill says since the concrete sleepers went in, the whole track bends.

"All the carriages just bounce and you can hear the carriages - well, not carriages, the containers more so," she said.

"They just bounce all the way through. And it's so noisy. It's just - and it's very scary actually. You sort of think they're going to come off."

ARTC chief David Marchant agrees that the track is rough and needs to be rectified.

But he denies the union's claims that the mudholes were caused by the sleeper replacement program, saying they are a long-standing issue exacerbated by recent rains.

"They aren't the worst it's ever been, but they are worse this year than for a number of years and we are addressing that and we are addressing it quickly," he said.

"We are addressing it in a way which solves a long-term problem."