Cabinet Minister Christopher Pyne has played down the "breathless speculation" that the federal government might go to an early election.

"There will be an election when there is an election," Mr Pyne told Sky News on Saturday.

"We're not panicking."

Labor frontbencher Richard Marles said reports the government was considering a double dissolution election showed there were worries about a "lack of agenda and disunity".

"From Labor's point of view, it's all fantastic," he told Sky News.

"Let's have an election as soon as possible."