Alan Greenspan gave a stark warning about the financial markets Wednesday.

"I think there are two bubbles. We have a stock market bubble and we have a bond market bubble," Greenspan said in an interview on Bloomberg Television. "I think [at] the end of the day the bond market bubble will eventually be the critical issue."

The former Federal Reserve chairman predicted a rise in interest rates and fretted over the national debt and budget deficit.

"We are dealing with a fiscally unstable long-term outlook in which inflation will take hold," he added. "In fact I was very much surprised that in the State of the Union message yesterday all those new initiatives were not funded and I think we're getting to the point now where the breakout is going to be on the inflation upside. The only question is when."

Greenspan noted how productivity levels have not risen much in the past 10 years. He also cited the recent drop in the value of the U.S. dollar versus other currencies.

"We are working our way towards stagflation," he said.

Greenspan is widely known for the "irrational exuberance" speech he gave at the American Enterprise Institute in 1996. The address warned about asset price levels, but came well before the stock market topped out in 2000.