The former head of the Federal Reserve said fiscal stimulus efforts have fallen far short of expectations, and the government now needs to get out of the way and allow businesses and markets to power the recovery.

“We have to find a way to simmer down the extent of activism that is going on” with government stimulus spending “and allow the economy to heal” itself, former Fed Chairman Alan Greenspan told a gathering held at the Council on Foreign Relations in New York on Wednesday.

At this point, “we’d probably be better off doing less than more” because “you’d be far better off to allow the normal market forces to operate here,” Greenspan said. That’s largely because stimulus spending is not proving as effective as many had hoped. “To the extent the evidence suggests very large deficits concurrently crowd out capital investment, there is a debit to the stimulus program that is somewhere between a third and a half of what the gross stimulus is,” he said.

Greenspan was Fed chairman from 1987 to 2006, and was succeeded by current central bank chief Ben Bernanke. Greenspan’s tenure at the Fed was defined by his opposition to government regulation and his confidence that markets would discipline themselves.

To many, the financial crisis was largely born of that ideology, and to some degree, Greenspan has himself said he overestimated the market’s willingness to understand and price for risk. Greenspan has also been widely accused of running an overly easy monetary policy during the early years of the last decade, in turn providing the fuel that powered the housing bubble, the rupturing of which drove the worst recession in generations.

Greenspan’s comments came in response to moderator and audience questions, and were accordingly wide-ranging. The former central banker noted that gold, the price of which has been surging, still represents the “ultimate means of payment.” What is happening in that market “is a signal there is a problem with respect to currency markets.” He reckons the problem is not a large one, but the jump in gold prices could be “the canary in the coal mine to keep an eye on.”

Greenspan, while worried about the outlook and what is happening with the housing market, said when it comes to a double-dip recession, “the probability of that is going down.” Given all the ground the economy has lost, “the tinder for a double-dip is not readily available,” although he added if housing goes down “all bets are off.”

Greenspan said that the U.S. needs to do something now to deal with budget deficits and it must do something very soon. He explained his anxiety is so high that “I’m coming out in the first time in my memory” in support of higher taxes in addition to reduced spending, including allowing the so-called Bush tax cuts to expire.

“Our choice is not between good and bad; it’s between terrible and worse,” Greenspan said. The nation has “a level of commitment … which I don’t think we can psychically meet,” absent huge changes in how the government finances itself.