Global investors freaked out last week with just a hint from Federal Reserve Chairman Ben Bernanke that the U.S. could pull back on its stimulus program later this year.

But a powerful group, representing global central banks, wants more.

In its annual report, the Bank for International Settlements suggested that global central banks should stop relying on stimulus to grow their economies and instead encourage authorities to speed up reforms.

"We are past the height of the crisis, and the goal of policy has changed -- to return still-sluggish economies to strong and sustainable growth," said the BIS, a consortium of central banks like the Fed and the European Central Bank.

"Can central banks now really do "whatever it takes" to achieve that goal?"

Related: Fed sets road map for end of stimulus

The report stated that the initial purpose of stimulus programs was to do "whatever it takes" to prevent financial collapse.

But at this point, the BIS called the programs "cheap money" which delays needed reforms in the economy and the financial system.

Investors have been skittish around any kind of central bank pullback talk. U.S. markets fell two weeks in a row after the Fed suggested it could start easing up on its controversial stimulus program, which is pumping $85 billion a month into the U.S. economy.

Central Bank action in China and Europe also has people on edge.

The People's Bank of China, which maintains tight control over the banking system, has been taking a tough line with Chinese lenders. It refused to inject cash into the financial system last week despite rocketing short-term borrowing costs, rattling investors.

The ECB, meanwhile, is in a wait and see mood and seems poised to move in the opposite direction.

The bank says it is ready to do more if an economic recovery fails to materialize later this year. It has discussed negative interest rates to ease a credit squeeze by encouraging banks in depressed southern Europe to lend more money to businesses and consumers.

Europe is constantly wrangling with the long-running debate over spending cuts and tax increases that have formed the core of the eurozone's response to its credit crisis. In recent months, European policymakers have gone out of their way to acknowledge that there are limits to austerity programs.