It's not just bad harvests and climate change – it's also speculators that are behind record prices. And it's the planet's poorest who pay

Just under three years ago, people in the village of Gumbi in western Malawi went unexpectedly hungry. Not like Europeans do if they miss a meal or two, but that deep, gnawing hunger that prevents sleep and dulls the senses when there has been no food for weeks.

Oddly, there had been no drought, the usual cause of malnutrition and hunger in southern Africa, and there was plenty of food in the markets. For no obvious reason the price of staple foods such as maize and rice nearly doubled in a few months. Unusually, too, there was no evidence that the local merchants were hoarding food. It was the same story in 100 other developing countries. There were food riots in more than 20 countries and governments had to ban food exports and subsidise staples heavily.

The explanation offered by the UN and food experts was that a "perfect storm" of natural and human factors had combined to hyper-inflate prices. US farmers, UN agencies said, had taken millions of acres of land out of production to grow biofuels for vehicles, oil and fertiliser prices had risen steeply, the Chinese were shifting to meat-eating from a vegetarian diet, and climate-change linked droughts were affecting major crop-growing areas. The UN said that an extra 75m people became malnourished because of the price rises.

But a new theory is emerging among traders and economists. The same banks, hedge funds and financiers whose speculation on the global money markets caused the sub-prime mortgage crisis are thought to be causing food prices to yo-yo and inflate. The charge against them is that by taking advantage of the deregulation of global commodity markets they are making billions from speculating on food and causing misery around the world.

As food prices soar again to beyond 2008 levels, it becomes clear that everyone is now being affected. Food prices are now rising by up to 10% a year in Britain and Europe. What is more, says the UN, prices can be expected to rise at least 40% in the next decade.

There has always been modest, even welcome, speculation in food prices and it traditionally worked like this. Farmer X protected himself against climatic or other risks by "hedging", or agreeing to sell his crop in advance of the harvest to Trader Y. This guaranteed him a price, and allowed him to plan ahead and invest further, and it allowed Trader Y to profit, too. In a bad year, Farmer X got a good return but in a good year Trader Y did better.

When this process of "hedging" was tightly regulated, it worked well enough. The price of real food on the real world market was still set by the real forces of supply and demand.

But all that changed in the mid-1990s. Then, following heavy lobbying by banks, hedge funds and free market politicians in the US and Britain, the regulations on commodity markets were steadily abolished. Contracts to buy and sell foods were turned into "derivatives" that could be bought and sold among traders who had nothing to do with agriculture. In effect a new, unreal market in "food speculation" was born. Cocoa, fruit juices, sugar, staples, meat and coffee are all now global commodities, along with oil, gold and metals. Then in 2006 came the US sub-prime disaster and banks and traders stampeded to move billions of dollars in pension funds and equities into safe commodities, and especially foods.

"We first became aware of this [food speculation] in 2006. It didn't seem like a big factor then. But in 2007/8 it really spiked up," said Mike Masters, fund manager at Masters Capital Management, who testified to the US Senate in 2008 that speculation was driving up global food prices. "When you looked at the flows there was strong evidence. I know a lot of traders and they confirmed what was happening. Most of the business is now speculation – I would say 70-80%."

Masters says the markets are now heavily distorted by investment banks: "Let's say news comes about bad crops and rain somewhere. Normally the price would rise about $1 [a bushel]. [But] when you have a 70-80% speculative market it goes up $2-3 to account for the extra costs. It adds to the volatility. It will end badly as all Wall Street fads do. It's going to blow up."

The speculative food market is truly vast, agrees Hilda Ochoa-Brillembourg, president of the Strategic Investment Group in New York. She estimates speculative demand for commodity futures has increased since 2008 by 40-80% in agricultural futures.

But the speculation is not just in staple foods. Last year, London hedge fund Armajaro bought 240,000 tonnes, or more than 7%, of the world's stocks of cocoa beans, helping to drive chocolate to its highest price in 33 years. Meanwhile, the price of coffee shot up 20% in just three days as a direct result of hedge funds betting on the price of coffee falling.

Olivier de Schutter, UN rapporteur on the right to food, is in no doubt that speculators are behind the surging prices. "Prices of wheat, maize and rice have increased very significantly but this is not linked to low stock levels or harvests, but rather to traders reacting to information and speculating on the markets," he says.

"People die from hunger while the banks make a killing from betting on food," says Deborah Doane, director of the World Development Movement in London.

The UN Food and Agriculture Organisation remains diplomatically non-committal,saying, in June, that: "Apart from actual changes in supply and demand of some commodities, the upward swing might also have been amplified by speculation in organised future markets."

The UN is backed by Ann Berg, one of the world's most experienced futures traders. She argues that differentiating between commodities futures markets and commodity-related investments in agriculture is impossible.

"There is no way of knowing exactly [what is happening]. We had the housing bubble and the credit default. The commodities market is another lucrative playing field [where] traders take a fee. It's a sensitive issue. [Some] countries buy direct from the markets. As a friend of mine says: 'What for a poor man is a crust, for a rich man is a securitised asset class.'"