The global food price crisis is exposing frightening levels of vulnerability in poor nations around the world. Yet these are countries into which the rich world, for half a century or more, has diverted hundreds of billions of dollars of humanitarian aid in pursuit of the high ideal of ending poverty. It is a good moment to take stock and ask what went wrong.

Compare two of the most vulnerable economies, Haiti and Botswana. In Haiti, spiralling food prices have in recent months prompted widespread rioting, claiming the lives of six people and forcing the resignation of the prime minister. This unrest has set back the search for political stability in an archetypal "fragile state". No such riots have occurred in the Southern African nation of Botswana. In a country that imports 90 per cent of its food, soaring prices have undoubtedly hurt the poor, but the state has the money and capacity to help them cope.

Why does Haiti sink while Botswana swims? A landlocked state with a small population and an arid landscape, Botswana has a high dependence on diamonds - the very "curse of wealth" that has destabilised many other African countries. At independence in 1966, it had just two secondary schools and 12km of paved road, and relied on the UK for half of government revenues. Botswana ought to be a basket case.

But Botswana has become Africa's most enduring success story. Its GDP per capita has risen a hundredfold since independence. Over the past three decades it has been the world's fastest-growing economy. It negotiated hard-fought deals for its diamonds with De Beers and used the royalties well. It has throughout remained one of sub-Saharan Africa's few non-racial democracies, despite being bordered (and occasionally invaded) by racist regimes in South Africa and Rhodesia.

The secret of Botswana's success lies in politics. The country's elite come from a single dominant ethnic group (the Batswana) whose governance systems, emphasising broad consultation and consensus-building, emerged largely unscathed from colonialism. Botswana's leading human rights activist calls it "gentle authoritarianism". The government broke every rule in the so-called Washington consensus, setting up state-owned companies, nationalising mineral rights and steering the economy via six-year national development plans. "We are a free-market economy that does everything by planning," one local academic told me, laughing.

In the second half of the 20th century, dozens of developing countries emulated Botswana's success and achieved similar growth rates. "Getting the politics right" was key for them all. These countries have built effective states that guarantee the rule of law, ensure a healthy and educated population, control their national territories and create a positive environment for investment, growth and trade. For many, the growth spurt began with the redistribution of land and other assets.

This story bears little relation to the cruder theories of development advanced by rich-country governments or, for that matter, some NGOs. Yet, getting the politics right really can "make poverty history". Aid alone cannot.

In many countries the state remains a work in progress and the rosy picture is not without flaws. Power battles and shifting alliances mean reverses are frequent. Raw power and gangsterism prevail in states that are more master than servant to their citizens. In his novel Nineteen Eighty-Four, written at the onset of the Cold War, George Orwell portrayed a totalitarian state built around the cult of Big Brother: "If you want a picture of the future, imagine a boot stamping on a human face - for ever." In the 20th century, some 170 million people were killed by their own governments, four times the number killed in wars between nations. But the worse deprivation and suffering now are not Orwellian in nature. They exist where states are weak: half of all children who die before the age of five live in states defined as "fragile".

Fixing this is not easy, but it can be done. Some states once branded as "failing" provide evidence. Malaysia went within a few decades from a post-independence meltdown of ethnic rioting to an industrial powerhouse. The economist Ha-Joon Chang points to his own country, South Korea, from where, in the 1960s, government officials were sent by the World Bank to Pakistan and the Philippines to "learn about good governance". The pupil swiftly outstripped the master.

If you define development merely as rising GDP per capita, then the story almost ends there - effective states create the basis for rapid growth. But development, parti cularly tackling poverty, is about far more than that. When the World Bank, in an unprecedented exercise, asked 64,000 poor people around the world about their lives, what emerged was a complex and human account of poverty, encompassing issues that are often ignored in the academic literature: the importance of being able to give one's children a good start in life, the mental anguish that poverty brings. The overall conclusion was that, "again and again, powerlessness seems to be at the core of the bad life".

Tackling such powerlessness is not just about election campaigns and government. Building "power within" - for example, women's assertiveness to insist on their right not to be beaten in the home - and "power with" - in the form of collective organisation - is essential to achieving the wider empowerment that transforms politics and societies.

In 1900, New Zealand was the only country with a government elected by all its adult citizens. By the end of the century, despite severe reversals, including fascism and communism, and succeeding waves of military coups against elected governments, there were ostensibly 120 electoral democracies in place. Democracies are often flawed and, as we have seen in several African countries, progress is reversible, but the overall trend remains positive.

Successful transformations

Effective states in east Asia and elsewhere have typically taken off under autocracies. In Latin America, active social movements and political organisations have rarely been accompanied by effective states. Does this mean active citizenship and efficient governments are mutually exclusive? Happily, the evidence suggests that the "Asian values" argument for benign dictatorship, once espoused by leaders in Singapore and Mal aysia, is wrong. A recent survey by the Harvard economist Dani Rodrik found that democracies produce more predictable long-run growth rates, greater short-term stability and more equality, and are better able to handle economic shocks.

Many of the countries that have had active citizens and been run efficiently have already ceased to be poor and disappeared off the development radar. Some of the most successful transformations in the past century, such as those of Sweden and Finland, have been triggered by social pacts within a democracy, showing what the combination of activism and good government can achieve.

Yet, though this combination is at the heart of development, it is seldom acknowledged in debates about the "development industry", typified by international institutions such as the World Bank and the IMF. Here, economic policy is king, and politics is often seen as an irritating process through which unworthy individuals use their power to unravel the plans of wise economists. "Getting the prices right" requires the state to get out of economic management, freeing the stage for the true heroes of development: the entrepreneurs.

It hasn't worked. The retreat of the state in Latin America, once a faithful devotee of Washington consensus prescriptions, failed to lead to lasting progress. Meanwhile, countries such as China and Vietnam, which maintained a central role for the state, prospered.

The importance of politics in development will only grow. The world is entering a new age of scarcity, in which food, water and carbon are rationed, either explicitly, through regulation, or implicitly, by price. In this environment, conflicts over access to basic resources are bound to intensify. Politics and power will decide who gets what.

All this poses challenges to the $100bn global development industry. Official donors such as the UK's Department for International Development are trying to reassess their thinking to understand better the role of politics in development. But they face a dilemma: any outside body, especially a government institution, interferes with domestic politics in developing countries at its peril. To get round this, there is always a temptation to turn political issues into technical ones - for example, by focusing on "governance" or "institution-building". But, by failing to confront issues of power, such approaches often give rise to the same frustrations as those that focus on economic policy: why won't these countries do what's good for them?

From grass roots to government

International organisations such as Oxfam have long been criticised by some developing-country partner organisations for preferring policy to politics. But they face real limits. Charity law, mission and bitter experience should dissuade them from becoming mere support groups for any political party in a given developing country. Instead, they have to promote empowerment without becoming politicised. It is a fine line to tread, but it is eminently feasible.

In Bolivia, for example, 20 years of support for the Chiquitano Indians helped them move from semi-slavery to becoming a political force, with the founding of indigenous people's organisations, such as that led by José Bailaba, and the election of Chiquitano mayors and senators. Following the election of South America's first indigenous president, Evo Morales, a land reform bill gave the Chiquitanos rights to a million hectares of traditional lands.

Even though the alchemy of development takes place primarily in the crucible of effective states with active citizens, global institutions such as aid donors, the UN and transnational corporations play a significant role.

Nation states will not wither away, even if their actions are constrained by an ever-growing web of global and regional trade agreements, bilateral investment treaties and the proliferating "soft law" of international conventions and codes of conduct on everything from financial services to human rights. Rich-country governments and their citizens need to ensure that this system of global government supports national development efforts based on the state and its people working together. They must also deter powerful countries and corporations from doing harm, whether through paying bribes or imposing policies that hurt the poor.

The fight against poverty, inequality and environmental collapse will define the 21st century, as the fight against slavery or for universal suffrage defined earlier eras. It is hard to imagine a more worthwhile cause.

Duncan Green is the author of From Poverty to Power

published by Oxfam on 23 June