Water wars could be a real prospect in coming years as states struggle with the effects of climate change, growing demand for water and declining resources, the secretary of state for energy and climate change warned on Thursday.

Ed Davey told a conference of high-ranking politicians and diplomats from around the world that although water had not been a direct cause of wars in the past, growing pressure on the resource if climate change is allowed to take hold, together with the pressure on food and other resources, could lead to new sources of conflict and the worsening of existing conflicts.

"Countries have not tended to go to war over water, but I have a fear for the world that climate instability drives political instability," he said. "The pressure of that makes conflict more likely."

Even a small temperature rise – far less than the 4C that scientists predict will result from a continuation of business as usual – could lead to lower agricultural yields, he warned, at a time when population growth means that demand for food was likely to be up by 70% by 2060. By the same time, he noted, the number of people living in conditions of serious water stress would have reached 1.8 billion, according to estimates.

"Climate change intensifies pressures on states, and between states," he told the conference, gathered to discuss whether climate change and natural resources should be regarded as a national security issue. "[Its effects] can lead to internal unrest … and exacerbate existing tensions. We have to plan for a world where climate change makes difficult problems even worse."

But Davey recalled previous global catastrophes that had been averted, including the threat of nuclear armageddon during the cold war, and successes such as the elimination of smallpox. He urged governments to work on adapting to climate change as a matter of urgency, as well as striving for an international agreement to cut greenhouse gas emissions.

His call was echoed by Ali Bongo Ondimba, president of the Gabonese Republic. He told the conference that Africa was the most vulnerable part of the world to climate change, but that African people had been responding to a changing climate for thousands of years – his own Bantu people had been forced, centuries ago, to move around Africa as areas dried out and food became scarcer.

Gabon had already started to take action to protect the 88% of its land that is covered by rainforest, and to reduce carbon emissions by its industries, with a view to a "transformation" by 2025.

He warned that seeking to lift people out of poverty could not be achieved at the expense of degrading natural resources. He warned that policies for economic growth across the continent must reflect this immediately: "The impact [of degradation] cannot be reversed by policies conceived too late."