The terrorist attacks in Mumbai seven weeks ago sent shock waves around the world. Now all eyes are fixed on the Middle East, where Israel's response to Hamas's rockets, a ferocious military campaign, has already left a thousand Gazans dead.

Seven years on from 9/11 it is clear that we need to take a fundamental look at our efforts to prevent extremism and its terrible offspring, terrorist violence. Since 9/11, the notion of a "war on terror" has defined the terrain. The phrase had some merit: it captured the gravity of the threats, the need for solidarity, and the need to respond urgently - where necessary, with force. But ultimately, the notion is misleading and mistaken. The issue is not whether we need to attack the use of terror at its roots, with all the tools available. We must. The question is how.

The idea of a "war on terror" gave the impression of a unified, transnational enemy, embodied in the figure of Osama bin Laden and al-Qaida. The reality is that the motivations and identities of terrorist groups are disparate. Lashkar-e-Taiba has roots in Pakistan and says its cause is Kashmir. Hezbollah says it stands for resistance to occupation of the Golan Heights. The Shia and Sunni insurgent groups in Iraq have myriad demands. They are as diverse as the 1970s European movements of the IRA, Baader-Meinhof, and Eta. All used terrorism and sometimes they supported each other, but their causes were not unified and their cooperation was opportunistic. So it is today.

The more we lump terrorist groups together and draw the battle lines as a simple binary struggle between moderates and extremists, or good and evil, the more we play into the hands of those seeking to unify groups with little in common. Terrorist groups need to be tackled at root, interdicting flows of weapons and finance, exposing the shallowness of their claims, channelling their followers into democratic politics.

The "war on terror" also implied that the correct response was primarily military. But as General Petraeus said to me and others in Iraq, the coalition there could not kill its way out of the problems of insurgency and civil strife.

This is what divides supporters and opponents of the military action in Gaza. Similar issues are raised by the debate about the response to the Mumbai attacks. Those who were responsible must be brought to justice and the government of Pakistan must take urgent and effective action to break up terror networks on its soil. But on my visit to south Asia this week, I am arguing that the best antidote to the terrorist threat in the long term is cooperation. Although I understand the current difficulties, resolution of the dispute over Kashmir would help deny extremists in the region one of their main calls to arms, and allow Pakistani authorities to focus more effectively on tackling the threat on their western borders.

We must respond to terrorism by championing the rule of law, not subordinating it, for it is the cornerstone of the democratic society. We must uphold our commitments to human rights and civil liberties at home and abroad. That is surely the lesson of Guantánamo and it is why we welcome President-elect Obama's commitment to close it.

The call for a "war on terror" was a call to arms, an attempt to build solidarity for a fight against a single shared enemy. But the foundation for solidarity between peoples and nations should be based not on who we are against, but on the idea of who we are and the values we share. Terrorists succeed when they render countries fearful and vindictive; when they sow division and animosity; when they force countries to respond with violence and repression. The best response is to refuse to be cowed.

• David Miliband is the foreign secretary milibandd@parliament.uk