Thailand's former prime minister Chavalit Yongchaiyudh recently ignited a furore when he proposed that the separatist campaign in his country's Muslim-majority southern provinces might be solved politically, with a form of self-rule. Thailand's ruling Democrat party immediately called Chavalit's remarks "traitorous".

But recent developments surrounding Afghanistan's elections have highlighted the shortcomings of using military force alone to resolve a civil war. This precedent offers an important lesson for Thailand and other countries facing intractable insurgencies. As Aristotle put it, "politics is the master science in the realm of action".

In June 2006, I sat in a village in southern Thailand's Yala province with several ordinary-looking young men who had taken part in extraordinary events. They had joined the militants who had attacked a dozen security checkpoints across three southern provinces on 28 April, 2004.

Armed mainly with machetes and kitchen knives, 106 attackers perished that day, 32 of them inside Pattani's historic Krue-Se mosque, where they had taken refuge. Five members of the Thai security forces were also killed.

None of the youths I talked to could give any clear explanation for their actions, except to say that an Islamic schoolteacher known as Ustadz So had recruited them into a shadowy militant movement. Ustadz had taught them that Thai rule over this historically Malay region was illegitimate, that Thai officials were cruel and heartless, and that the time had come for the Muslim population to rise up and expel the Buddhist infidels.

According to counter-insurgency expert David Kilcullen, the violence in Thailand's deep south – which has now claimed more than 3,500 lives – was some of the most intense in the world between 2004 and 2007, "second only to Iraq and Afghanistan during this period." Yet, despite such alarming levels of bloodshed, the insurgency has been under-reported and under-researched, barely registering on the international community's radar screen, largely because the conflict cannot be reduced to a sound-bite-friendly narrative of identifiable bad guys and good guys.

First, the militant movement itself has no name, operating in a highly decentralised manner through small local cells that operate relatively independently – what might be described as self-managed violence franchises. Thai political expert Chaiwat Satha-Anand has called the movement a "network without a core". The lack of a defined enemy makes reporting on and understanding the conflict difficult for journalists, diplomats, and government officials.

Second, outcomes are not predictable. On 28 April, 2004 the insurgents' primary targets were Buddhist members of the security forces, but most of those who died were Malay Muslims. The would-be perpetrators also became the primary victims. Some victims have been killed extra-judicially. The one group not targeted by anyone has been westerners. Again, reporting such a messy story is a challenge; most western news editors have preferred to highlight Iraq and Afghanistan.

A third factor is that the violence has become a marginal issue even within Thailand. The breaking news teams returned to Bangkok long ago, where a military coup, ongoing mass protests, and other big political stories have kept them busy ever since. And, because the violence tends to come in dribs and drabs – rarely since 2004 has any one day's casualties reached double figures – the south has never risen back up the news agenda.

Many Thai military and police officials now privately admit that the insurgency cannot be defeated through security measures. After a dip in the number of incidents during late 2007 and 2008, violence once again increased in 2009. The military's belief that Malay Muslims could be re-educated and re-socialised into accepting a Thai identity has proved untenable.

The same goes for earlier hardline rhetoric about rooting out the militants and destroying their organisation. The current government under prime minister Abhisit Vejjajiva misses the point with rhetoric about creating peace simply through promoting justice, or by funding large-scale socio-economic development projects.

Malay Muslims in Thailand's southern provinces are demanding their own political space. Few of them seriously believe that a tiny separate Patani state, wedged between Thailand and Malaysia, would be viable. What many are seeking is some form of special status within Thailand, enabling them to pursue their own cultural and religious traditions without interference from Bangkok.

Until the Thai government grasps this simple point, young men like those I met in Yala will still be recruited into militant activity. As Chavalit understands, the southern Thai conflict is a political problem in need of a political solution – just like Afghanistan and other more familiar wars.

• Duncan McCargo is professor of south-east Asian politics at the University of Leeds and author of Tearing Apart the Land: Islam and Legitimacy in Southern Thailand, which won the 2009 Asia Society Bernard Schwartz book award

• Copyright: Project Syndicate, 2009.