Matthew Bristow spent two years documenting the cocaine industry in Colombia. In the first of three films, he meets the farmers guardian.co.uk

Up close, erythroxylum coca looks almost pretty – a plant with curving branches, green leaves and small yellow flowers that mature into red berries.

It has been cultivated on the slopes of the Andes since before the Incas, and invested with divine properties. When chewed, its leaves act as a mild stimulant and help overcome hunger, thirst and fatigue.

But these virtues do not alter the fact that having an ideal climate and terrain for coca – the raw ingredient of cocaine – has been a catastrophe for Colombia. The crop has wrought violence, narco-trafficking and corruption.

Divine or otherwise, coca has proved resilient, verging on indestructible, in withstanding the decades-old "war on drugs" declared by Richard Nixon and prosecuted by successive US presidents.

Military helicopters continue to scythe over treetops in the Colombian jungle and hundreds of millions of dollars are still poured into the fight – but there is a growing conviction that it cannot be won.

It may evolve and change shape, move from jungles to cities and from bloody battles to discreet bribes, but it will not end with a flag planted in the ground and victory declared.

An individual coca bush is fragile, but the forces behind it are powerful and adaptable: peasant farmers who turn the leaves into paste, clandestine laboratories which turn it into powder, guerrillas and armed gangs who traffic it abroad, middle men and state authorities who launder the revenue. Each link in the chain has a strong incentive.

A peasant in certain remote parts of Colombia has a choice: grow corn, rice, potatoes and vegetables for prices that fluctuate and sometimes barely make it worthwhile, or grow coca, safe in the knowledge of a handsome return.

Colombia's US-backed eradication effort includes satellites and fumigation-spraying aircraft, but growers have adapted with more resistant strains and smaller plots hidden under taller plants.

Government inducements to wean peasants off coca with loans and alternative economic activities have faltered.

"Government policies related to zero coca, and strict verification procedures, take a long time and limit the state's ability to work with communities in transitioning from a coca economy to a legal economy," a recent US Agency for International Development (USAID) report said.

"When security and coca eradication are not synchronised with the arrival of socio-economic projects, the mood of a community can quickly become hostile."

A new book, Shooting Up: Counter-insurgency and the War on Drugs, by the respected Brookings Institution scholar Vanda Felbab-Brown, says eradication campaigns in Afghanistan and Colombia have left drug production unaffected but alienated locals, gifting political capital to insurgents.

Plan Colombia, the military-heavy US aid programme, has had significant success in helping the country's security forces push Revolutionary Armed Forces of Colombia (Farc) guerrillas out of cities and deep into the jungle.

A country that once risked collapsing into chaos now has political stability, a growing economy and a popular president, Alvaro Uribe. But Farc and a smaller leftist rebel group, the ELN, have adapted to their restricted theatre of operations and continued trafficking cocaine, which remains their main income source.

In recent months, Farc has made a military comeback, ambushing troops and kidnapping and killing a provincial governor. Analysts think the pendulum could be swinging back their way.

"The Farc seem to be bouncing back," Leon Valencia, the director of the Nuevo Arco Iris (New Rainbow) thinktank, said. "The decline of the democratic security policy has begun."

Rightwing paramilitary groups also remain in the game. Originally set up by ranchers in the 1980s to combat leftist guerrillas, the paramilitaries mutated into narco-trafficking private armies.

They controlled swaths of territory and co-opted businesses and politicians until a government scheme from 2005 demobilised 32,000. Many leaders were extradited to the US, but many lower-ranking "paras" who failed to find jobs or promised state assistance have returned to what they know best – trafficking drugs.

"According to the government, the [demobilisation] process was successful. However, shortly after the demobilisation process, new successor groups emerged in the entire country that continued the criminal activities," Jose Miguel Vivanco, the Americas director for Human Rights Watch, said.

A recently-published report by the organisation – Paramilitaries' Heirs: The New Face of Violence in Colombia – makes grim reading.

The city of Medellin, once the showcase of Colombia's counter-narcotics fight, illustrates the intractability of the problem. A steep fall in violence paved the way for an apparent urban renaissance, but murder rates rose again last year as drug gangs battled for control.

Prominent local figures, with government backing, are now trying to negotiate a truce. That has raised suspicion of a return to the era of discreet pacts, when officials gave cartels free rein to traffick cocaine in return for social peace.

With victory in the so-called drug war ever more elusive, there are growing calls around the world – from thinktanks, law enforcement officials and former presidents – to decriminalise cocaine.

Just as the end of prohibition doomed the bootleggers, the logic goes, decriminalisation could put traffickers out of business. It is an experiment no government has yet dared to try.