Dubbed an 'urgent security priority' by the US, Yemen has become a regional hub for al-Qaida. In the first of two special reports, Ghaith Abdul-Ahadmeets the group's new fighters

The market at Jaar, a small city in Abyan province in southern Yemen, is on a filthy, dusty road strewn with garbage, plastic bottles, cans and rotten food. Plastic bags fly on the hot wind and feral dogs sniff around the vegetable stalls. Minibuses and donkey carts jostle for space on the crowded street.

Standing in the middle of the chaos is one of the jihadi gunmen for whom the town has become famous. Thin, short, with a well-groomed beard and shoulder-length hair, he is dressed in the Afghan style: shalwar kameez, camouflage vest and an old Kalashnikov. He is either a bandit imposing a protection racket on the merchants or a rebel protecting them from the corrupt regime – and most probably a bit of both.

He waves cheerfully to the people passing by, but few give him a second glance. The jihadis – like the chaos and the filth – are an established part of the landscape of south Yemen. They attend state-run mosques and Quranic learning centres and help fill the ranks of the country's security forces.

Recently, their influence has grown more threatening. In the past two years al-Qaida has established a local franchise in Yemen, al-Qaida in the Arabian Peninsula (AQAP), which has claimed responsibility for audacious attacks – including the attempt to assassinate the British ambassador to the capital, Sana'a, earlier this year.

In Yemen, recruits can study ideology and take guidance from militant leaders, including the Yemeni-American cleric, Anwar al-Awlaki, who has been described as "terrorist number one" by the Democrat chairman of the House homeland security sub-committee, Jane Harman. Awlaki is believed to have given guidance to the so-called underwear bombing suspect, Umar Farouk Abdulmutallab, and to Major Nidal Hasan, accused of murdering colleagues in shootings at Fort Hood.

With its conservative Islam, ragged mountains, unruly tribes and problems of illiteracy, unemployment and extreme poverty, Yemen has been dubbed the new Afghanistan by security experts.

The Guardian spent two months in the country, travelling to the tribal regions of Abyan and Shabwa, where al-Qaida has set up shop and where suspected US drone attacks have killed scores of civilians and few insurgents. Speaking to jihadis, security officials and tribesmen, it became clear how a combination of government alliances, bribes, broken promises and bungled crackdowns has allowed Islamists to flourish and led to the emergence of the country as a regional hub for al-Qaida.

You don't have to go deep into the mountains to hear the jihadi message. One Friday, sitting on the roof of a hotel in Sana'a, I hear the amplified prayers of a preacher ring out at the end of his sermon: "God condemn the Jews and the Christians … God make their wives and children our slaves … God defeat them and make the believers victorious."

Ahmad al-Daghasha, a Yemeni writer who specialises in Islamic and jihadi issues, says two factors are responsible for the growing influence of al-Qaida. "First there is the local situation, which is miserable, politically and economically," he says. "That situation is translated into many forms of resistance – the jihadis and al-Qaida are only one. Then there is the foreign oppression that we all see on television – whether in Iraq, Afghanistan or Palestine – that gives al-Qaida's rhetoric legitimacy."

In the south, government control is slipping away fast. Bandits, lawless tribes, secessionists and jihadis are all fighting the regime. Though they have few ideological connections, the groups are all contributing to one thing: a failing state where extremism can flourish.

On my first day in Jaar I toured the town with the deputy governor of Abyan province. We left the market and drove to a neighbourhood built by the Yemeni socialists in the 70s to house east European agriculturalists. The small wooden prefabs are being rebuilt with cinder blocks, as if huge grey tumours were sprouting everywhere.

At the entrance to the neighbourhood, two gunmen stood guard and graffiti sprayed on the walls declared allegiance to al-Qaida. "None of those men have been to Afghanistan, you know, but it's the look that they want to acquire," said the deputy governor.

Until last year, Jaar had been in the hands of the jihadis. The government claimed to have taken it back in an army offensive led by the minister of defence, but the neighbourhood was still out of bounds for the security forces. "Government officials cannot come here," the deputy governor said. "But I can come because I have been negotiating on behalf of the government with them for a few years now."

The rise of al-Qaida in Jaar has been a gradual process of radicalisation as generations of volunteer fighters have returned from conflicts abroad: the Afghan war against the Soviets in the 1980s, as well as the Nato-led war against the Taliban and the war in Iraq in 2003. Veterans of these conflicts, as well as jihadis who have never fought abroad, are in the streets of Jaar fighting for influence. In the 1970s and 1980s, Jaar had been predominantly a socialist town. But when the regime in Sana'a fought the socialists in a short civil war in 1994, the Islamists fought alongside them. When the socialists were defeated, the Islamists were encouraged to take control of the area. Quranic centres, the Yemeni equivalent of madrassas, were established with government support.

Over the next 10 years, the town became a base for the Islamists: they had jobs and they received their salaries from the government and money that poured in from Saudi Arabia, in support of the Quranic centres.

I spoke to Faisal – a thin skeleton with a thick moustache balanced awkwardly on his small head – on the floor of his Spartna living room. A former Socialist party member and head of the Young Artist Association in the Abyan, he watched the Islamisation of Jaar happen.

"The socialists were defeated on 7 July 1994," he said. "On July 8 a group of Islamists came and picked me up, blindfolded me and took me to the HQ of political security. I was handcuffed and beaten there. They wanted to know if I was a communist and their commander declared I was one. Then they tied my arms to a tree and hung me there and started beating me up with a stick.

"Things started changing after that," he said. "The Islamists were given jobs, they became headmasters and officers." They closed the cinema and converted it into a mosque. Art disappeared and gradually women started wearing the full black niqab. "Last year they killed 10 men and threw their bodies in the streets, saying they were homosexuals," he said.

One of the leaders of change in the city during this time was Khaled Abdul Nabi. I met him in his madrassa-like compound. Young men doubling as students and bodyguards lurked in the alleyway in front of his house and at the bottom of his stairwell.

Khaled sat on the floor, pulling at his beard. From floor to ceiling behind him stretched bookshelves filled with thick, leatherbound books on jurisprudence and theology. A pistol was placed neatly in front of him.

In 1994, he said, they had been given promises by President Ali Abdullah Saleh that he would implement sharia law and form an Islamic state, so they had formed special units, operating under army leadership, to fight for him. "We formed a small unit with other brothers and stormed into the prison in Jaar and the police station and liberated the town before the arrival of the army. But none of the president's promises came true. He lied to us and we believed him, probably because we were naive at that time."

Nevertheless, after the war, the Islamisation of Jaar began. "Islamic preaching spread in this place in an extraordinary way. Mosques and sharia teaching centers were being built, we had lots of support and of course there was also the reaction to what was happening in the Islamic world, people became more committed to religion so they could fight the crusaders."

Abdul Nabi went on to form the Abyan-Aden Islamic Army in 1998, one of the first jihad-inspired groups operating in Yemen. It is accused of being behind several violent acts, including bombings and assassinations of security officers, as well as the kidnapping of 16 foreign tourists in 1998, which led to the deaths of four hostages.

In August 2008, Yemeni security forces killed five of Abdul Nabi's men in Abyan province and claimed they had arrested 28 al-Qaida supporters, including Abdul Nabi, himself.

After meeting Saleh, Abdul Nabi allegedly agreed to support the president in his fights against the Shia rebels in the north and separatists in the south and last year he was released in a general amnesty with about 175 Islamic militants, many of them his own men. He returned to Abyan to rebuild his organization, which is now affiliated to al-Qaida, and called for the formation of an Islamic state in southern Yemen.

"I agree with George Bush in one thing," he said, pulling at his beard. "He gave us a really accurate wisdom: you are either with us or against us, you are either with Islam or with the crusaders. I tell the Muslim clerics in the whole world you are either with the flag of the mujahideen and God is great or you are with the flag of the cross … there is no other option."

One of the problems he faced now, he said, was with younger generations of jihadis. When jihadi leaders try to moderate their positions, the young followers will often splinter and form more radical groups, so each generation is more radical than the next.

"The shebab [young Islamists] are part of the Islamic situation in Afghanistan, Somalia, Nigeria and Iraq and jihad is a religious duty, like fasting. But the problem is that most of them, yes they are true jihadis with good intention, they lack the knowledge."

The next generation

I was sitting with Faisal in his home in Jaar when the message came through that a young commander, Jamal, who is attached to al-Qaida in Yemen had agreed to see me.

A thin teenager was sent to lead the way. We followed him through dirt alleyways between rows of small houses of concrete cinder blocks. Plastic bottles and shards of glass crunched under our feet. A window flickered with silver light from a television and two dogs chased one another to a corner and then fought viciously.

In the darkness the town appeared even more desolate and wretched.

We entered one of the concrete shacks, which was lit by a small red bulb. There were two rooms, one by the entrance that doubled as a kitchen and a bathroom, and one that was furnished as a bedroom with brand new furniture. We sat on the linoleum-covered floor.

Jamal was in his mid 20s, with a round face, long curly hair and a pair of thin glasses that gave him the look of an art student. "Who am I?" he asked, repeating my question. "I am a mujahid. Young men dream and have ambitions in life and my ambition is to die fighting for God."

Jihad had become his life, he said. He was fighting against what enraged God … "the drunks, the apostates and the people who stop following the religion of God."

Jamal, a jihadi fighter for six years, had been to prison a couple of times and released each time the president issued a pardon. Now he was a fugitive again. "The director of security accused us of planting an explosive device in front of his house."

How had a young man living in a poor, obscure small town in the south of a poor nation, who had not travelled further than its capital city, become a threat not only to the government of Yemen, but to the world in general.

"There are too many Arabic tragedies, in Iraq, in Chechnya, in Afghanistan and in Palestine, that makes us want to fight in the way of God," he said.

"Look this is how we started. [In 2003], after the outbreak of the Iraq war, Jaar became a big training ground for the Saudis going to Iraq. Unlike the Yemenis, the Saudis had no experience in fighting. They were very religious and had lots of money, but they didn't know how to shoot. We started training them – you know we Yemenis are taught to shoot when we are children – and then a whole ring was organized to send them to Iraq via Syria."

Saleh's government knew about the jihadi training camps, he said, and had no quibble with them as long as they didn't fight in Yemen. "Saleh told us go to Iraq but not to come back and create problems for him here."

In the winter of 2005-2006, the world began to take note of the flow of jihadis heading to Iraq and the Americans started to put pressure on Syria, Yemen and Saudi Arabia to stem the flow of militants. "The government exposed the ring," Jamal said. "They started arresting people when they reached the border. We started clashing with the government and we killed some of their security forces."

In 2006 he was arrested, which led to the first of several meetings with Saleh. Saleh agreed to release the prisoners in return for their promise of inactivity. Three days later Jamal was back on the streets, but trust between him and the regime did not last long.

"They put a lot of pressure on us," he said. "I was monitored. You leave your house and there is a government spy. You come back and there are two. So we changed our procedures." When the government arrested some of the jihadis, fighting broke out again. "We fought with them again. We fought the government until all of our brothers were released."

A cycle of arrests, fighting and deal making ensued, escalating the strength and anger of the jihadis. Sometimes they would be promised compensation by the president, but when they went back to Sana'a to collect the money they would be sent from one government department to the other. Weeks would pass, and so the clashes would erupt again. "Before our last meeting with the president in 2009, Jaar fell under our control. By that time, our brothers stopped going to Iraq. They said if we are not arrested on the way and we reach Iraq, either the Americans will arrest us or we would be tortured by the shia [Iraqi government]. Why not stay and fight here.

"We entered Jaar, and the town fell in our hands. We were more than 40, the police and army left, and we called Allahu Akbar, and planted mines and explosive devices in the streets, and for the first time we went back to our homes and we slept in our beds, we were no longer fugitives, we took over the security of Jaar and we imposed sharia."

A small mouse darted across the floor between our legs. It hit one of the legs and scurried under the bed.

Even this young commander had trouble with the generation of radicals coming after him.

"We were betrayed by the people of Jaar," he said. "When we used to hide in the mountains some kids from the town used to come and bring us food and clothes. We trained those kids how to use a weapon, how to wire explosive devices, how to build electrical circuits. They were young kids. We trained them how to attack, how to hide behind a wall."

He clutched an imaginary gun and manoeuvred while he was sitting cross-legged on the floor. "Those young kids started looting and beating up people. They destroyed the town."

His voice became a mixture of blame and regret. "Because of the young, we failed in ruling the town and we had to leave and head back to the mountains."

Even for Jamal, who represents the post-Iraq war generation, there is another generation after him who don't know which government property to loot and which to leave alone, a generation he thinks is unruly.

I asked Jamal if he considered himself part of al-Qaida's organisation in Yemen. "We are all connected, all the jihadis are connected," he opened his arms and pointed at the three of us sitting on the floor. "One of us is Qaida," and he pointed at himself, "the other is protecting him," and he pointed at me, "and the other is providing logistics." And he pointed at the teenager who had brought me there.

"The two," he pointed at us, "would only know the Qaida person they are in contact with, and that Qaida person [he pointed at himself] would be the only one in that group to know the leadership."

What al-Qaida gave him, he said, was organisation. "Before Wahaishy [the head of AQAP] and Rimi [the commander of its military wing] arrived here we were chaotic, we would fight the government whenever we wanted. Now we only move when we are given orders."

As we walked back through empty dark streets I asked the teenage boy leading me how the young looked at people like Jamal.

"He is like a hero for us all, we want to be like him." Why? "Because he stands for his people. He won't let the government do whatever they like."

When I met the deputy governor again, I asked him about the meetings the jihadis had with the president and the promised money. He said: "The authority wants to contain those men. They block roads and attack military checkpoints and collect fees from shop owners. Because this is not a state of law, this a state of buying people, they treated the jihadis and al-Qaida in the same way they treated the tribes, they paid them money to lie low."

"You have to understand that the military campaign will cost money, money for soldiers, for vehicles, then money in prison, money for a court case, so the state says why should we pay three million to fight them when we can pay them one million for things to calm down and avoid their evil. But the jihadis take the money, buy weapons and become stronger, and now the state regrets that policy and it is changing."

To an extent, he said, they had been trying to buy a truce. But it had been mismanaged.

At Faisal's house, I asked him what he thought of the government's attempt to crack down on al-Qaida.

"Don't believe the government when they say we are fighting the jihadis," he said. "The government gives them money, the government negotiates with them, the government uses them to fight its enemies, and then they tell the Americans give us money so we can fight al-Qaida."

He closed his eyes and sighed. "It's a comedy," he said.

• Tomorrow Ghaith Abdul-Ahad follows the trail of the US's 'terrorist No 1', Anwar al-Awlaki