Driving east out of Aden, we were just a few hundred metres past the last army checkpoint when we saw the black al-Qaida flag. It flew from the top of a concrete building that had been part-demolished by shelling.

From here into the interior, all signs of control by the government of Yemen disappeared. This is the region of newly proclaimed jihadi emirates in south Yemen that are run by affiliates of al-Qaida in the Arabian Peninsula (AQAP), the Yemeni franchise of the movement founded by Osama bin Laden.

AQAP has existed in this ragged, mountainous terrain for years, but in the last 12 months the jihadis have moved down from the high ground to take control of cities in the lowlands. They are in the process of setting up an al-Qaida utopia here, where security is provided by jihadis, justice follows sharia law and even the administration of electricity and water supplies is governed by the emir.

Azzan, a market town in Shabwa province a year ago, is one of the three proclaimed Islamic emirates in south Yemen. When the Guardian approached it, the town entrance was defended by more than a dozen fighters equipped with armoured vehicles that had been commandeered from the government. We were met by three young jihadis and taken to the spot where the 17-year-old son of AQAP's spiritual leader, Anwar al-Awlaki, was killed, presumably by an American drone. Awlaki himself was killed in a separate strike last year.

At a small store on the side of the road, young men sat at computers copying the sermons of Awlaki, the al-Qaida leader Ayman al-Zawahiri and other household names of the global jihad. A poster on the wall advertised a film called The Survivors, featuring accounts of leaders who had survived drone strikes.

The city's old police station has been converted into a sharia court. Inside, in a room whose walls were hung with the symbols of the jihadi court – a black flag, a kalashnikov and a long stick used for delivering corporal punishment – sat the judge. He opened a small notebook as a demonstration of how the al-Qaida justice system had resolved 42 cases in a fortnight.

"People come to us from parts we don't control and ask us to solve their problems," he said. "The sharia justice system is swift and incorruptible. Most of the cases we solve within the day."

Had they had cut off any hands in dispensing justice?

"Cutting the hand of the thief is not to punish the thief, it is to deter the rest of society," he said.

Driving out of Azzan to the west for 100 miles we came to the centre of another Islamic emirate, at Jaar. Jihadi fighters met us with their newly commandeered armoured vehicle, freshly painted with their insignia and furnished with the black flag.

We threaded a way through the crowded market, past stalls of vegetables and live chickens, as gunmen on motorbikes patrolled the dusty potholed streets. Many of the town's buildings appeared to have been reduced to concrete rubble by air strikes.

In this wretched place, AQAP and their affiliates are attempting to build a new society. Unlike in Somalia, Iraq and Afghanistan, in Yemen they are trying to implement sharia by winning over the hearts and minds of the people.

In Jaar, the jihadi administration has abolished taxes, provided free water and electricity and installed sewage pipes. Their trucks distribute water to villages and Bedouin settlements.

People living in the desert on the outskirts of the town told us the jihadis had connected their village to the electricity grid for the first time in their lives. The Islamist administration has even allowed the people to continue chewing the stimulant qat. The only thing they have insisted on is moving the market for it to the outskirts of town.

A young, shy jihadi named Fouad took us into an abandoned building, where a meal was spread out on the floor. "Eat, eat, these are good times," he said, cheerfully shredding chunks of mutton with his thick stubby fingers and throwing them over.

"Times have changed, things are much better," said Fouad. "The days of suffering and hiding in the mountains have gone."

Another fighter dressed in Afghan style sat with us, but his big belly and over-stuffed magazine pouches prevented him from kneeling and scooping the rice, so he picked up a bone and reclined, sucking at it like a happy child making whistling sounds.

"Thanks be to Allah," said the fighter.

Fouad had once studied English at the college of literature at Sana'a University, but now he had adopted the dress and outlook of a jihadi.

His head was covered with a big white headscarf pulled down on his forehead and throwing dark shadows over his broad bony features. He spoke softly in classical Arabic, but his message was contemporary and had little to do with the caves of Afghanistan.

"The media tries to portray the mujahideen as ignorant people who failed in their lives, rejected by their societies and this is why they take this path," he said. "The reality is that many of the mujahideen are educated and have higher degrees, but they left their studies to care about the nation. They saw their nation insulted and living under oppression and believe it is their duty to take this path."

Fouad said that "democracy" – a word that for him covered the autocratic Arab leaders who ran fake elections in their countries – had been shown not to work.

"Democracy has failed in the Arab world," he said. "It failed in Tunis and in Egypt and Libya. It failed in Yemen. The people agree.

"Democracy only brought injustice and ignorance and backwardness and a desire to follow the west. The first people to revolt against these treacherous regimes were of course the mujahideen. But the people did not respond to them at the beginning because of the strength of the police state. The people feared them. At last, however, they started to revolt. They saw that it was either injustice and slavery or freedom. So they revolted. And we support all the revolutions. Sheikh Osama supported it.

"We also benefited from these revolutions. They gave us freedom. We were able to come out."

The revolutions had weakened the police states, Fouad said, and the jihadis had been able to exploit that. "We were able to control towns and areas. We were able to tell people about our mission. All of this was during the last revolution.

"We were aiming for this control from the start. Control under sharia is our basic goal. Nothing else. We just want to serve the people, give them what they have been missing for a long time."

The jihadis had tried to take Jaar many times before, sometimes holding the town for a few weeks before being driven out. But last year, the revolution that toppled the Yemeni president, Ali Abdullah Saleh, split the army and turned units against each other, weakening security forces. The jihadis took advantage, and this time they appear to have succeeded.

A muezzin called for prayer and the whole town began to converge on the mosque.

Fouad walked us through the market, pointing out the perfection and piety of a town where merchants leave their shops unattended while they go to pray.

"Look, the merchandise is left alone and no one steals," he said.

What if someone doesn't want to pray?

"We just take them aside and advise them on the importance of prayer," he said.

And if they still don't want to pray?

"Then we will lock them somewhere quiet and give them reading material until they realise how wrong they were."

Fouad asked with meek and apologetic smile if he could have our phones. We were then blindfolded and driven in a car to meet the prisoners.

When Fouad removed the white piece of cloth, we were outside a small compound surrounded by heavily armed men, some in local sarongs, others in shalwar kameez. A couple of them had their faces wrapped in checkered scarves.

They stood guard outside padlocked metal doors. We were led into the first room, where a dozen captured soldiers sat on the floor, their bare feet in new, shiny metal chains with a small copper padlock. They were tired. Some had long beards.

A man stood in the middle of the room to military attention and talked, his voice monotone: "We are soldiers who fought defending our country … we fought greatly until we ran out of ammunition … we ask the brother president to look into our situation and agree to the demands of the brothers in Ansar al-Sharia and exchange us with their prisoners …"

We were taken into another cell, and another soldier stood and narrated very similar lines while someone filmed him.

In the fourth room, we asked one of the men squatting on the floor how were they being treated.

"Like a prisoner. Like a prisoner," he repeated, his eyes staring at me wide open.

After we had been shown the prisoners, a heavily armed squat man said: "They are poor soldiers, they give each soldier one magazine, that's what, 30 bullets? We each carry 10 magazines how many bullets is that? You calculate and we have more supplies in the middle of the battle, we use Google maps and send scouts days before the attack. They are poor."

In the car on the way back a man – identifiable from his voice as the commander – said: "We ask the government to respond to us, and exchange those men."

And what if they don't?

"Sharia gives us three ways to deal with them, either we release them, which we won't, or exchange them or kill them," he said.

Later we walked around Jaar unaccompanied. A farmer with a thin fuzz of white stubble on his face was making his way back from the fields on the outskirts of town. What did he make of jihadi rule?

"They got three men from three different areas and they cut their hands," he said. Had they stolen? "Yes, but what? An air conditioner, a few items … but now no one dares to raise his voice in Jaar, let alone steal. This town has gone quiet. Even in the market, no one shouts. Al-Qaida have imposed security, but if they suspect that someone is spy he will disappear."

The fertile land around Jaar appeared deserted. The irrigation canals, neglected for almost a year, had dried up and the yellow earth was cracked and powdery. Where mangos and papayas had once grown were withered trees and swirling clouds of dust.

Many of the people have left, fleeing the shelling by the government and aerial attacks. Tens of thousands of refugees are packed in schools in Aden, where they live among uncollected garbage, raw sewage and poverty.

"The people have left, each family leaving one son in the house and the rest fled to Aden," said the old farmer.

One of those who has fled is Faisal. He woke up one morning last year and saw that the jihadis had taken control of the town. "Not one bullet was fired," he said.

The people tried to march in the streets of Jaar in protest against the town's takeover, Faisal said, but they were shot at and the crowd dispersed. So he joined a caravan of refugees streaming towards Aden, where he now lives in a camp.

A few months after taking Jaar, the jihadis pushed into the nearby town of Zinjibar. They surrounded the police and security services. The government's feared central security units abandoned their camp and the next day the army did the same. The jihadis took over the banks and raided the heavy weapons stores and ammunition dumps. Heavy shelling and fighting in Zinjibar in the months that followed sent tens of thousands more refugees to Aden, which now has the air of a city under siege.

In March, the emboldened jihadis scored another tactical victory, attacking a military camp on the outskirts of Aden itself. Instead of attacking the heavily guarded trenches and tank positions, the jihadis followed a mountain trail and attacked the rear of the army. In a few hours they had killed 182 soldiers and captured 72. They took the weapons and left.

Many in Sana'a and Aden couldn't believe that the Yemeni army could be so easily defeated by a band of tribespeople. Where were all the American-backed anti-terror units?

In Aden, the Guardian met a young lieutenant, his hair cropped short and his moustache pencil thin. He was still being paid by the army, but hadn't worn a uniform in more than a month. He was part of the reconnaissance unit of the 25th armoured brigade, which had been doing most of the fighting in Abyan province in the last year.

"There are many conspiracy theories about how we lost Zinjibar," he said. "Many thought that there was a deal between Saleh and the jihadis. The truth is much simpler: the army leadership is rotten and corrupt. Why would a soldier fight if the army is split in Sana'a?

"Do you know how many of them attacked the camp? Between 55 and 65."

The lieutenant's assessment of the jihadis' strength chimed with something Fouad had told us. For two decades, Abyan province and other parts of south Yemen were closely connected to global jihad. Thousands went to fight in Afghanistan. Later, new waves of jihadi Yemenis left for Afghanistan, Iraq and Somalia, and Abyan became one of the preferred routes for Saudi young men to seek training and then travel to Iraq.

"We have military commanders who lead and have new tactics and new ways to wage war," Fouad said. "The Yemeni army is much weaker than people think. They don't have a reason to fight. [The army] fights only for money, which serves a foreign agenda whether they know it or not. They follow their leaders' orders."

"We have learned," Fouad said.