On a side street off Mogadishu's Wadnaha Road frontline a young officer is explaining the unwritten rules of the city's intractable civil war as his men exchange fire with an unseen enemy.

The fighters shooting at him are from the Hizb al-Islam, he explains. He knows this because they fight longer than al-Shabab, the other main Islamist group besieging Somalia's tiny government-held enclave, but also because they told him. "We have friends there. They tell us before they leave their base that they are going to attack. When they want to fire mortars they tell us so we can take cover."

If the conflict that has turned Mogadishu into a virtual no-go zone for 19 years occasionally resembles a grim farce, there is nothing farcical about the scene around us.

Nearby lies an array of flip flops in different shapes and sizes and always in singles: blues, reds, purples, tiny plastic ones with flower designs and large leather ones attesting to previous skirmishes, advances and retreats. A jungle of trees and shrubs has taken over the deserted street so that the soldiers have to push the branches with their elbows and guns to make a path. Houses and shops are shattered, empty and riddled with bullet holes.

Somalia is the world's invisible conflict, and perhaps its least comprehensible. Since January last year, when Ethiopia pulled out of the country, the Islamist government of Sharif Ahmed has been locked in an attritional struggle with al-Shabab, a more radical offshoot of the Islamic Courts movement, the alliance of tribal sharia courts which once controlled most of southern Somalia. The government is also under attack from Hizb al-Islam, many of whom fought alongside Ahmed against the Ethiopia.

Al-Shabab and Hizb al-Islam control most of Mogadishu and south and central Somalia, having squeezed the internationally backed government into a sliver of land defended by an African Union force. But it is hard to keep up with the shifting frontlines of this conflict: when I was in Mogadishu last May the government controlled all of Wadnaha and Factory roads, the main arteries that cross the city.

Soon after I left, the commanders and their troops in that area joined the opposition, and the government lost three miles of territory including the camps at the ministry of defence and the stadium.

When the warlord Yousuf Neda Adi switched sides again – this time rejoining the government with his troops – the government line stretched back and gained another few hundred metres. But Adi now believes the government may have been behind a recent assassination attempt against him.

But there is more at stake here than a few square miles of territory. Al-Shabab have established themselves as the Somali franchise of al-Qaida, aspiring to be named as al-Qaida in Somalia – just as with jihadi groups in Yemen and Morocco. They are imposing a regime of extreme sharia law on the areas they control that makes the Taliban seem moderate. Western security experts, Somalis living abroad and local fighters say the country is fast becoming the favoured destination for wannabe jihadis.

The addition of the whine of US drones to Mogadishu's symphony of tank, mortar and machine gun fire is evidence of the deep anxiety the conflict is causing in Washington and other western capitals. As one minister told me over a breakfast of goat liver, bananas, papayas, chapattis and sweet milk tea: "For the first time in many years the international community is interested in Somalia, not because of our suffering but because of al-Qaida. The British and the Americans are interested in helping us because they see the anarchy in Mogadishu is hitting them back home."

Beheading video

Abdey Qadir is a tall figure with small, sunken eyes and a thick beard that grows only under his chin, giving him the appearance of a fierce goat. He is an intelligence officer in the Amniyat or security division of al-Shabab.

We meet in a room on the government side of the frontline. He pulls a Chinese mobile phone from his pocket, fiddles for a bit, then holds it in his giant hands and shows me a grainy bluish film.

A man dressed in a white shirt and dark trousers is lying on his stomach on the ground. He is blindfolded with a black cloth, his arms tied behind his back. Another man is standing astride him, one foot pinning his shoulder to the ground. The victim's feet shake but he is silent and his mouth is closed. There are trees around and the person who is filming shouts "Allahu akbar. Allahu akbar." The "executioner" pulls the man's short hair up, the head lifts, he stretches his right arm under his neck and starts cutting from left to right. In short fast moves, the knife moves up and down, in and out. The body shakes and a pool of blood flows calmly and gathers under the head. The executioner pulls the knife to the right and then goes back to the start and cuts deeper this time to separate the head.

The film stops and there is a thick cold silence in the room.

"We killed him because he was a spy," Abdey says calmly. "We captured him trying to cross from the government lines."

Qadir explains that the practice of beheading and removing limbs, for which al-Shabab have become notorious, has been an important element in establishing the group's grip on large areas of the country.

"One of the reasons for our strong name is not only the war, it's the strong fierce rule that is based on cutting heads as punishments for the crimes," he says. "We have gained respect. We implement a strong rule that no one can deviate from which has also made us very popular with Arab and other mujahideen. We have courts all the time that implement sharia, but when we are in the middle of war and the fighter captures the traitors and the apostate soldiers of the government then we implement the sharia immediately and cut the head."

Qadir tells me proudly that he doesn't himself carry a gun.

"My duties are to bring news, watch the people who move weapons to the government side from the weapons markets and find the enemies of al-Shabab in our area … To kill people you don't need a gun … Not always."

I ask him why he fought a government that imposed sharia on Somalia and is led by one of their former allies in the Islamic courts movement. "According to our beliefs Somalia was never an Islamic country – it has to be liberated from the apostasy. After that we move to Kenya, Ethiopia, Djibouti … The resistance never stops at specific borders."

Al-Shabab's origins date to the mid-1990s when a group of militant jihadis split from the Itehad al-Islami, the main Islamist organisation at the time, in Somalia and later joined a loose alliance known as the Islamic courts.

The more militant elements in the alliance gave Ethiopia, ever nervous about the Islamist presence on its doorstep, the pretext to invade.

Ethiopia's occupation was backed by the US but after a war of insurgency led by the courts alliance, the Ethiopians withdrew, handing security to the African Union.

In a clan-based society such as Somalia where it's not uncommon to hear someone say of a close cousin: "We meet in the 10th grandfather" – or approximately 300 years ago – the militias are tribal; the forging and breaking of alliances happens according to tribal interests. Even the parliament is a tribal entity based on a sub-sub-clan representation.

Foot soldiers

Al-Shabab's success – like other Islamist organisations – can partly be attributed to their "modern structure", based more on merit rather than tribal loyalties. Beliefs, rituals and loyalty to the commander of the faithful replace the traditional loyalties.

Their foot soldiers are young men, radicalised by years of war, many from the marginalised tribes of the farming south that have been dominated for the past two decades by the strong pastoralist tribes. Their tribal elders can no longer offer any resemblance of respect.

"Most of the new recruits joining us now are the zealous young, their hearts are filled with passion and zeal, who can't wait to face the enemy. They are 14, 15, 16," said Qadir.

"They empower the young," a writer in Mogadishu who lives in al-Shabab-controlled Bakara market told me. "They go to the young, give them power, the power to face that rotten structure of the tribe, power in the shape of a gun. Power as self esteem and belief … This is why they succeed. Now I am worried about my own young brother."

With power, discipline and structure, al-Shabab managed to provide "security" to the local population, making it possible for people to safely leave their houses, go shopping, do business and, unlike government soldiers who are known to be little better than looters and criminals, their fighters enjoy a good reputation.

They also levy taxes from businesses and farmers and even local herders.

"We tax the people, the companies, the farmers and the herders. But we don't use the word tax. Instead we use the term aid. We also control some ports and airports that give us revenues.

"The big money transfer companies we go to them once a month – they pay between ten thousand dollars and twenty during the war, at the time of peace few thousands only," says Qadir.

Al-Shabab is in nominal alliance with Hizb al-Islam but they often clash with each other over control of "liberated" areas and a war of assassination is going on between the two parties. Recently they have started to outbid each other on radicalism. When Hizb banned radios in Mogadishu from broadcasting music, al-Shabab issued a statement a week later banning schools from ringing bells. After al-Shabab started getting support from al-Qaida in Yemen and other jihadi groups, Hizb called on Osama bin Laden to come to Mogadishu.

Foreign backing

Just as the government receives military and financial support from Ethiopia, Djibouti, the EU and US, al-Shabab also look abroad for money, weapons and fighters.

"The government takes support from the west so we take support from our brothers the muhajiroon - immigrants," says Qadir. "Some are part of the fighting brigade, some don't leave their hiding places. They work in manufacturing explosives and strategy and those are not seen.

"They are Asians, Yemenis and Arabs with American passports, but there are also many Africans – Kenyans, Tanzanians and Moroccans." A large number of the muhajiroon arriving in Mogadishu are Somalis with western passports, he says. Some of them went on to become suicide bombers.

"We take films of the shelling and the bombing by the government and the African Union, and we show to the young in the diaspora and they come here enraged and passionate," he says. "We have our supporters in America, Australia and in Europe. Their duty is to recruit men and bring them to Somalia. The young men, most of them haven't seen war in their lives, go to military training for six months and then they fight."

Another commander with Hezb al-Islam explains the dynamics of the different foreign fighters flocking to Somalia: "Most are from Africa – Nigerians, Sudanese and Zanzibaris. There are Arabs also, most of them Yemenis, and a few Asians. And there are the Somalis from Britain, Holland, Sweden and Norway." Many of the foreigners have been trained and able to instruct Somali fighters and returning Somalis in tactics and first aid. "The foreigners, especially from Pakistan and Yemen, have a very high training. They also teach us how to make explosive belts how to plant time bombs in walls and under the floor." Later a Hizb al-Islam commander tells me his group was also attracting fighters from abroad. "Now the foreigners coming are Arabs from Europe, from the US and from Yemen. They are very experienced fighters in directing mortar and artillery fire and very good snipers.

"The Somalis are better in open field attacks but the foreigners are better in sniping and artillery."

Many things have changed in Mogadishu over the last year. Gone are the plastic chairs in the presidential office and in have come wooden chairs with leather padding. The air-conditioned office is by far the coldest place in Mogadishu; a sweater is needed to stop you from shivering, while outside the sweltering heat envelops everything and everyone.

Even the president looks happier. The trappings of power seem to suit him. He no longer carries the world-weary look I saw when he took office last year. His face slips easily into a confident smile. He wears a thin, gold watch encrusted with glittering gems.

But Ahmed, who was described by Hillary Clinton as "our best hope", now rules only over a hilltop compound, the Villa Somalia, and a few adjacent streets. His government is on the verge of collapse, the parliament is split and infighting and corruption are paralysing the administration. Officers in the army say that they haven't been paid for months, the soldiers say they have no food to eat, and a major arms dealer told me that senior officers sell him their newly supplied guns and ammunition.

"We have learned a lot in the past year," says Ahmed, his fingers flipping the turquoise stones of a prayer bead as he speaks. "We don't think just in terms of military offensives. We think about humanitarian services, of understanding the people and orienting them towards their sacred responsibility of their holy duty towards their government."

Recruits

A few days after meeting him, I head back to the presidential compound to attend the army day ceremonies. On one side of the hall are dozens of newly trained recruits, all in uniforms and boots supplied by the foreign powers that trained them, from France to Sudan and Djibouti.

On the other side are officers – former officers from the army, militia commanders and warlords. In between are ministers, dignitaries and more warlords.

Thickset bodyguards in sunglasses lead the president into the room. A brass orchestra strikes up the national anthem and everyone stands. A thin and elderly officer, carrying a rusted ceremonial sword and wearing a peeling red helmet, goose steps to the front of the hall, saluting the president and the flag with his sword.

On the wall at the front a projector shows a film in sepia shades of a Somali army parade, men dressed in camouflage or beige uniforms marching in perfect rhythm, followed by tanks, trucks and artillery pieces, and planes passing in the sky, accompanied by the commentary of a deep-voiced man. The image moves from the parade ground to the stands to show the former president Siad Barre in dark sunglasses.

At this, the hall erupts in applause and cheers for the former dictator. The film was from the late 1970s, when Somalia had one of the strongest armies in Africa, explains one of the officers next to me.

After an hour of speeches and as the president takes the podium, I stand outside watching a scuffle break out among the newly trained soldiers over the scraps of leftover food from the dignitaries' lunch inside. The Ugandan soldiers standing guard at the gate attempt to keep order but soon gave up.

Then a big explosion rocks the building. The insurgents have started shelling the Villa Somalia compound just as the president begins to speak. The soldiers keep fighting for the scraps of food but a Ugandan tank parked close to the hall starts firing back at the insurgents' positions in the crowded markets of the city underneath. Six shells whoosh from the tank.

Eighteen people were killed and 64 injured from the shelling, I was told the next day when I went to Madina hospital. The director and the staff had spent the night in the operation room. "We did 35 operations during the night," the director tells me.

Just another day in Mogadishu's very uncivil war.

• This article was amended on Tuesday 8 June 2010. In the sentence ''When they want to fire mortars they tells us so we can take cover." the word tells has been corrected to tell.