A Kenyan soldier clambers up to his sentry post and stares out across vast plains of bush, acacia trees and red dust. The savanna is peaceful now, but he knows that when darkness falls the enemy will return, typically a band of 15 to 20 men armed with AK-47 rifles. "Every night they are in front of us," the soldier says. "They shoot and go. They run away."

Along the frontline, the Kenyans have piled clusters of green sandbags to provide cover. Behind them, a military base is protected by high walls crowned with razor wire. About 1,200 troops from Kenya and Sierra Leone are garrisoned in this desolate Somali hinterland. On an average day, green, heavily armoured vehicles set off to patrol the crucial port city of Kismayo, running the gauntlet of roadside bombs, a deadly tactic imported from Afghanistan and Iraq. In punishing heat, soldiers can be seen rolling a surveillance drone across the tarmac of the Italian-built airport.

This is where the war on terror in east Africa is being waged. Troops from the African Union and the fledgling Somali national army are battling al-Shabaab, the extremist Islamist group notorious for carrying out beheadings, recruiting boys to fight and forcing girls into marriage that claimed responsibility for last month's attack on the Westgate mall in Nairobi, which claimed 70 victims.

Some analysts interpreted the Kenyan atrocity as a sign of weakness, the thrashings of a dying animal. But there are signs that al-Shabaab is regrouping and evolving, recruiting members more quickly than it loses them and, in the words of Somalia's president, becoming "an extended hand of al-Qaida". Officials admit that, after forcing al-Shabaab out of the capital, Mogadishu, in 2011 and Kismayo in 2012, the campaign against it has lost momentum and stalled. Military maps show swaths of red labelled "AS infested area", while the African Union force, Amisom, lacks a single helicopter in a country similar in size to Afghanistan.

A series of propaganda photographs published on Somali websites last week, apparently from al-Shabaab strongholds, show uniformed men riding through town on motorbikes and in pickup trucks, with banners celebrating the Westgate attack and, bizarrely, sporting contests such as a tug-of-war and an egg-and-spoon race. Children feature heavily in the images. "This is intended as a message they are still alive," one Somali government official said.

While a UN report in 2011 put al-Shabaab's strength at about 5,000 fighters, a Kenyan military intelligence officer serving with Amisom put the true figure almost three times higher, and probably growing. The group may have lost key urban centres, but it still controls a third of Somalia's total territory, he estimates.

"Al-Shabaab trains its recruits on a daily basis," said the officer, who did not wish to be named. "They train more new troops than are killed, so they could even be increasing. They are powerful and you cannot underestimate them. They are still very active, not in fighting but in moving, especially in areas they control."

The organisation has turned to improvised explosive devices (IEDs) to attack Amisom convoys, already injuring four Kenyans who had to be evacuated home. "They bury them along routes where they expect our troops to go then spring the ambush. We cannot rule out support by al-Qaida," said the officer. "We're not sure they're getting logistical support, but they are getting expertise. Some of the IEDs we come across are not locally assembled; they are assembled with foreign expertise."

The officer added that he had heard unconfirmed reports that the Briton Samantha Lewthwaite, the so-called "white widow" wanted by Interpol, was operating in the mountains of Somalia's Puntland province. He also cast doubt on claims that US-born "jihadist rapper" Omar Hammami had perished last month after falling out with al-Shabaab's leadership: "There is no confirmation he has been killed. They are rumours. I believe he is still around."

Nevertheless, al-Shabaab is understood to be suffering logistical problems, shortages of ammunition and recent internal power struggles, though it appears that the hardline Ahmed Abdi Godane has emerged supreme. Witnesses say that he maintains control over towns such as Barawe with just a handful of armed loyalists, whose presence is enough to instil fear and obedience.

Al-Shabaab (the Youth) first emerged as the radical youth wing of Somalia's now defunct Union of Islamic Courts in 2006. It filled a vacuum, imposing a strict version of sharia law in areas under its control, including stoning to death women accused of adultery and amputating the hands of thieves. It soon nurtured ambitions to join forces with al-Qaida, but was reportedly rebuffed by Osama bin Laden, who warned in a letter that it was causing too many civilian casualties in Mogadishu. Bin Laden's death, however, removed that obstacle and al-Shabaab declared itself an al-Qaida affiliate early last year.

Questions remain over the precise nature of the relationship, but the year-old Somali government believes they are now virtually indistinguishable. "Al-Qaida and al-Shabaab, there's no difference here in Somalia: they are one," President Hassan Sheikh Mohamud told the Guardian in an interview in Mogadishu. "The leadership, the foreigners that are fighting here in Somalia, those who died here in Somalia, all of them were al-Qaida people.

Al-Shabaab forces in pickup trucks drive through a town.

"Experts are sent by al-Qaida to train and arm and give all the new techniques of al-Qaida to al-Shabaab. Al-Shabaab is an affiliate, an extended hand of al-Qaida, there's no doubt, there's a lot of proofs of that."

Mohamud said there was no evidence the Westgate mall attack had been planned within Somalia or carried out by operatives from there, suggesting that al-Shabaab had common cause with allies across borders. "Al-Shabaab is an organisation that is based on certain ideologies and the ideology has no citizenship. This is the nature of this organisation: it's not domestic, it's not Somali only, this is an international regional organisation and its crimes have impacted at international and regional level. This is a threat to the region. So what we need as regional countries, within the framework of African Union, is to collaborate in order to uproot these evil forces."

Mohamud admitted that he had no idea of Lewthwaite's whereabouts. "These people move across borders; maybe if yesterday she was in Kenya, today she's somewhere else, maybe she's in Somalia or she's in Tanzania, Uganda or any other place. They can move, they can slip into the porous borders of the African countries."

The loss of ports and businesses has been a financial setback to al-Shabaab and, the president claimed, the government is now close to full control of the financial sector so that it can monitor incoming funds. But the Islamist group is said to run a parallel administration with strict discipline and greater efficiency, including accountants who impose taxes on goods, services and personal incomes.

The illicit trade in elephant ivory, smuggling of charcoal and expropriation of cash intended for respectable Islamic charities are among al-Shabaab's other revenue streams, generating between $70m and $100m a year (£43m-£62m), according to the UN. Some members of the Somali diaspora have also been implicated: two women in Minnesota were jailed after running a teleconference line in which al-Shabaab members openly solicited funds.

Officials fear that, in cities outside its control in Somalia and beyond, al-Shabaab will follow the al-Qaida textbook by diffusing into semi-autonomous cells plotting more attacks like Westgate. Few doubt that the group retains a lethal presence in Mogadishu despite the capital's tentative recovery. On 7 September, a car bomb exploded outside the Village, one of a chain of restaurants owned by British-trained chef Ahmed Jama. As people gathered to help, a suicide bomber in a soldier's uniform blew himself up, killing 15 people and maiming several others. Jama was in his car, having driven away just five minutes earlier.

"It was a big boom, something I never heard before in my life, like a two-tonne explosion," he recalled, pointing to dark scorch marks still visible in the parking bay. "I came back and it was a disaster. There were bodies burning. There was a small shop where a lady had been selling cigarettes and she was burning. Her sister, who had come from London, was screaming and died later in hospital."

Jama, who studied catering in Solihull and owns a restaurant in west London, considered quitting for the first time since he moved back to Somalia five years ago. Then René Redzepi of Danish restaurant Noma and other star chefs from around the world stepped in with a financial donation. "I was really close to the point of closing the restaurants," Jama admitted. "I have a wife and children who don't want to be here. But the world chefs touched me and made me feel like I should continue and not fear al-Shabaab. I was getting demoralised until then but it's given me new energy."

Sometimes the Village stays open until 2am, bringing a nightlife to Mogadishu that was unthinkable three years ago. Jama added: "I'm optimistic. The future is getting better. I came home in 2008 and every year is improving. It takes time. It needs patience." The war is being fought on fronts big and small. Last Friday, before prayers, saboteurs blew up a lamppost on one of Mogadishu's busiest thoroughfares. It seemed to be an attempt to disrupt efforts to make the streets feel safe again for pedestrians in the evening. In addition, one resident suggested, members of al-Shabaab are brainwashed into believing that lampposts have a sinister power and are "waiting for their deaths" so must be destroyed.

As well as highlighting their military prowess, al-Shabaab propaganda photos published on Somali websites also show men involved in tug-of-war games.

At Lido beach each Friday there is a festival mood and constant hubbub as thousands of young people gather, kicking a football, performing gymnastics or simply bracing in the sea and letting the surf wash over them. Girls laugh and frolic joyfully in the water, their brightly coloured jilbabs soaking as the tide comes in. "I think it's fantastic, it's vibrant and it's really a testimony that peace is coming back to Mogadishu and Somalia," said human rights activist Farida Simba, sitting in a packed beachside restaurant that opened a few months ago.

Simba's organisation, the African Initiative for Women in Africa, works with the mothers of young men recruited by al-Shabaab. The main reasons they join al-Shabaab, she said, are poverty and lack of education. "The mothers had no choice but to let young men go. The recruiters would go to the families and give them $50. They were poor and had no choice." Recruiters also use physical force, or indoctrination in madrasas, or threaten to kill the families of young men unless they join, she added. "The mothers feel helpless."

Such evidence exposes the limitations of a purely military solution, even as the African Union has called for a "surge" of more than 6,000 troops to take Amisom's strength up to around 23,000. Kismayo may have been liberated, but its 300,000 population suffers deficiencies in food and clean water, medical facilities, basic infrastructure and state schooling. On Tuesday, there was a not a single ship in port.

The absence of a functioning jail means uncertainty over what to do with al-Shabaab members who are captured or who defect. There is no programme for defectors' rehabilitation and reintegration into society.

Most of the Somali population are children or teenagers who have known only conflict and have little prospect of a job. "Al-Shabaab is both an organisation and an idea," a local politician told visiting European ambassadors in Kismayo on Tuesday. "You might be able to defeat the organisation, but the idea is still there. You must invest in education."

Officials express frustration that, despite a number of high-profile conferences, the international community is proving slow to offer practical support. Mohamud, described by critics as weak and lacking political acumen, said: "The world has to focus on one thing and only one: support the Somali government to control its own territory. Blaming, finger-pointing will not help at all.

"Unless that state and its institutions are there and control the territory, we will always have dark holes where al-Shabaab and others can go. Yesterday it was al-Qaida/al-Shabaab, the other day it was the piracy, tomorrow we don't know what will come out, and anything can come out unless there is a functioning Somali state that controls the Somali territory."