The Taliban’s elusive one-eyed leader Mullah Omar lived within walking distance of US bases in Afghanistan for years, and American troops once even searched the house where he was hiding but failed to find a secret room built for him, a new biography claims.

The account exposes an embarrassing failure of US intelligence, which put a $10m bounty on Omar’s head after the 9/11 attacks in the US. Officials repeatedly suggested that, like the al-Qaida leader Osama bin Laden, he was hiding in Pakistan and died there.

It also upends the Taliban’s own account of their movement, revealing how Omar handed over practical control of the insurgency to his deputies in 2001, even though they claimed him as leader for the rest of his life, and for two years after his death in 2013.

While statements issued in his name were scrutinised around the world, he was living as a virtual hermit, refusing visits from his family, filling notebooks with jottings in an invented language and occasionally hiding from US patrols in irrigation tunnels.

Mullah Omar. Photograph: National Counterterrorism Center/Reuters

Omar regularly listened to the BBC’s Pashtu-language news broadcasts in the evenings, but rarely commented on news of the outside world as it filtered in, even when he learned about the death of Bin Laden, the man whose attack on the US led to the end of Taliban rule.

Bette Dam, a Dutch journalist and writer who has been reporting from Afghanistan since 2006, worked on the biography, Searching for an Enemy, for more than five years. It was published in Dutch last month, and a summary of some findings are being published in English by the newly launched Zomia thinktank, which shared the report with the Guardian and the Wall Street Journal.

Dam previously published an account of the former Afghan president Hamid Karzai’s rise to power and is a guest lecturer on Afghanistan at Sciences Po in Paris.

For her research on Omar she visited parts of the country controlled by insurgents and interviewed both Afghan government officials and some of the most senior surviving Taliban members, including the man who hid, protected and fed him until his death.

Jabbar Omari had been a provincial governor when the Taliban ruled Afghanistan, but gave up any active role in the movement to serve as a kind of valet and bodyguard to Omar after 2001.

Dam had been told he was the only man who could provide reliable information on Omar’s last years, but had been unable to track him down until Afghan authorities took him into protective custody at the end of last year. They agreed to facilitate an interview in a Kabul safehouse, and he agreed to talk.

Omari came from Zabul province and like the Taliban leader belonged to the Hotak tribe. His connections in the area would be vital to keeping Omar hidden when he fled back to the province of his ancestors after handing operational control of the Taliban to his deputy, Mullah Obaidullah, at a gathering of top Taliban figures in a Kandahar basement.

For the first four years of the insurgency Omar lived in the regional capital, Qalat, within walking distance of the Afghan governor’s compound and the area later chosen to house the main US base for the area, forward operating base (FOB) Lagman. His wives moved to Pakistan and he refused Omari’s offer to bring his son to visit.

A US soldier stands on top of a vehicle near Qalat, where Omar lived for four years. Photograph: Goran Tomasevic/Reuters

His shelter was the home of Abdul Samad Ustaz, Omari’s former driver, who was then operating a taxi. It was a small compound, with a courtyard and adobe home concealed behind high walls that are traditional for privacy in much of Afghanistan.

He created a secret room in the corner angle of the L-shaped residential building, its entrance a door hidden behind what looked like a high cupboard on the wall.

The family were not told the identity of their guest, only that he was a high-level Talib and that they would be killed if they spoke to anyone about it. “I have frightened my wife,” Ustaz told a friend at the time.

US forces nearly stumbled across Omar twice. The first time, a patrol came by as Omar and Omari were in the courtyard. Seized with fear at the sound of approaching footsteps, they hid behind a high pile of firewood, and the soldiers passed without entering.

The second time, US troops searched the house but did not find the concealed entrance to the secret room. Omari did not know if the search was a routine patrol or in response to a tip-off.

Mullah Omar in 1978. Photograph: Taliban

Although the house had always been near centres of Afghan power, when US troops began building FOB Lagman in 2004, just a few hundred metres from his hideout, Omar decided to move.

He headed to a remote hamlet, little more than a cluster of houses, beside a river that sometimes ran dry, around 20 miles south-east of Qalat, in Shinkay district. There, Ustaz built Omar a small shack on the edge of the river, connected to underground irrigation channels that ran up into the hills and could offer a possible escape route if US or Afghan forces came to search the area.

Soon after he moved, the US started building another base just three miles away. At the peak of the war, FOB Wolverine was home to more than 1,000 troops from the US and the other Nato nations supporting them. Omar was terrified of being caught but did not move again.

He rarely went outside, except to take the sun in winter, and would often hide in the tunnels as US planes flew over or troops passed by. “It was very dangerous for us there,” Omari told Dam. “Sometimes there was only a table width between us and the foreign military.”

Omar’s hideout was at the back of a house occupied by an extended family of Taliban supporters. Only two brothers knew the identity of their guest, but everyone in the village was aware that a senior Taliban was laying low in their area.

As disillusionment grew over corruption within the Afghan government and civilian casualties caused by its US backers, Omar started to receive gifts of food and clothes from the villagers.

He would go for days barely talking, his only interaction being with his guard and cook when they washed together in the kitchen before prayers, Omari said. Omar had an old Nokia mobile phone, without a sim card, that he used for recording himself chanting verses from the Qur’an.

Very occasionally there would be contact with the wider movement via a trusted messenger whom Omari refused to name, but who travelled every few months between the leadership in Quetta and Omar’s hiding place.

At first the messenger carried cassette recordings of messages from Omar, but after he was picked up by Pakistani intelligence and questioned, Omar reverted to simply telling the man what he wanted conveyed to the leadership.

Although statements were issued in his name every Eid, Omar took almost no active role in the Taliban after delegating power in 2001. He appears to have functioned more as a spiritual leader, being called on very occasionally for counsel when the movement was split.

Perhaps most significantly he was asked about plans to open an office in Doha that would allow the Taliban to attempt peace talks with the US and other negotiators, and gave the project his blessing.

The only other recorded attempt at intervention was a call in 2007 to the Taliban leader in Helmand province to release two men accused of killing his brother. It is not clear if the message reached the Helmand leader and he defied it or if it arrived too late or got lost in transit, but the two men were beheaded.

Omar fell ill in 2013, coughing, vomiting, and eventually losing his appetite, but he refused any form of medical care. Omari offered to bring him a doctor or drive him to Pakistan, but Omar appeared resigned to his fate and died on 23 April.

Omari buried him that night, and made a video to show to Omar’s son Yaqub and half-brother Abdul Manan. They had not seen Omar since 2001, but travelled to his hideout several days later and insisted on opening the grave to confirm it was him.

They confirmed that it was Omar’s corpse buried in a simple grave in a remote corner of Zabul, but it would be two years before they admitted to their own fighters, and the rest of the world, that the one-eyed ascetic who once defined their movement had died.