Afghan intelligence believed Osama bin Laden was hiding in an area close to Abbottabad four years ago – but no action was taken after the claim was furiously rejected by Pakistan's president, Afghanistan's former intelligence chief has said.

Agents working for the National Directorate of Security (NDS), the country's intelligence service, worked out that the world's most wanted man must be inside Pakistan proper, rather than the semi-autonomous tribal areas, as early in 2004, Amrullah Saleh told the Guardian.

He said they believed Bin Laden must be there based on "thousands of interrogation reports" and the assumption that Osama – "a millionaire with multiple wives and no background of toughness" – would not be living in a tent.

"I was pretty sure he was in the settled areas of Pakistan because in 2005 it was still very easy to infiltrate the tribal areas, and we had massive numbers of informants there," he said. "They could find any Arab but notBin Laden."

Their intelligence became more precise in 2007 when they believed he was hiding in Manshera, a town a short distance from Abbottabad where the NDS had identified two al-Qaida safe houses.

But the former spy chief said that Pervez Musharraf, then president of Pakistan, was outraged at the suggestion that Bin Laden was hiding in such a prominent part of the country.

In a meeting with Musharraf and Hamid Karzai the Pakistani president became furious and smashed his fist down on the table. "He said, 'Am I the president of the Republic of Banana?'" Saleh recalled. "Then he turned to President Karzai and said, 'Why have you have brought this Panjshiri guy to teach me intelligence?'"

He said Karzai had to intervene as Musharraf got increasingly angry and began to physically threaten Saleh.

Afghanistan's former top spy – who has long been a hate figure in Islamabad among officials who believed he was implacably anti-Pakistani – also said he had no doubts that Mullah Omar, the leader of the Taliban movement, was hiding in a safe house owned by the Inter-Services Intelligence (ISI), the Pakistani spy agency, in the city of Karachi.

"He is protected by ISI, General Pasha [Lieutenant-General Ahmad Shuja Pasha, director-general of the ISI] knows as I am talking to you where is Mullah Omar and he keeps daily briefs from his officers about the location of senior Taliban leaders, simple," he said.

Saleh was speaking to the Guardian soon after addressing a rally of several thousand Afghans in Kabul organised as a show of strength of what he called Afghanistan's "anti-Taliban constituency" who are alarmed at the prospect of peace talks with insurgents.

The killing of Bin Laden, who was sheltered by the Taliban regime in the 1990s, has prompted heady speculation that an "end game" to the 10-year conflict is now at hand, with the Afghan government and the Taliban-led insurgency striking a deal.

But "deal making" were dirty words to the crowd gathered in a huge tent in Kabul lined with banners saying "We didn't vote for Karzai to make deals" and "Don't sacrifice justice for dealing".

Speeches were interrupted several times by chants from the crowd of "Death to the Taliban! Death to the suicide bombers! Death to the Punjabis!" – a reference to the protesters' view that the Taliban are under the control of the ISI.

Saleh is a burly and comparatively young man who earned the respect of the CIA during his sometimes brutal leadership of Afghanistan's intelligence service. He received a rapturous reception from the flag-waving crowd when he marched into the tent

Saleh lambasted Karzai for calling the Taliban disaffected "brothers".

"They are not my brother, they are not your brother – those are our enemies," he declared, to cheers.

Saleh warned the government that his movement would not remain content with peaceful demonstrations if Karzai did not change course. Later he told the Guardian that if Karzai "sold out in order to bring the Taliban" there would be no choice but to "rise up".

"We have been exposed to a lot of weapons, it is not very difficult to resort to fighting and create influence," he said.