Some American teenagers dream of a glamorous career in the FBI, a chance to shoot guns and catch criminals, but the idea had not occurred to Ali Soufan. At least, not until The X Files. “Mulder and Scully were going round the world looking for aliens,” he says, laughing. It looked fun. Besides, he did not believe he would get in; he was just intrigued by the process. Why did he think they would not accept him? “Well, look at me; I don’t look like an FBI agent, at least I didn’t at the time.” He was an Arab-American and a bit of an intellectual. “I just felt that I wasn’t a law-enforcement guy, that wasn’t what I wanted to do in my life. But I went, I took all the tests.” They offered him a job. Soufan, who had just completed a master’s in foreign relations, thought he would return to academia if it did not work out.

It was 1997 and Soufan was in his mid-20s. Because of his background – he was born in Lebanon and spoke Arabic – he was assigned to the Joint Terrorism Task Force, which was focused on Palestinian and Iraqi groups. But he had become interested in Osama bin Laden while reading Arabic newspapers as a student and in 1998 wrote a memo on Bin Laden for his superiors. It made it all the way up to the head of the national security division, John O’Neill, who would become a mentor and friend (O’Neill later became head of security at the World Trade Center and was killed on 9/11). The FBI and the CIA were already monitoring Bin Laden but Soufan claims “they only looked at him as a financier of terrorism, not as a terrorist operative”.

Soufan, though, predicted that Bin Laden would be “a lot of trouble down the road ... We had to take him seriously. [He was] trying to present himself as a former mujahid who fought against the Soviets and now he’s trying to bring all these mujahideen to do a reconstruction in the Muslim world. He was trying to open highways and farms and all these things. When you see these guys who have a lot of [fighting] experience – many of them cannot go home because they were wanted [in their countries] – and he’s forming some kind of Islamic army, I think, if you know a little bit of history, you know that’s not going to go well.”

In his book, Anatomy of Terror, Soufan traces the evolution of terror organisations, from Bin Laden and 9/11 through the ill-advised “war on terror” to the destabilisation of the Middle East and the emergence of Islamic State and other jihadi groups – and the foreign fighters attracted to fighting for them. The missing link in the west’s counter-terrorism strategies, he says, is “not looking into what these folks believe in. Ideology is the cornerstone of these organisations. That’s why we should not [be distracted] by different names, different groups – this is al-Qaida, this is Daesh – I think we have to go into the glue that pulls these things together. It is ideology and we have to deal with that. If we don’t, we will continue to suffer for years to come.”

I suggest we have to have empathy – understanding the enemy, seeing the world through their eyes

He sums up the ideology as “hijacked terminology from Islam”, a narrative that “successfully peddles the idea the entire world is engaged in war against Islam, that anyone who does not believe in the ‘us versus them’ narrative is not to be trusted. We can tackle that message by exposing the basic hypocrisy of this movement that claims monopoly to the purest Islamic piety even as it routinely bombs mosques and marketplaces and kills other Muslims – more than 90% of their victims are Muslim.”

Terrorism thrives in chaos. Soufan points out that stage one of al-Qaida’s strategy, laid out in the jihadi handbook Management of Savagery, is to create, or take advantage of, regions of chaos or “savagery” and move in to fill the vacuum. Al-Qaida is playing the long game, he says; Isis was too quick to create its caliphate and it could not hang on to it. “Before we only had one vacuum, in Afghanistan,” says Soufan. “They were operating from there and spreading their message. Now we have so many vacuums – Syria, Yemen, Libya, northern Nigeria, Tunisia, the Philippines – and it’s expanding. That’s very dangerous. We have to pay attention to these kind of things.”

We meet in a hotel suite in London. Soufan – warm, smiling, quick to laugh – seems far too cheerful for a person who has spent his career witnessing brutality. His dark suit is nondescript, but his boots are the most pristine I have ever seen. (Soufan is no stranger to a bit of glamour: he produced the TV series The Looming Tower, based on Lawrence Wright’s Pulitzer-winning book. It stars Jeff Daniels as O’Neill and Tahar Rahim as Soufan and follows their stories.)

He was an FBI agent for only eight years, but it was a tumultuous time – he investigated the US embassy bombings in Kenya and Tanzania and led the investigation into the suicide attack that tore a hole in the warship USS Cole, killing 17 sailors and injuring many more. At the time of 9/11, Soufan was one of only eight Arabic speakers in the FBI and the only one in New York. He interrogated suspects at Guantánamo Bay and foiled terror plots. Later, he left the FBI, disillusioned and considered a bit of a trouble-maker after speaking out against CIA interrogation methods. Soufan now runs an intelligence consultancy called The Soufan Group.

Facebook Twitter Pinterest John O’Neill (Jeff Daniels) and Ali Soufan (Tahar Rahim) in The Looming Tower. Photograph: JoJo Whilden/Hulu

He remains critical of much of the US’s foreign policy. The Iraq invasion, he says, “was a colossal mistake ... we should have focused on Afghanistan, on al-Qaida and the Taliban. We should have swiftly brought them to justice. By 2003, al-Qaida was a dead breed; it was the invasion of Iraq that gave them new oxygen. Imagine if all these trillions of dollars we wasted was focused on rebuilding Afghanistan, creating better education, better economic opportunities in areas where al-Qaida and Isis are recruiting. I think the world could have been a better place.”

He says the west has failed in the Middle East because there is no strategy. “In Afghanistan, we had no idea what to do after the military victory. It’s the failure of imagination. We always say we could not imagine it would take fewer troops to take out Saddam [Hussein] than it would take to secure Iraq after Saddam; we could not imagine somebody flying a plane into a building; we could not imagine we’ll be in Afghanistan for 17 years. Our imagination is very limited, limited by our perceptions, knowledge, experience, partisan politics. That is why I suggest we have to have empathy – understanding the enemy, seeing the world through their eyes. This is extremely important, knowing them on a deeper level, so we can start predicting their moves.”

He says Donald Trump’s wish to get out of Syria may be a good idea, but only if a regional agreement is made – “and that has nothing, I hate to say it, to do with the Syrians themselves. It is a mini Yalta [conference] between Turkey, the US, Iran, Russia, Israel, the Gulf states, because everybody is fighting for their own piece of the pie. I think it’s good to get out of meddling with other people’s problems, but we need to figure out what we’re leaving. Do we surrender Syria to Russia and Iran? What’s going to happen to our allies in the region?”

Facebook Twitter Pinterest Ali Soufan’s book, Anatomy of Terror. Photograph: Ali Soufan

Soufan says the threat from Isis “is not over. It will take a different shape and I won’t be surprised if we start to see an alliance between Isis and al-Qaida in different areas in the Middle East. Al-Qaida is thinking in long-term strategy. They’re building alliances.” He believes “al-Qaida is stronger today than they were before 9/11. They are focusing locally, but even if a small portion of these guys decides to go global again I think we’re going to have a big problem.”

In the west, he says, “one of the things we really need to focus on is building bridges with communities that might feel isolated because of the counter-terrorism strategies. Our strongest asset in defeating [the Islamist terrorism] narrative is our western values.” Values of inclusiveness and tolerance. Values that have been rocked by the rise of rightwing extremism. How much of a problem is it? “Big. It’s huge. Look at antisemitism in the US, in the UK, in France. Hate crimes are on the rise. It’s part of an agenda that is going around the western world. They call themselves the ultra-right; I call them neo-Nazis.”

He is critical, too, of the UK’s counter-terror strategy, Prevent, which has focused on Muslim communities. “The policy can be successful or not by the way it is perceived. That’s why it should be very clear from a branding perspective that we’re not only talking about Muslims, we’re talking about all sorts of radicalisation.”

Soufan grew up in Lebanon, where his father was a journalist, in the grip of civil war. “When there was bombing, you hide,” he says. “If there is shelling, it depends how heavy, but you’ll go to the basement or the stairway. I thought it was very normal until we moved and you realised: ‘Wow, that wasn’t normal at all.’” He remembers two rival militias fighting outside, with the family’s building trapped in the middle. “They were fighting all day and we were stuck in the stairway and you could hear the RPGs, the guns. Then one of the militias brought a tank and they start hitting towards the others.” All the doors of the building were blown in by the echo of the tank fire. Did he lose family members and friends? “Friends,” he says. “People in school.”

People who were involved in torture are coming back to run the show, to rewrite history, and I think that is a problem

At 16, the family moved to the US. He says he loved it – and still does. “I know we’re going through a harsh period now, but it will continue to be the shining city on the hill. The lights might be a little bit dim now, but we’ll put them on again. We have no other option.”

Soufan was in Yemen, investigating the USS Cole attack, when 9/11 happened. It became clear the two attacks were linked. It also soon became clear that the CIA had withheld information Soufan had asked for repeatedly, information that might have helped prevent the attacks on the World Trade Center and the Pentagon. “Unfortunately, we still have no accountability for that,” says Soufan. “Not one single person was held accountable.”

Could he have done anything differently? “Sometimes you wake up in the middle of the night and think about it. But you know …” There is a long pause. “I’ve been thinking about it for many years. I can ask for the information and I did, many times, on paper. But if people are telling you: ‘We don’t have anything,’ you have to take it on face value. I hope, as institutions, we learned from it. Information that is essential to stop bad people from killing innocent people is not something you or your institution owns, it’s for the American people.”

In 2002, Soufan was brought in to interrogate Abu Zubaydah, the US’s first “high-value detainee” captured since 9/11, who allegedly had links to al-Qaida. Zubaydah gave them intelligence that stopped a terrorist attack. “He’s an interesting figure, because I was really surprised how fast he cooperated,” says Soufan. Several months later, a CIA team was brought in to perform what was euphemistically termed enhanced interrogation techniques (EIT) on Zubaydah, even though Soufan said they had already got out of him everything they could. It was lower-level techniques – enforced nudity and sleep deprivation – but Soufan complained. (Zubaydah was later waterboarded repeatedly.) “That was why I was pulled out of the black site and the FBI was pulled out of the interrogation. That actually put a big target on my back.”

Gina Haspel confirmed as CIA director after key Democrats vote in favor Read more

Soufan says he complained for several reasons. First, he believed EITs did not produce good information (he likes to point out that the lie that Iraq had WMDs, used to justify the invasion, was produced as a result of torture). Also, torture helps terrorist recruitment – images from Abu Ghraib and Guantánamo Bay “still help radicalise young men around the world” – and is fundamentally “not who we are”.

So, how does he feel about Gina Haspel, who ran a covert CIA detention site during the “war on terror”, becoming director of the CIA? “She has a lot of baggage and that is going to be used against her,” he says. “People who were involved in torture are coming back to run the show, to rewrite history, and I think that is a problem. That’s a page we thought we had turned over and now, with her nomination, they are insisting on turning back the page, trying to erase it and write something else.”

Soufan lasted at the FBI for another two years following his experience with Zubaydah, eventually leaving in 2005. Did he worry about the effect of complaining on his career? “Absolutely – and that’s why I left eventually, because it became unbearable. You’re supervising a counter-terrorism team and every time you want to do something the CIA say no. For the sake of my team …” He pauses. He was the problem? “Yes, specifically me. That’s fine, that’s what I believed in. It was the right thing to do and I’m glad I did, because, as I predicted, this became public.” Soufan later testified at a Senate judiciary committee hearing on torture. “Imagine if I was part of it and the FBI was part of it,” he says, leaning forward in his chair. “I don’t think we would have a very good position in the US to talk about human rights and values and principles.” He seems optimistic that, one day, we can all be better.

Anatomy of Terror: From the Death of Bin Laden to the Rise of the Islamic State by Ali Soufan is out now (WW Norton & Co, £13.99). To order a copy, go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders, minimum p&p of £1.99.