Britain’s most senior counter-terrorism officer has said the police and security services are no longer enough to win the fight against violent extremism, and the UK must instead improve community cohesion, social mobility and education.

In his first major interview since taking up his post last year, the Metropolitan police assistant commissioner Neil Basu told the Guardian that up to 80% of those who wanted to attack the UK were British-born or raised, which strongly indicated domestic social issues were among the root causes.

Grievances held by people who were “malleable” to terrorist recruitment were highly dangerous, he said, calling for sociologists and criminologists to take a leading role in helping police tackle the problem.

Basu, who is highly regarded in Whitehall, is seen as a potential next head of the Met. His comments are a significant break in tone, if not strategy, about how to combat terrorism and prevent it from becoming a multigenerational struggle that damages the UK’s social fabric.

Basu said Prevent, which he sees as the most important plank of Britain’s counter-terrorism strategy, had been “badly handled”, but its work was vital and had to become more transparent and community led.

He also said extreme rightwing terrorism was rising, with an increase in nationalism since the Brexit vote potentially fuelling violence.

Giving a personal view on the best ways to reduce terrorism, Basu said: “Policies that go towards more social inclusion, more social mobility and more education are much more likely to drive down violence … than all the policing and state security apparatus put together. It is much more likely to have a positive effect on society.

“The prescription for me is around social inclusion – it’s social mobility, it’s education, it’s opportunity.”

Basu said counter-terrorism operations increased by 50% from 2015 to 2017 and have since remained at a high level. The terror threat is still severe despite Isis losing territory in Iraq and Syria.

Both Islamist and extreme rightwing terrorists have continued to recruit Britons, despite efforts to thwart them. “Nothing I am saying remotely excuses these heinous acts of criminal violence,” Basu said. “But the deeper causes need examining. My teams are world class at stopping attacks and locking terrorists up. But we need to stop the flow of recruits into terrorism.

“Don’t forget that 70%-80% of the people we arrest, disrupt or commit an attack here, are born and raised here. Born or at least raised here. That has got to tell us something about our society – that we have got to look at why they would be prepared to do that.

“I want good academics, good sociologists, good criminologists … to be telling us exactly why that is.”

Basu accepted many people went through negative experiences without ever dreaming of committing violence, and some terrorists came from middle-class backgrounds and seemingly wanted for nothing.

But he said some people were more “malleable” than others to terrorist recruitment and there were common themes. “It might be everything from high anxiety, to lack of confidence, lack of education, things that may have happened to them when they are young, bullying, racism, bigotry, lack of opportunity, early experiences with law enforcement even, domestic violence,” he said.

The counter-terrorism network Basu heads had been stretched by the high volume of terrorist activity from Islamists. On top of this and rightwing extremism, he said there was a growing threat from states such as Russia, following the Salisbury novichok poisonings.

Forensics officers remove the bench where the former Russian spy Sergei Skripal and his daughter were found in Salisbury. Photograph: Will Oliver/EPA

Basu said there was no one path that led to terrorism and a list of factors could result in violence if not checked at some point. “All of those things will be as relevant to a terrorist cause as they will be to other people of violence in other crime types,” he said.

Policies were up to the government, he said, but they must tackle “education, access to health, not disproportionate outcomes in criminal justice, feeling like you’ve got an opportunity to get on in life”.

Basu added: “These are wider societal problems. They are not paying more police and more security services to stop more terrorist attacks. That’s not the cure for this. Like every other aspect of law enforcement, we [counter-terrorism policing] are a suppression tool for a problem. We are dealing with the symptom and we do need to deal with the root causes of it.”

While the majority of the terrorism threat was from Islamist extremists, far-right propaganda could help create a permissive environment for some to commit violence, he said, and society needed to determine how much of that rhetoric was acceptable.

“At the moment, we seem to be accepting a level which I think is potentially breeding some intolerance,” he said. “That intolerance, for a small number of people, can spin up very quickly to a violent act, and we have some examples of that. We have some very awful examples of that.”

He said that despite its importance, Prevent had been the least successful part of the UK’s counter-terrorism strategy so far, compared with the other three strands – Pursue, Protect and Prepare – which were all “outstanding”.

Prevent, which critics have called a “toxic brand” , needed “better communication, more transparency [and] an ability not to create a vacuum for people to attack it, by not actually trying to defend it”.

Asked whether Prevent, in its early years, when headed by a former senior intelligence officer, had come across as turning a community into a security issue, Basu said: “Not when it started. It morphed into that. It started off as a safeguarding, vulnerability programme. It was, in my view, badly handled.

“This won’t be won by government or by people like me. It will be won by people who walk into community halls up and down the country and explain.”

Basu rejected notions that British Muslims should “assimilate” and defended the rights of religious conservatives of all faiths, saying: “Assimilation implies that I have to hide myself in order to get on. We should not be a society that accepts that.

He added: “You should be able to practise your religion without suffering some condemnation of that; so my view is, do no harm. And that does not matter whether you are conservative Islamic, conservative Christian, conservative Hindu, conservative Sikh. You should be able to practise your culture or religion openly and still be accepting of others, and others be accepting of you. That is a socially inclusive society.”