London, England (CNN) -- Britain is facing an increasing threat from lone home-grown terrorists, especially among radicalized Muslims in the prison system, a report from a security think tank said Friday.

There is the potential for a "new wave" of terrorism in Britain that is shifting from large-scale bombings requiring considerable amounts of training and operational support toward lone individuals carrying out attacks on a smaller scale, according to the report from the Royal United Services Institute.

"Rather than sending out trained 'cell leaders' to conduct preparation for sophisticated operations, AQAP (al Qaeda in the Arabian Peninsula) and other related organizations have recently been content to send out a higher number of lone individuals (or at least lightly supported ones) whose chances of success are considerably lower, but whose number and presence raise similar public anxieties," the report said. "Eventually, it is reasoned, one of them will be lucky enough to succeed in a major way against high-profile targets in Western countries."

One trend contributing to the "new wave" of terrorism is the escalating rate of radicalization of Muslims in the British prison system, which the report says may produce as many as 800 "potentially violent radicals."

"Jihadist radicalization is believed by the prison authorities to be taking place at a rapid rate, especially in the eight high-security institutions where most terrorist prisoners are kept," the report said.

Prison probation officers believe that around one in ten of the 8,000 Muslim prisoners in high-security institutions in England and Wales are successfully targeted, the report said. These prisoners were not previously convicted of terrorism offenses, it said.

"The prison chaplaincy, including its Muslim adviser, told a parliamentary committee last year that the situation was becoming progressively worse," the report said. "It remains to be seen how much of this 'new radicalism' may find actual expression in violent acts or attempts at recruitment."

The use of more lone individuals in carrying out attacks can be seen in the case of Yemeni-American cleric Anwar al-Awlaki, suspected in the failed bombing of a Detroit-bound passenger plane on Christmas Day last year, the report said.

The U.S. Treasury Department last month designated al-Awlaki a key leader for al Qaeda in the Arabian Peninsula, and he is believed to be hiding in Yemen.

The report also cited the failed Times Square bombing May 1 in New York, to which 30-year-old Pakistani-American Faisal Shahzad has pleaded guilty.

"This new breed of jihadists have only tenuous links to any major terrorist organisation and receive little training and few resources, increasing the difficulty for the police and intelligence services in tracking and intercepting them," the report said.