The UK’s first-past-the-post (FPTP) parliamentary election system is broken and outdated and can no longer be relied upon to keep out extreme elements and encourage moderate, consensual politics, a thinktank has said.

A report by the Constitution Society said an increased concentration of support for the two main parties into defined geographical areas meant there was less and less direct competition between Labour and the Conservatives, negating a need to appeal to the middle ground.

Instead, the report said, the dominant voice of members and the limited viability of smaller parties meant both the Conservatives and Labour had moved away from the centre and were dominated by internal arguments such as Brexit.

This process of polarisation could increase if party members gain more control over the selection of parliamentary candidates, as has begun with Labour and, to a lesser extent, the Conservatives, according to the group, a non-party education foundation that promotes understanding of the British political system.

“In these circumstances FPTP can even abet extreme politics, since should a radical faction gain control of one of the major political parties, FPTP works to preserve that party’s position,” the report said.

“This is because the psychological effect of the plurality system disincentivises a major party’s supporters from voting for a minor party in protest at its policies, since to do so would likely only help the major party’s main rival. Rather than curtailing extreme voices, FPTP today empowers the (relatively) extreme voices of the Labour and Conservative party memberships.”

Though the system was different in the US, it argued, a similar two-party dominance of politics led to the election of Donald Trump. “This is exactly the kind of result FPTP supposedly prevents, and demonstrates that it is in truth no defence against the extremes.”

The FPTP system, under which a single MP is returned from each geographical constituency, has long been criticised for punishing smaller parties. However, supporters argue it provides stable government and encourages moderation.

But the report said British politics was “increasingly balkanised”, with the Tories and Labour retreating into rural and urban strongholds, and nationalists dominant in Scotland, meaning the number of marginal seats had dropped in recent years.

It said the reduced direct competition between the two main parties had eroded their role under FPTP as “the gatekeepers of the political system, and to suppress political radicalism”.

In a 2011 referendum on whether to replace FPTP with an alternative vote (AV) system, almost 68% of people said they preferred to maintain the status quo. But the report said the low turnout and unpopularity of AV with many proponents of electoral reform meant this should not be seen as an endorsement of the current system.

It recommended alternatives such as the two-round system, single transferable vote or the additional member system, with the choice to be considered by a so-called citizens’ assembly.

David Klemperer, a research fellow at the Constitutional Society and the author of the report, said: “Rather than ensuring stable, cohesive politics, first past the post simply prevents parliament from reflecting the social and political divides of Britain today. Political debate now occurs as much within the main parties as between them, reducing their coherence, leading to unstable governments and depriving voters of a clear choice at general elections.”