If you're looking for lessons from last Tuesday's Virginia election results, don't just look to Ken Cuccinelli's gubernatorial defeat – for which any one or combination of a dozen factors can be blamed – look as well at how the Republican candidates performed in the other two statewide races. E W Jackson, a fiery cultural rightist who makes Ken Cuccinelli sound like Wendy Davis, lost by 10.5pt in the lieutenant governor's race. Mark Obenshain, who is about on par with Cuccinelli and Bob McDonnell on social issues, is up by a hair and probably heading for a recount in a 50-50 draw to become the next attorney general. What can we learn?

First, style matters. Three Republicans with roughly similar views on social policy performed very differently: the most flamboyant, Jackson, was crushed; the quietest, Obenshain, did best; and Cuccinelli, in the middle, fell short against Terry McAuliffe.

Cuccinelli's supporters complain that he was defamed by McAuliffe's "war on women" advertising, and Tim Carney, among others, has argued that Cuccinelli was in fact active behind the scenes in opposing the Republican legislature's transvaginal ultrasound mandate for women seeking abortions. The trouble is that whatever the nuances of Cuccinelli's antiabortion views, the image he'd cultivated as Christian conservatism's champion – the very thing that earned him such ardent loyalty on the right – contributed to impressions that what McAuliffe said about his views must be true. Cuccinelli tried to send one signal to his base (I'm with you 100% – whatever that might mean) and another to the general voter (I'm not an extremist). That set up an uncertainty that McAuliffe could exploit, and did.

E W Jackson avoided any uncertainty: he was proud to be an extremist. Jackson was on the ticket because Cuccinelli's supporters changed the nominating procedure from a primary to a convention to guarantee that their man would rout a moderate rival like Bill Bolling, who chose not to run at all. This was overkill – Cuccinelli almost certainly would have won a primary – and it wound up costing the GOP the lieutenant governor's race, if not contributing to Cuccinelli's own defeat. Not only did Cuccinelli not benefit from association with Jackson, but the idea that right-wing activists had taken over the Virginia GOP only reinforced a message that Cuccinelli didn't want to be sending: if his supporters were radicals of a sort that would nominate Jackson, what did that make Cuccinelli?

Style isn't just a question of how a candidate speaks. It's a question of how his supporters present themselves as well. Even if Cuccinelli didn't support the transvaginal ultrasound legislation, he was seen to be the leader of the kind of people who were responsible for it.

A second lesson is suggested by the Obenshain race. The media was abuzz after the election with analysis of why McAuliffe failed to win by a bigger margin. The margin that's most interesting, however, is the virtually nonexistent one in the attorney general contest. In an off-year election, with Obamacare embarrassing Democrats, the best a Republican not identified with "extremism" could do was 50%. That ought to be a wake-up call.

Take it in the context of Virginia's recent history. In 2006, Senator George Allen lost his seat to a Democratic challenger by the narrowest of margins. In 2008, the state went for Obama and chose former Democratic governor Mark Warner over former Republican governor Jim Gilmore in that year's US Senate race. Republicans recovered in 2009 – sweeping the statewide offices with Bob McDonnell, Bolling, and Cuccinelli – but in 2012 the state again went for Obama and again chose a former Democratic governor, Tim Kaine, over a former Republican governor (and senator), George Allen, in a Senate contest. A year later, instead of sweeping the statewide offices as last time, Republicans have lost at least two out of three.

Virginia's demographics are diversifying and Northern Virginia continues to be colonized by federal workers and contractors, but neither trend has accelerated so much in four years to account for the GOP's collapse. A Republican Party that could win in 2009 should have been able to get more than 50% of the vote in 2013.

If the GOP brand has decayed that much in four years, it's a problem that won't just affect Virginia in the near future: as the University of Virginia's Larry Sabato notes, the Old Dominion "has come the closest to the national average in the last two presidential elections and is probably a necessary piece of both party's Electoral College plans in 2016".

Tea Party-vs-establishment tensions nationwide – illustrated by Alabama's first congressional district special election on Tuesday, where a Chamber of Commerce-backed Republican beat a Tea Party one – have led to talk of a Republican Civil War. That's something the Virginia GOP in particular has been fighting for at least 20 years: the first campaign I got involved in during high school was Oliver North's 1994 US Senate bid from Virginia, a three-way race in which North faced opposition from Marshall Coleman, a moderate Republican running as an independent. The Virginia party's establishment-activist split has never been entirely overcome.

Until now, Virginia was culturally Southern enough that Republicans didn't have to decide what kind of party they wanted to be – a hard-right party or a centrist one, the party of Mike Farris or the party of Tom Davis. The GOP could send mixed messages, run a mediocrity like George Allen, and indulge in infighting without losing too many elections. Those days are gone, as the Obenshain race shows. If Obenshain had seemed a little more like Cuccinelli, let alone E W Jackson, he would have lost. Whether a figure like Bolling could have done better than Cuccinelli, meanwhile, is moot: he can't do better if he can't even get the nomination, and Bolling knew it wasn't worth trying.

Both in Virginia and nationally, ideological activists have had the upper hand in the GOP lately, and they've overplayed it. The lesson here isn't that they represent ideas that can't win – Chris Christie shows that anti-abortion Republicans can even win in New Jersey – but that they would win more often if they check their own impulses to let their freak flags fly and hunt the "Rinos" to extinction.

• This commentary was originally published 8 November 2013 in the American Conservative