One Republican nerve-centre was located in a small, detached house in a tree-lined area of southeast Denver. Behind the closed curtains, election operatives had for months been shaping the GOP’s political messaging in Colorado, applying the kind of sophisticated data analysis typically associated with their Democratic opponents.

The people working there utilised a database into which they could plug in the name of any Colorado voter and ascertain not just their political leanings but their personality traits, enabling them to micro-target advertising.

The Republican operatives, who were parachuted into the state and did not want to be identified because of the confidential nature of their work, explain part of the GOP’s success in Tuesday’s elections.

From Colorado through to Iowa and North Carolina, strategists involved in successful Republican campaigns claimed the party had finally cracked the data-driven science behind contemporary elections, a realm of political warfare in which Democrats have reigned supreme since Barack Obama’s election in 2008.

The consultants working in Colorado had overseen the messaging for local Republican candidates in the state legislature. Some were considered long-shots, but with votes still being counted on Wednesday, all but one of them appeared on course to victory, compounding Republican Cory Gardner’s successful unseating of the Democratic incumbent senator, Mark Udall.

Cory Gardner wriggled free from Democratic attempts to paint him as an extremist, to unseat Mark Udall in Colorado. Photograph: Rick Wilking/Reuters

Those Republican victories were rooted in turnout as much as voter persuasion. In the lead-up to the election, Democrats had bragged about what they anticipated would be a superior get-out-the-vote effort, modelled on the campaign that secured the election of Colorado senator Michael Bennet in 2010.

But on Tuesday, from the Rocky Mountains to the Atlantic coast, it was Republicans who appeared to have mastered the process of turning out base voters.

In North Carolina, Republicans had 12 staffed regional offices open across the state a year before election day, an operation bolstered by the work of outside groups such as the Koch-brothers funded Americans For Prosperity.

“In 30 years of observing North Carolina politics I’ve never seen the Republicans with such a sophisticated ground game,” said Raleigh-based political consultant John Davis. “They learned from their mistakes in 2012.”

In 30 years of observing North Carolina politics I’ve never seen the Republicans with such a sophisticated ground game

But the advances it made in the mechanics of data-driven electioneering is just one part of a GOP success story that saw the party sweep to victory in eight out of nine key Senate races, regain control of the upper chamber, triumph in crucial gubernatorial contests and expand its control over the House of Representatives.

Democrats point out that the party in control of the White House almost always loses midterm elections and say that Obama’s low approval ratings dragged the party’s candidates down.

The party was also defending an unusually large number of seats, four of which – Montana, South Dakota, West Virginia and Iowa – had been vacated by retiring Democrats who, had they sought re-election, might have fared better than the candidates who ran to replace them

But this was not simply an election that Democrats lost; it was a resounding Republican win. Partly that was because of a concerted effort to rebrand the party on the back of a slate of notably disciplined candidates fighting over the electoral centre-ground.

Hank Schwab of Atlanta shows the campaign hat he’s worn for the past 30 years at a gathering in support of David Perdue. Photograph: Jason Getz/Getty Images

That achievement can be traced back to dozens of Republican primary races in which the party establishment defeated challenges from the kind of deeply-conservative Tea Party candidates who have embarrassed the party in the past.

A symbolic milestone in that effort was reached on 24 June, when Mississippi’s incumbent Republican senator Thad Cochran triumphed in a bitter, protracted primary battle with the fiercely right-wing Chris McDaniel.

If there was a lesson for the GOP in that victory it was that Cochran emerged victorious after his campaign changed course, vigorously pursuing Democratic-leaning African American voters. After Cochran’s win, not a single Tea Party challenger took the Republican nomination for a tight Senate race in 2014.

The resulting Republican crop of candidates was better placed to appeal to moderate voters than the one fielded two years ago, especially as they ditched the conservative cultural wars that marked the George W Bush years in favour of more mainstream campaigns focused on Obama’s unpopularity.

Nowhere was that discipline more evident than in North Carolina, which provided one of the greatest upsets of the night, when Thom Tillis, the Republican speaker of the House, came from behind in the polls to defeat Democratic incumbent Kay Hagan.

Hagan was a well-liked incumbent backed by a vastly greater amount of outside cash than her opponent. But Tillis, who overcame a Tea Party challenge in May, was ruthlessly disciplined, pounding North Carolinians with the simple message that Hagan was a Mini-Me Obama in all but name. He refused to be drawn by Democratic attack ads into any other subject.

Indeed it was Republicans, not Democrats, who made history when it came to gender Tuesday

It was a similar story in Colorado, where Gardner wriggled free from Democratic attempts to paint him as an extremist. On the campaign trail he was doggedly on message throughout the race, sticking to a four-point plan that could have been borrowed from a Democrat running in the mid-West: the economy, education, energy and the environment.

Democrats can, with some justification, cry foul. Both Gardner and Tillis were chameleons, disavowing their previous right-wing positions in a shift to the middle ground. Gardner, for example, was one of the more conservative members of the US House of Representatives, and somehow managed to distance himself from an extreme anti-abortion initiative on the state ballot while actively supporting the federal version.

As speaker of North Carolina’s House, Tillis used his gavel to oversee a dramatic shift rightwards in the state legislature, rendering the state legislature one of the most conservative laboratories for radical policies outside of Kansas.

Both men, however, shed their political skin, rebranding themselves anew for their statewide electorates. Both Gardner and Tillis became advocates of over-the-counter contraception, for example, a cunning manoeuvre that offset Democratic attempts to paint them as anti-women.

Indeed it was Republicans, not Democrats, who made history when it came to gender Tuesday, returning two women senators from states who have only ever sent men to the upper chamber: Shelley Moore Capito in West Virginia and Joni Ernst in Iowa.

Shelley Moore Capito applauds and sings ‘I Gotta Feeling’ by The Black Eyed Peas after winning the Senate seat. Photograph: Tyler Evert AP/AP

Ernst’s success against Democrat Bruce Braley had many of the ingredients found in successful Republicans races across the country. Democrats were pleased when Ernst won her primary in June, believing she could be cast as a wild-eyed Sarah Palin clone who was too radical for the notoriously purple state.

Ernst had adopted conservative positions during the primary battle: she called the president a dictator and said the Environmental Protection Agency should be abolished. But attempts by Democrats and their allies to use use those blunders to tarnish Ernst’s reputation failed.

“There was no nuance. It just rang hollow after a while,” said Michael Brickman, the Iowa GOP’s communications director. In the end it was Braley who was more easily defined by his opponents, who exploited his foot-in-mouth remarks about farmers to characterise him as an out-of-touch and arrogant lawyer.

Ernst’s superior, carefully-choreographed public image, combined with an improved ground operation, were key to her success.

“It’s about embracing technology, putting people in the field and having an inspiring candidate,” Brickman said.