Quick: name the most memorable moment from the three presidential debates between Barack Obama and John McCain?

Nothing coming to mind? (And no, Sarah Palin saying "you betcha" in the vice-presidential debate doesn't count.)

How about the 2004 debates between John Kerry and George W Bush? Well, there was that line about a "global test" and then that great internet rumor that Bush had a radio receiver on his back. But anything else ring a bell?

This isn't just a memory test for Guardian readers – it's a perhaps less-than-subtle reminder that even though presidential debates are often viewed as the most important part of a campaign (aside from the national conventions), they actually affect the final election results a lot less than you might think. If Mitt Romney is hoping that the debates turn the tide in what is an ever-widening polling gap between him and Barack Obama, he might want to think again.

The idea of debates being a less consequential campaign event will likely seem counter-intuitive to many. For weeks, these TV setpieces have been hyped as the signal events of the fall campaign. And doesn't history provide many lessons of game-changing debate moments?

Was it not Richard Nixon's five o'clock shadow that doomed him against the much smoother John F Kennedy in 1960? Or what about Ronald Reagan's famous quip in 1984, when asked if he was too old to be president:

"I will not make age an issue of this campaign. I am not going to exploit, for political purposes, my opponent's (Walter Mondale) youth and inexperience."

Or what about Michael Dukakis's robotic response to a query about whether he would still oppose the death penalty for a rapist-killer if his wife, Kitty, had been the victim?

These are just the sort of game-changing moments that allow the Romney campaign a flicker of hope.

But not so fast. While political pundits love to impart great meaning and importance to these events, political scientists aren't so convinced. As John Sides, a political scientist at George Washington University, wrote recently in Washington Monthly:

"Scholars who have looked most carefully at the data have found that, when it comes to shifting enough votes to decide the outcome of the election, presidential debates have rarely, if ever, mattered … in the average election year, you can accurately predict where the race will stand after the debates by knowing the state of the race before the debates."

Debate gaffes, like your garden-variety campaign gaffes, rarely, if ever, move public opinion polls or affect the final outcome of elections. Even if Dukakis had given a more emotional response to the death penalty question, or if George Bush hadn't been caught checking his watch during the 1992 debate, both candidates were still likely to lose.

Reagan's zinger against Mondale was highly effective, but incumbents who win 49 states don't need a great debate line to ride to victory.

The first of two possible exceptions to the general rule of debate irrelevance is the 1980 debate between Ronald Reagan and Jimmy Carter, which came only days before the election. While Reagan held a small lead going into the debate, by the weekend before election day, polls showed him approaching a landslide-like advantage. The debate, rather than "changing the game", simply confirmed the direction the polls were already headed – albeit in a decisive manner.

Reagan, who was regularly portrayed by Carter in the most negative terms imaginable and, in particular, as a war-monger, came across as cool, calm and collected – anything but a loose cannon. If Americans needed reassurance that they could trust Reagan in the White House, the debate clinched it.

In 2000, Al Gore's constant sighing during his first presidential debate with George W Bush turned him into an object of ridicule (even Saturday Night Live poked fun at Gore's tic). According to Sides, Gore's unfortunate show of contemptuous impatience may have shifted the polls by two to three points in sympathy with his opponent, Bush. Considering the narrowness of Gore's eventual "defeat", the debate could be considered one of several decisive factors.

But for those two examples, there is always the experience of John Kerry, who, in 2004, won not one, not two, but all three presidential debates against George W Bush, according to post debate polling – and still lost the election. Quite simply, Kerry's strong performance couldn't cancel out the advantages of his opponent. Or, still more importantly, while the debates gave him a boost in the polls, they didn't have the effect of fundamentally reversing the trajectory of the race. This is perhaps the most important factor: debates might affect the polls around the margins, but they have never had a seismic impact on a presidential campaign.

And this brings us back to Mitt Romney. If the latest public opinion polls are to be believed, the Romney campaign is in serious and deepening trouble. Not only is he trailing in the national polls, but he is also losing badly in swing state polls from Ohio, Virginia and Florida, each of which suggests that Obama is opening up sizable, even insurmountable, leads.

No matter how well Romney does in his debates with Obama, what reason is there to believe that he can do well enough to reverse this increasingly dire trend? Have any actions to date by the gaffe-tastic Romney given credence to the notion that he has the political chops to change the direction of the race in these debates? If anything, because of his now almost daily faux pas, even the smallest slip-up has the potential to dominate post debate coverage. The guy is practically going to have to bat 1,000, just to eke out a win.

Moreover, it's not all about Romney. There is even less reason to believe that Barack Obama will make a mistake so grievous and costly that he will open the door to a Romney comeback. There's a reason, after all, people call him "no-drama Obama".

Unless the president uses the debate to unleash a profanity-laced diatribe against the people of Ohio and Florida, it's very hard to imagine anything he could do or say that would badly hurt him. To be honest, even if he were, inexplicably, to do such a thing, I'm not convinced it would cost him the election. Considering the current polarization of the electorate and the dearth of undecided voters – short of Obama's breaking out the legendary "whitey" tape, or admitting that, yes, he was born in Kenya – he's likely safe.

None of this is to suggest that voters shouldn't tune in to the debates. They are an excellent opportunity to more fully appreciate the positions of the two candidates and understand the rather profound differences between the two parties. Just don't expect it to make much of a difference in a race where the outlines of the final results are becoming increasingly clear.