If you asked Americans to identify the most noticeable change in U.S. politics over the past two decades, they’d probably answer that politics has become more polarized and that this has made it harder for the government to address the problems the country faces.

To say politics has become polarized is another way of saying that the politicians we nominate and elect have moved away from the ideological center, that the Democratic Party has become more liberal and the Republicans more conservative, with little or no overlap. Liberal Republicans are all but extinct, and conservative Democrats aren’t far behind. Genuine bipartisan compromise has gone from standard practice to quaint anomaly. In Washington and increasingly in state capitals, once a majority of the party in control of a chamber decides what it wants to do, everyone else in the party is expected to line up behind it — and everyone in the other party lines up to oppose it.

Public opinion polls consistently report that Americans aren’t happy with these developments — they don’t like partisanship or gridlock — and that their views on issues are closer to the center than to the extreme positions in either party. And it’s not just the voters. Politicians themselves are frustrated at not being able to get things done; they chafe at their loss of independence and public respect; they loathe the endless fundraising needed to wage unending partisan warfare.

So if voters and politicians don’t like it, why does this polarization persist? In a democracy, like in any open market, having everyone pursue their own self-interest is supposed to generate the best outcome for society. What is causing this political market failure?

In the vision of politics that many of us carry around in our heads, it is the “median voter,” at the center of the ideological spectrum, who ultimately is supposed to determine the long-term course of government policy. In this model, the best way — the only way — for a party to increase its political market share is to moderate its views to attract such independent swing voters. When either party has tried a different strategy (Barry Goldwater in ’64, say, or George McGovern in ’72), it has failed.

But something fundamental seems to have changed in the political marketplace. The winning strategy is no longer to be more moderate than your opponent, to offer a bigger tent. Instead, it is to be more zealous and committed to your party’s ideology.

This transformation has its roots in what has become the dominant reality of American politics: the arms race in campaign finance. Candidates and parties now raise and spend enormous sums, well beyond what would reasonably be needed to provide for a well-informed electorate and well beyond what is raised and spent in other advanced democracies.

These days, the average Senate candidate raises and spends $9 million to win election, which works out to slightly more than $4,000 for each day of a six-year term. For the average House candidate, it’s $1.4 million, or just under $2,000 per day in office (including Saturdays, Sundays and holidays). These sums are several times what they were 25 years ago.

Given this dramatic increase in campaign spending by those with the most intimate knowledge of campaigns, and with the most at stake in the outcomes, it’s probably safe to conclude that this spending must work — that it can determine the outcome of close contests. In fact, it appears to work so well that it has now been embraced by a growing legion of “independent” entities with their own fundraising and campaign spending.

And how is the money spent? Anyone with a telephone, TV set or Internet connection has surely noticed that it is mainly used to produce an ever-increasing volume of negative, distorting and ideologically tinged advertising about opposing candidates and parties.

Contrary to what many believe, the central effect of such negative advertising isn’t to move voters from supporting another candidate to backing yours, as Mitt Romney and his allies have discovered during this primary season. The main effect is not even to move undecided voters into your column. No, the real effect of negative advertising is to energize and solidify support among your ideological base while turning everyone else off to the other candidate, the campaign and the entire electoral process. Negative advertising isn’t about changing minds; it’s about altering the composition of the voter pool on Election Day by turning moderate voters into non-voters.

This is particularly true in low-turnout elections such as primaries and midterm contests. But it is even true these days in high-turnout elections.

Peter Hart, the dean of American political pollsters, notes that President George W. Bush won Ohio in 2004by boosting voter turnout among conservatives in exurban and rural areas. If turnout in those areas had been the same as in the rest of the state, Bush would have lost Ohio and the election. And a key to the strategy was massive amounts of negative advertising.

Similarly, in 2008, Barack Obama was able to use negative advertising to move the traditionally Republican states of North Carolina and Virginia into the Democratic column, increasing turnout of reliably liberal voters around Charlotte and in Northern Virginia while dampening enthusiasm for the GOP candidate, Sen. John McCain, everywhere else.

Energizing the base has another important advantage: It increases campaign contributions from both small donors and rich zealots. That money can be plowed back into yet more negative advertising along with sophisticated get-out-the-vote efforts on Election Day. This self-reinforcing cycle creates a strong incentive for politicians to abandon the center and move permanently to the ideological extreme. You do not energize the base through moderation and compromise.

What makes this an effective and rational strategy, of course, is the phenomenon known as “free riding.” When you think about it, it’s pretty irrational for any of us to vote. During these endless campaigns, it takes an extraordinary amount of time and energy to inform yourself about the candidates and their positions. And it takes time and energy to interrupt your daily schedule and vote. And for what? Rarely, if ever, does any single vote make a difference in the outcome. The rational thing is to just stay home and let everyone else do their civic duty while we still enjoy the full benefits of democracy.

Unfortunately, if everyone follows that individually rational strategy, the political market fails, democracy doesn’t work, and we end up with a far-from-optimal outcome. And that is precisely the market failure that shrewd political campaigns seek to turn to their advantage.

There is a vigorous academic debate over whether negative advertising depresses or increases voter turnout. I suspect it does both, depressing turnout among moderates and independents while stimulating it at the ideological extremes. In that process, what has changed is the composition of the turnout rather than its overall level.

So, if this is such a winning strategy, why is it taking hold only now?

The most obvious reason is that American politics has gone through a gradual realignment since the civil rights movement of the 1960s, which finally ended the Democratic hold on the South. The economic decline of the Midwest Rust Belt and the rapid growth in the Sunbelt were also big parts of this process. One result of this geographic shift has been that states and congressional districts became more politically and ideologically homogenous and thus more heavily tilted toward one party or the other — and all the more susceptible to aggressive redistricting strategies. In such an environment, appealing to moderate swing voters has become far less important.

Technology has also played a role. It is much easier today to pursue a strategy of energizing the base and suppressing the moderate vote when you can tailor your messages to different groups. Because of the Internet, sophisticated databases and cable television networks that hew to one ideology or another, such targeting is now easier and more effective.

More recently, the rise of “independent campaigns” run by PACs, unions, business organizations and other special interest groups has also heightened the polarization. These groups tend to air ads that are even more negative and more ideologically charged than the ones from the candidates themselves. And the candidates have been all too happy to reap the benefits while distancing themselves from such tactics.

The irony is that the politicians who prevail in these gladiator contests inherit a system so bitter, so partisan and so ideologically polarized that they can’t accomplish anything. They know that they and their constituents would be better off if they cooperated and compromised more, but they just can’t. If they try, they face a serious risk of being run out of office, either in the next primary by someone who better appeals to the party’s political base, or in the general election by an opponent whose extremism has allowed him or her to energize the other side’s core voters.

Politics has become a tragedy — a tragedy of the commons, that is. The individual pursuit of rational self-interest by parties and politicians, which in political and economic theory is supposed to generate the best outcome, has instead led to a cycle in which extremism, partisanship and stalemate all beget more of the same. We keep thinking it can’t continue like this, but it only gets worse.

Some may complain that this analysis falls into the trap of moral equivalency, failing to note that Republicans practice the politics of extremism and suppression of the moderate vote and that Democrats offer more moderation and compromise. But while it is true that the move away from the political center has been asymmetric and probably began with the Republicans, the success of that strategy has now forced everyone to play the same game.

These days, congressional Democrats and Obama campaign strategists make no secret of their belief that previous attempts at moderation and compromise have not been to their benefit and that they have no choice but to energize their base with a tougher, more left-leaning campaign.

And to those who now expect Romney to move to the center as he shifts from primary- to general-election mode, I’d point out that didn’t happen with either John Kerry or Bush in 2004, or Obama or McCain in 2008. The problem with Sarah Palin wasn’t that she was too ideologically extreme — that part of her selection as McCain’s running mate is still considered a base-energizing, game-changing masterstroke. The problem was that her hard-edged ideology was not matched by a hard-core understanding of the issues.

We know the solutions to escalating polarization: A disarmament treaty for the campaign finance arms race involving spending caps and contribution limits. A ban on campaign spending by independent groups. A requirement that all broadcasters and cable networks provide free advertising time to all candidates. A requirement that everyone vote or face a fine. Transferring redistricting powers from party leaders to unelected, nonpartisan experts. And that hardy perennial, a third-party movement.

Simply to list these ideas, however, is to acknowledge how unlikely it is that the system can correct itself.

Arms races, free riding, tragedies of the commons — these failures in economic markets are well understood. The solutions usually involve some form of government action or regulation. But when similar failures occur in political markets, there are no institutions capable of stepping in and forcing the necessary collaboration or collective action.

Government can’t be the solution when it is the problem.

pearlstein@washpost.com

Steven Pearlstein is a Washington Post business and economics columnist and Robinson professor at George Mason University. This essay is adapted from the Harold Gortner Lecture he delivered at the university on April 16.

Read more from Outlook, friend us on Facebook, and follow us on Twitter.