Nearly three years ago, on a night of great history, a slender 47-year-old black man who had just been elected to the nation's highest political office offered the American people an optimistic vision for the country's future. Quoting Abraham Lincoln, Barack Obama spoke of national unity: "We are not enemies, but friends. Though passion may have strained, it must not break our bonds of affection."

That night, Obama offered the American people a clear sense of his overriding priority as president – it wasn't just to fix the ailing US economy, provide healthcare for all or end the war in Iraq. But rather, after eight years of political turmoil and disunity, through the force of his personality and political temperament, Obama would, as Lincoln said, "bind up the nation's wounds".

Things have not quite worked out as Obama planned. Even with poll results suggesting that Americans prize compromise and are tired of overt partisanship, the level of division and acrimony in Washington has grown exponentially since Obama took office. The recent debt limit debate is the apogee of Washington's dysfunction: and indicative of a political system that is seemingly incapable of dealing with national challenges. Indeed, whatever one may think of Standard & Poor's recent downgrade of US debt, the ratings agency view that "the effectiveness, stability, and predictability of American policy-making and political institutions have weakened" seems almost self-evident.

How has America been reduced to one party holding a gun to the US economy and the other trading away its political principles to stop the trigger from being pulled? The problem is that the US today has one party intent on utilising government resources as a force for social good and another that rejects any significant role for the public sector. Compounding this collision of ideologies is a populace so indifferent to the workings of their own government that they are unable to choose which model they prefer.

This clash over the proper role for government in the US is one that has defined our national politics for more than a century. But in the last several years this conflict has become an existential one, with Republicans basically abdicating their responsibility to govern. When in power they made little effort to deal with the nation's many challenges. In opposition, and particularly in the two and a half years since Obama took office, they have used the tool of the Senate filibuster and various other procedural impediments to try to stop nearly all Democratic initiatives in their tracks. Whatever legislation passed in the past few years is almost solely a product of Democratic cohesion (an attribute that is generally in short supply) – and a brief window in which Democrats enjoyed a filibuster-proof majority in the US Senate.

From this perspective, threatening economic cataclysm in order to further reduce the size of government, by refusing to raise the debt limit, now seems like an inevitable step in Washington's scorched earth politics. That it forced Democrats to agree to trillions in painful spending cuts without any commensurate revenue hikes shows how successful this strategy of policy extortion can be.

So why do Democrats put up with it? They have little choice. The American political system discourages radicalism and relies on compromise. Yet the violation of even the most customary rules of governance has made such deal-making now nearly impossible. It was once considered a given that, with the rarest of exceptions, a president would be able to appoint his own charges to key policy-making positions; and the debt ceiling was considered an occasionally politicised but generally pro forma exercise. No longer. In a system designed around collegiality, Democrats have few tools in their arsenal to combat the GOP's political obstinacy.

As a result, America is increasingly moving toward a parliamentary system in which politicians, rather than voting along regional lines or in pursuit of parochial interests, cast their ballot solely based on whether there is a D or R next to their name. Such a system might work well in the UK, but in the US, with its institutional focus on checks and balances and the many tools available for stopping legislation, a parliamentary-style system is a recipe for inaction.

What compounds the Democrats' challenge is that they are the party of activist government. When in opposition, they find it hard to use the Republicans' jamming techniques; when in power they feel the almost quaint need to act responsibly. Any scent of scandal or illegitimate behaviour that undermines the electorate's confidence in government in turn undermines the Democrat's brand.

Case in point on the debt: liberals far and wide urged Obama to consider invoking the 14th Amendment of the US Constitution, which suggests that the country's public debt must continue to be paid. As the left argued, this would have resolved the crisis. In a worst-case scenario, Republicans might impeach the president in the House of Representatives, but, as the argument went, wouldn't this be a good way to rally the country around Obama? But such tactical recommendations miss a crucial element; if low-information voters (the vast majority of Americans) look to Washington and see the nation's political leaders arguing about impeachment and constitutional crises that have little connection to their own lives it exacerbates their lack of confidence in government. For many conservatives it would only confirm their irrational belief that President Obama is a power-hungry tyrant.

Thus for Democrats, gridlock is their most pernicious enemy; a point Republicans understand all too well. The more they stop government from operating effectively the more it emphasises their key political narrative that there is no reason to have any confidence in public institutions. Tom Schaller, a political scientist at the University of Maryland, said to me that Republicans understand that if you have a vat of sewage and you pour in a glass of wine you still have a vat of sewage. But if you pour a glass of sewage into a vat of wine, guess what, you now have a vat of sewage. In short, a little political poison can go a long way. So while liberal complaints that President Obama is far too solicitous of Republicans and far too wedded to his post-partisan agenda (all probably true), a reversion to bare-knuckled politics is not necessarily going to make things any easier or better for progressives.

As the Democratic pollster Stanley Greenberg noted in the New York Times, "voters feel ever more estranged from government" and "they associate Democrats with government". More crises in Washington are not going to help that process.

Why do voters put up with such a situation? Polling suggests that the electorate wants their leaders to focus on jobs, rather than the deficit; and work toward compromise, rather than gridlock. So why then do they reward political parties, such as the Republicans, that act decidedly against not only their preferences but also their interests?

The answer lies in the apathy of the American people toward their own government. The ultimate check on Republican nihilism would be voter revolt. But in the last congressional election, voters rewarded unprecedented Republican obstructionism with control of the House of Representatives.

What's worse, voter preferences are often contradictory. Polls suggest that the electorate wants political leaders to cut spending, but then also demand no cuts in any government programme that isn't foreign aid. They want Congress to focus more on creating jobs, but recoil at policies, such as the bailout of the US auto industry or the stimulus package, that did just that. One problem is that Americans have been so inundated with anti-government rhetoric over the past 40 years they seem to have trouble identifying any link between government engagement and a robust economy.

Worst of all, Americans may prefer Democratic policies, but they have little confidence in government's ability to fulfil those promises and then blame both parties for inaction. They are so mistrustful of government and shockingly uninformed about its working that, perversely, via the ballot boxes, they directly contribute to the political stalemate they so regularly decry.

The result is a political system that is perhaps more incapacitated than at any point in modern history. Across the US, states have to cut social services and benefits because they are receiving no support from the federal government. Infrastructure is crumbling, millions of American students are trapped in under-performing schools, the existential threat of climate change is off the political radar screen and job growth is barely on the agenda. Even the most recent agreement to cut the bloated federal deficit does virtually nothing to deal with the greatest driver of national indebtedness – healthcare spending. What all of this suggests is that the episode played out over the past few weeks of one party threatening to plunge the nation into economic catastrophe is not some rare event – it's the new norm in American politics. And perhaps the most glaring indication that Barack Obama's vision of new post-partisan America will be a dream perhaps permanently deferred.