In his final State of the Union address, President Obama had a few words on how we ought to change the political system. First on his list? Gerrymandering. “We have to end the practice of drawing our congressional districts so that politicians can pick their voters, and not the other way around,” he said.

This point has become almost de rigueur when people list the ills of American democracy, especially among Democrats who feel that Republicans have drawn district maps to give them an unfair advantage in the US House of Representatives. The solution seems so simple: Why can’t we just have independent, nonpartisan districting commissions who can take the politics out of line drawing?

If only it were that easy.

The bigger problem is with single-member congressional districts, which elect only one representative to legislative office. But first, let’s talk about why gerrymandering is not quite the evil it’s made out to be.

In general, there are three objections to gerrymandering:

1. It creates ugly-looking districts

Yes, state redistricting commissions draw some ugly-looking districts. But then again, so did our forefathers. Consider the state of Rhode Island.

Certainly, you can get a computer to make many districts that look more compact than they currently do. But why should we prize aesthetics in congressional districting? This is an entirely arbitrary standard that does nothing to make elections more fair. In fact, some of the most-often ridiculed districts were drawn by public interest groups in order to uphold the goals of minority representation enshrined in the Voting Rights Act.

2. It reduces competition

A more widespread objection to redistricting is the claim that gerrymandering has reduced competition in American congressional elections.

Certainly, fewer and fewer House elections are contested. By the latest estimates of the Cook Political Report, only 16 (out of 435) House seats are considered toss-ups. Most House seats are pretty solidly safe for one party or the other.

But is this because of gerrymandering?

It’s possible that a few states have made some seats less competitive through their redistricting processes. But the reason that most seats are not competitive anymore has far more to do with the geographical sorting of the two parties.

Consider the state of Washington. Here are the presidential results from 2012.

The urban areas on the coast are solidly Democratic. The rural areas inland are solidly Republican. If map-drawers wanted to draw districts to maximize competition, perhaps they could find a way to split the state into ten horizontal bars. But inland and coastal Washington are very different places. To the extent that we want representatives who have some connection to the place they represent, this would be a very dumb idea.

Washington State is not an outlier, just a clear example. In general, urban areas are solidly Democratic; suburban and rural areas are solidly Republican. This poses serious problems for electoral competition.

This decline in competition didn’t happen because of gerrymandering. It happened because the two parties became more clearly sorted, both ideologically and geographically. And as a result, both swing voting and split-ticket voting have declined considerably. In many states, there’s just no way to draw many competitive districts without drawing many skinny rectangles.

This geographical problem winds up being particularly problematic for Democrats, who over-concentrate their voters in urban areas, giving Democrats more districts in which 80% or more of voters are Democrats. By comparison, Republican voters control more districts in which 60-70% of voters are Republicans. Because Republican voters are spread out a little better across more districts, the party can win more congressional seats.

3. Parties draw the map to unfairly benefit themselves

In partisan redistricting, the goal is to get the most seats. That means making the most economical use of your voters, trying to spread them as widely as possible. Certainly, both parties do this.

But both because Republicans control more state legislatures now, and because of the geography of Republican voting (remember, Republican voters are more evenly distributed geographically), Republicans have more successfully drawn themselves more seats than they would under a proportional representation direct translation of votes to seats.

Political scientist Nicholas Goedert has found that in Republican gerrymandered states, Republicans outperform their expected share of seats by 19 percentage points, whereas Democrats outperform their expected share of seats by only 5 percentage points. So this criticism is accurate.

But there is an upside to partisan gerrymandering—it makes seats more competitive than they would otherwise be. Since the goal of partisan gerrymandering is to spread out your party’s voters more evenly, this produces more competitive districts.

The real problem is single-member districts

In drawing district lines, there are many trade-offs. It’s really hard to have districts that are both compact and competitive. Partisan redistricting generally makes districts more competitive than they would otherwise be, but this hardly absolves partisan redistricting of other distortions.

Ultimately, though, competition has to be the biggest concern. Absent competitive districts, the parties will have little incentive to move to their middle and so will move to their extremes, particularly as their geographical centers of power grow further apart.

The only way we’ll solve this problem in the current partisan polarization is to get rid of the single-member district, and move to multi-member districts—wherein more than one representative can be elected to office.

Reihan Salam, the executive editor of the conservative publication National Review, once called the single-member district “The Biggest Problem in American Politics.” Salam is a Republican living in liberal Brooklyn. Under single-member, winner-take-all systems, he will never be represented by a Republican. But if five New York City districts merged into one in order to elect five representatives, a Republican could plausibly get elected to represent the 20% or so Republicans who occupy New York City. That New York City Republican would also be a different kind of Republican than a Republican from Texas or Alabama, creating more diversity within the Republican Party.

Use this plan across the country, and you’d get more electoral competition and more diversity within the parties—particularly if you add in ranked-choice voting, in which voters can list candidates in order of preference. And no more worrying about gerrymandering. To understand better how this would all work, I recommend checking the website of FairVote, which has been a leading advocate for proportional representation and multi-member districts.

It’s easy to complain about the process of gerrymandering, in part because the solution seems so simple—just take it out of the hands of politicians. But the real problem here is not that districts have funny shapes. The real problem is that voters are sorted by partisanship and geography in ways that make it difficult to draw competitive single-member districts. To solve that problem, we’re going to need to get more creative.

This piece was originally published in New America’s digital magazine, The New America Weekly. Follow him on Twitter @leedrutman.