I first encountered the upper middle class when I attended a big magnet high school in Manhattan that attracted a decent number of brainy, better-off kids whose parents preferred not to pay private-school tuition. Growing up in an immigrant household, I’d felt largely immune to class distinctions. Before high school, some of the kids I knew were somewhat worse off, and others were somewhat better off than most, but we generally all fell into the same lower-middle- or middle-middle-class milieu. So high school was a revelation. Status distinctions that had been entirely obscure to me came into focus. Everything about you—the clothes you wore, the music you listened to, the way you pronounced things—turned out to be a clear marker of where you were from and whether you were worth knowing.

By the time I made it to a selective college, I found myself entirely surrounded by this upper-middle-class tribe. My fellow students and my professors were overwhelmingly drawn from comfortably affluent families hailing from an almost laughably small number of comfortably affluent neighborhoods, mostly in and around big coastal cities. Though virtually all of these polite, well-groomed people were politically liberal, I sensed that their gut political instincts were all about protecting what they had and scratching out the eyeballs of anyone who dared to suggest taking it away from them. I can’t say I liked these people as a group. Yet without really reflecting on it, I felt that it was inevitable that I would live among them, and that’s pretty much exactly what’s happened.

So allow me to unburden myself. I’ve had a lot of time to observe and think about the upper middle class, and though many of the upper-middle-class individuals I’ve come to know are good, decent people, I’ve come to the conclusion that upper-middle-class Americans threaten to destroy everything that is best in our country. And I want them to stop.

Who counts as upper middle class? It depends. Back in 2013, one survey found that 85 percent of Americans saw themselves as part of a broad middle class, stretching from lower middle (26 percent) to middle middle (46 percent) to upper middle (12 percent). We could define it by income—say, all single adults who earn more than $100,000 a year, or all married couples that earn more than $200,000—but that’s too crude. Let’s just say that upper-middle-class status is a state of mind. We’re talking about families that earn well into the six-figure range yet don’t feel rich, either because of their student loan debt or the enormous cost of the amenities they consider nonnegotiable: living in well-above-average school districts for those with children or living in “cool” neighborhoods for those without.

We often hear about the political muscle of the ultrarich. Billionaires like the libertarians Charles and David Koch and Tom Steyer, the California environmentalist who’s been waging a one-man jihad against the Keystone XL pipeline, have become bogeymen for the left and right respectively. The influence of these machers is considerable, no doubt. Yet the upper middle class collectively wields far more influence. These are households with enough money to make modest political contributions, enough time to email their elected officials and to sign petitions, and enough influence to sway their neighbors. Upper-middle-class Americans vote at substantially higher rates than those less well-off, and though their turnout levels aren’t quite as high as those even richer than they are, there are far more upper-middle-class people than there are rich people. One can easily turn the Kochs or the Steyers of the world into a big fat political target. It’s harder to do the same to the lawyers, doctors, and management consultants who populate the tonier precincts of our cities and suburbs.

Another thing that separates the upper middle class from the truly wealthy is that even though they’re comfortable, they’re less able to take the threat of tax increases or benefit cuts in stride. Take away the mortgage interest deduction from a Koch brother and he’ll barely notice. Take it away from a two-earner couple living in an expensive suburb and you’ll have a fight on your hands. So the upper middle class often uses its political muscle to foil the fondest wishes of egalitarian liberals. This week offered a particularly vivid reminder of how that works. In the windup to his State of the Union address, Barack Obama released a proposal to curb the tax benefits associated with 529 college savings plans, which primarily benefit upper-middle-class families, to help finance the expansion of a separate tax credit that would primarily benefit lower-middle- and middle-middle-class families. Only 3 percent of households actually make use of these accounts, and 70 percent of the tax benefits go to households earning more than $200,000, so you can see why Obama might have thought no one would get too worked up about the proposal. If anything, he might have thought, and hoped, that his critics would get more exercised about his call for big capital gains tax increases, which would have allowed him to play the part of Robin Hood—a role Obama loves to play.

That’s not quite how things turned out. From the get-go, the 529 plan, like the capital gains tax-hike plan, was totally politically unrealistic, as Republicans in Congress were never going to sign on. But within days of the State of the Union, the Obama administration was forced to reverse course and abandon its plan to make 529 plans less generous. House Minority Leader Nancy Pelosi, who represents San Francisco, and House Budget Committee ranking Democrat Chris Van Hollen, who represents the wealthy Maryland suburbs of Washington, D.C., were the key drivers of the decision, according to a report by Rachael Bade and Allie Grasgreen in Politico. My guess is that both Pelosi and Van Hollen saw firsthand the fury of upper-middle-income voters who sensed that Obama, normally a paragon of upper-middle-class virtues, was daring to mess with one of their precious tax breaks. Paul Waldman, writing for the Washington Post, had it right when he observed that “the 529 proposal was targeted at what may be the single most dangerous constituency to anger: the upper middle class.”

Many smart people—the libertarian Peter Suderman of Reason, the neoliberal Josh Barro of the New York Times, and conservative Patrick Brennan of National Review, among others—have made the point that if Obama and his allies can’t even tweak the tax treatment of this tiny little savings plan, they sure as hell can’t succeed in raising other taxes enough to finance entitlement spending as the baby boomers retire in ever-larger numbers in the decades to come, let alone expand social services and public investment with an eye toward making the United States just a bit more Scandinavian.

To conservatives, myself included, this is not in itself a bad thing. European social democracies finance their generous welfare states through high consumption taxes that virtually everyone pays. In America, a cross-party consensus that middle-income, including upper-middle-income, households should pay low taxes necessarily limits the growth of government. This is why Obama’s plan to bring back the Clinton-era tax rates for households earning more than $250,000 was abandoned for a higher figure of $450,000 shortly after his re-election. Democratic lawmakers like Pelosi and Sen. Chuck Schumer were terrified of angering voters who earned more than $250,000 and who insisted, despite all evidence to the contrary, that they were merely middle class, and who would have skinned them alive had they not been shielded from higher taxes.

You might be wondering why I’m so down on the upper middle class when they’re getting in the way of the tax hikes that will make big government even bigger. Doesn’t that mean that while liberals should be bothered by the power of the upper middle class, conservatives should cheer them on? Well, part of my objection is that upper-middle-income voters only oppose tax hikes on themselves. They are generally fine with raising taxes on people richer than themselves, including taxes on the investments that rich people make in new products, services, and businesses. I find that both annoyingly self-serving and destructive. The bigger reason, however, is that upper-middle-class people don’t just use their political muscle to keep their taxes low. They also use it to make life more expensive for everyone else.

Take a seemingly small example—occupational licensing. In North Carolina, teeth-whiteners without expensive dental degrees would like to be allowed to sell their services but are opposed by the state’s dentists, as Eduardo Porter noted in a recent New York Times column. Are the good dentists of North Carolina fighting the teeth-whiteners because they fear for the dental health of North Carolinians? It doesn’t look like it. A more plausible story is that dentists don’t want to compete with cut-rate practitioners, because restricting entry into the field allows them to charge higher prices. We often hear about how awesome it is that Uber is making taxi service cheaper and more accessible for ordinary consumers but how sad it is that they are making life harder for working-class drivers who drive traditional cabs. Notice that upper-middle-class credentialed professionals like dentists, lawyers, and doctors rarely get Uber’d to the same degree. Even when innovative services try to do things like, say, offer a free alternative to expensive insurance brokers, state and local governments will often step in to say, “Oh, no you don’t.” Want to offer a low-cost, high-quality education by, say, replacing expensive professors with Filipino instructors who teach calculus over streaming video? Sorry, pal, you first have to get approved by an accreditation body controlled … by the existing schools you’re trying to out-compete, which employ upper-middle-class people who don’t take any crap.

Or take immigration policy: Dean Baker of the liberal Center for Economic and Policy Research has called for increasing the number of doctors, dentists, and other professionals allowed into the U.S. while limiting the number of less-skilled people, like would-be retail clerks, custodians, and housekeepers. The reason is that high-skilled immigrants squeeze the wages of upper-middle-class professionals, who can afford to take a hit while lowering the cost of various services for poorer people by giving them the option of going to cheaper doctors and dentists. By contrast, bringing in retail clerks, custodians, and housekeepers makes life cheaper for the upper-middle-class professionals while squeezing the wages of working people, particularly immigrants who already live in the U.S. Want to guess how popular the idea of increasing the wages of nannies is with the upper-middle-class people who employ them? I’d love to know, but I’m sorry to report that upper-middle-class pollsters have yet to ask the question.

You’d almost get the impression that while working- and lower-middle-class people are expected to compete, whether with the Ubers of the world or with Chinese manufacturing workers or with immigrants with modest skills, members of the upper middle class ought to be immune. The result is that all Americans have to pay more to get their teeth whitened, to get a formal education, or to do any of the other million things that we can only get through licensed providers.

Even more egregious is the way that upper-middle-class NIMBY-ism pushes for strict land-use restrictions that drive working- and lower-middle-class people out of the country’s most desirable and productive cities, as Timothy B. Lee of Vox reminds us. We see this most vividly in affluent suburbs, where the local public schools are just as exclusive as elite private schools, and where high home prices do the work of high private school tuitions. Sometimes exclusivity is justified on other grounds. The UCLA economist Matthew Kahn has found that in California, at least, it is liberal cities that have the most stringent zoning regulations. It seems that upper-middle-class California liberals use their supposed environmentalism to justify policies that wind up excluding the less well-off, despite the enormous environmental benefits that would flow from allowing more people to live in coastal California communities.

Closer to my home in New York, I’ve found that upper-middle-class people are the chief culprits behind the gentrification wave that is driving many poor families out of close-in neighborhoods in Brooklyn, my hometown. Stephen Smith has done an excellent job of explaining the dynamic. Most affluent people would be just fine with living in condos in Manhattan if they could afford to do so. But rich Manhattanites fight new development with every fiber of their being, which forces slightly less rich people to move to Brooklyn. Here is where things get interesting. Early on, as gentrification first takes root, these new upper-middle-class arrivals root for development, particularly when it means things like a new Whole Foods and other amenities that make their neighborhoods seem less “sketchy.” Once they have their fancy grocery stores and their Pilates studios and whatever else it is that floats their boat, however, they sharply shift toward absolutely hating new development, as new development means having to share their new amenities with more newcomers. These new restrictions on supply mean that homeowners who arrived at the right time, before the drawbridge was raised, see their homes get more and more valuable. Landlords can charge higher and higher rents.* The neighborhood gets less and less “sketchy,” which is to say less diverse and less inclusive. How convenient.

What can we do to break the stranglehold of the upper middle class? I have no idea. Having spent so much time around upper-middle-class Americans, and having entered their ranks in my own ambivalent way, I’ve come to understand their power. The upper middle class controls the media we consume. They run our big bureaucracies, our universities, and our hospitals. Their voices drown out those of other people at almost every turn. I fear that the only way we can check the tendency of upper-middle-class people to look out for their own interests at the expense of others is to make them feel at least a little guilty about it. It’s not much, but it’s a start.

*Correction, Jan. 31, 2015: The article originally misstated that homeowners who arrived before the drawbridge was lowered see their homes grow in value. The gentrifying homeowners raise the metaphorical drawbridge to keep their neighborhoods exclusive. (Return.)