Walk into an H&M, Zara, Forever 21, or any other fast-fashion outlet and you’ll see it: people throwing bargain-priced jeggings, peacoats, and clingy tank tops into their carts as if they were buying staples at the supermarket. Shoppers with stodgier tastes do the same with $12 capris at Walmart and $20 blazers at Kmart. In 1985, Americans on average bought 31 items of clothing a year. Today, we buy roughly 60—more than one per week. And when we lug home our haul we’re not shy about making room in the closet: We throw out 78 pounds (PDF) of textiles per person—five times as much as we did in 1970.

What gives? It isn’t like we have more cash to burn—US median incomes have stagnated over the same period that our clothing habit has exploded. What’s happened is clothes have gotten cheaper. Starting in the early 1990s, US clothing prices began a steady decline (before picking up a bit during the Great Recession), driven down by cheap imports from Asian clothing factories that were fed a steady supply of ever-cheaper cotton.

As food prices hover near all-time highs and millions go hungry, the biofuel boom has gotten much of the blame. But what about our hunger for cotton? Nearly half our clothes are made of the fluffy fiber, and around 2 percent (PDF) of cropland worldwide is devoted to it. That might not sound like much—but consider that cotton is the thirstiest crop in the world. And it commands fully 16 percent (PDF) of the insecticides consumed each year, more than any other single agricultural product.

China and India grow half of the globe’s cotton. Have you heard of the farmer suicide epidemic (PDF) in India? It’s largely concentrated among cotton farmers growing for the global market who are squeezed between high prices for seeds and low prices for cotton, according to a 2011 report by New York University’s law school.

It’s even worse in West Africa, where large amounts of prime farmland are devoted to cotton for export. Why do farmers put up with it? Because it’s often their only opportunity to get access to supplies, says Tom Bassett, a University of Illinois at Urbana-Champaign geography professor who studies cotton production in the region. Farmers buy pesticides and fertilizer from the cotton companies on credit before the season starts and use those resources not just on their cotton, but also on their food crops. The system keeps farm families fed but does little for the rest of sub-Saharan Africa’s 800 million residents—a quarter of whom are chronically malnourished.

Now, it’s impossible to draw a straight line between cotton production and any particular case of malnutrition. But we can question why farmers in places like Africa and South Asia are devoting land, resources, and time to growing fiber for skinny jeans while people in their own region don’t have enough to eat. The reason? The cotton market is big business—its gravitational pull is much stronger than that of families who need food.

The eminent agriculture development expert Hans Herren, president of the Millennium Institute, told me that farmers in West Africa could profitably grow cotton in a way that also bolsters food security, but only under dramatically different market conditions. “With proper crop rotations and other low-input sustainable-farming techniques, you could double crop yields and produce plenty of food alongside cotton,” he says. But farmers in those regions are set up to sell cotton to cotton traders; getting food from farm to market is more difficult.

Here in the land of disposable fashion, none of that matters much. In the age of Michael Pollan and slow food, more and more people think about where their dinner comes from and how it affects people and ecosystems on the way to their plates. Maybe it’s time we did the same for our wardrobes. If we did, we might have an easier time coming around to the idea of “slow fashion.”

So what are your options for a guilt-free closet? Vintage and secondhand, of course, are good options, and some major retailers (Patagonia, Eileen Fisher) encourage customers to send back used clothes—then repurpose them or offer them for sale at a steep discount. If thrift stores aren’t your thing, many manufacturers (including H&M) now offer some products made from organic cotton, which requires fewer chemicals and a little less water. But most of it is grown in the same regions as conventional cotton—meaning the farmers still get a raw deal. By far, the most effective strategy is to give up the supermarket sweep approach to clothes shopping and instead buy a few durable pieces. As for me, I’ll be thinking twice next time I’m tempted to grab a cheapo item off the rack at a chain store. Come to think of it, I just might splurge on a spendy wool sweater I’ve been coveting. Considering how long it will last, it might not be so extravagant after all.