(CNN) — A few months ago, a small farmer in the Northeast approached me at a conference, intense and red-faced. How could I say that Americans shouldn’t pay more for their food?

She sold lettuce and beets to well-heeled women, their ears dangling gold and fingers sporting diamonds. Yet many of them balked at the prospect of paying an extra dollar per pound. To grow her food without extensive chemicals, and to sell her wares at market, she needed to fetch a higher price. Surely, couldn’t these women could pay more?

Well, yes, I conceded, those women could probably afford to pay more. That doesn’t mean we have to. Because it’s not the farmers who get most of the money we spend on food. It’s everyone who’s standing past the farm gate.

When we buy food, we think we are paying the farmer. This is true in a very basic economic sense: some portion of what we spend at the store does trickle back down to the hands that worked the land. Understandably, we think that if food costs more, it must be because the farmer is getting more for it.

There might be good reasons for prices to rise – the worst drought in a half-century, for instance — or there might be profiteering reasons for it — padding a supermarket’s bottom line. But the base assumption is that when we pay more, the farmer makes more.

The problem is, that is almost entirely untrue.

For every dollar we spend on food, only about 16 cents goes to the farmer. The other 84 cents go towards what economists call “marketing,” which refers not to commercials and advertising, but the entire chain that ensures food makes it from farm to plate.

Those 84 cents pay for the diesel and truck and driver to move the food from farm to processing plant or warehouse; the mill or the factory where food is processed, or the cost of storing it until it is sold. They also pay for the people who sell it wholesale or to grocers, the restaurant cooks who prepare it for us when we eat out, the satellite and databases to track shipments, and the workers, forklifts, warehouse and refrigeration at the grocery store.

Take, for instance, the humble onion. In 2008, shoppers paid about 67 cents for every pound of onions they bought, with about 13 cents going to the farm, meaning that the farm got 19 percent of what shoppers paid. (The workers who picked the onions got between 1 and 2 percent, or just about one penny per pound.)

Across all vegetables, the average share paid to farmers is more like 25 percent; fresh fruits pay an average of 30 percent to the farmer. All the rest of it went to distribution, logistics, overhead – everything that it takes to get food from farm to plate.

Today, nearly all our meals arrive in our neighborhoods via supermarkets (or supercenters, the term for operations like Walmart and Target that also sell groceries). Those two kinds of stores sell about 80 percent of our food. Walmart, the biggest supermarket in the country, sells roughly one-quarter of the food bought in the U.S., making it the largest grocer in our history (and that of the world).

But by selling food, supermarkets have also become a de facto infrastructure for distributing it — a fact not lost on Walmart executives.

“The misconception is that we’re in the retail business,” Jay Fitzsimmons, a senior vice president and treasurer for Walmart, told investors in 2003. But in reality, “We’re in the distribution business.”

Bringing distribution in-house is a big part of why Walmart now ranks as America’s largest grocer. When the mega-retailer expanded into food in the late 1980s, it set off a wave of consolidation within the supermarket industry as competitors scrambled to match prices. To stay in the game, most had to follow Walmart’s example and bring distribution in-house, a feat that only mega-sized companies could readily afford.

Local mom-and-pop grocers went out of business, and struggling chains merged or got bought by bigger ones; giant chains correspondingly gained market share. By 1998, 49 of the 50 largest supermarkets in the country handled their own distribution — and saved from 25 to 60 percent on operations as a result.

When little grocers went out of business, so did small and mid-size farmers. The new crop of bigger food retail outlets needed bigger quantities of food — more than a single, smaller grower could provide. Some farmers (including MacArthur “Genius Grant” recipient Will Allen) abandoned supermarkets altogether and found other ways to sell their goods — usually direct marketing like farmer’s markets and community supported agriculture clubs.

But many simply went out of business; the number of mid-sized farmers dropped by about 13 percent from 1997 and 2007 — and their share of sales dropped by 39 percent. Meanwhile, the number of big farmers inched up by 7 percent, but their market share skyrocketed, and they went from selling us half our food to nearly three-quarters of it.

The important thing here is to look at why they’ve been so successful: they built, and therefore shrank the cost of, distribution networks.

That’s actually part of what the red-faced farmer from the conference was talking about. She was selling at a farmer’s market in part because there is a dearth of infrastructure available to move food between small-to-midsized farms and plate. Infrastructure is one of the biggest and most expensive obstacles to expanding local agriculture.

And this is where it gets interesting: maybe the key to feeding America well, and from its own farms, is not to send everyone to the farmers market. Maybe it’s not joining CSAs. Maybe it’s coming up with a way to reduce the distribution costs for modest American farmers and grocers. Because if we can find some wiggle room in the 70 to 85 percent of the purchase price that goes to that, chances are there would be room to send a little more back to the farm.

And what that means — sorry for the wonkery here, but it has to be said — is coming up with an affordable infrastructure for modest American farmers and grocers so they have a fighting chance when competing against the giants.

As individual shoppers, making that happen can be tough. The easiest thing to do is prove that there’s a market for good, locally grown food in your community at an affordable price. This helps to prove to people in power — supermarket executives, government bureaucrats with budget lines — that it’s a worthwhile investment.

Patronizing farmers markets is one option; in-season produce tends to be affordable, in part because the middle-man has been removed from the equation. (One caveat: “affordable” is a loose term. During a lean and writerly and writerly year in Detroit I took a notepad with me to the farmer’s market for a week, jotted down prices, and compared them at a local market, so I knew when I was getting a good price and when I needed to make do with supermarket fare.)

Asking your local grocer where and how they get their produce — and suggesting that you’d like to see more from local growers — never hurts. (As a former supercenter produce worker, though, I cannot guarantee that it will help, either.)

The problem is that making good food easy and affordable is a challenge that won’t be solved by your shopping cart alone. There are some promising, larger efforts already underway: the emerging farm-to-cafeteria movement is a stealth infrastructure project, connecting small and midsize farmers with stable, institutional purchasers who can give them more economic stability than farmer’s markets. And some of the nation’s most lauded farmers markets are already part of a USDA project called Food Hubs, an interesting mix of public and private funding designed to link American farmers with retail and wholesale buyers and eaters.

Showing public support for those programs, helping them to succeed, and calling for better ones are the only way we’re likely to reach a better balance in that farm-marketing split — moving it, ideally, so that it lands a bit more on the farm, where it belongs.