There are currently tens of millions of Americans without health insurance. Some can’t afford coverage at going rates. But as recently as 2009, one in seven applicants were rejected by the four largest insurance companies, who refused to sell them insurance at any price. Uninsurable Americans are mostly sick to begin with: They have heart disease, diabetes, cancer, and other pre-existing conditions that set off alarm bells for insurance sellers.

Ask why the already-sick can’t buy insurance and you get an immediate and seemingly obvious answer—their health costs are too high. But just because covering people with pre-existing conditions might be more expensive doesn’t explain why they can’t buy insurance at all. At any price. Shouldn’t they be able to buy insurance at a higher rate than healthy people, a rate that would protect the insurance company from the greater costs of their coverage?

An intriguing answer to that question comes from Nathaniel Hendren, a graduating Ph.D. student at MIT, in a study that got him offers from economics departments at Harvard, Stanford, and Princeton, among others. According to Hendren’s argument, not only are sick people a lot more expensive to care for, but they also know a lot more about what their cost of care is likely to be in the future. And it’s this inside information that makes the market for covering pre-existing conditions break down.

To understand Hendren’s theory, it’s useful to think about an extreme case. Consider the agonizing decision of whether or not to treat terminal cancer with costly and painful chemotherapy, which often provides only a small chance of remission. If you ask me what I, a healthy 41-year-old, would do, I have no idea—I’ve never really thought much about it, and in any event I have no real basis for weighing the costs and benefits. How painful would treatment actually be? And how would I face my own end-of-life decision?

Someone who already has cancer, by comparison, has a much greater appreciation for the treatment options available and presumably he has a much clearer sense of how far he’s willing to go for a chance at survival. Different people will have reached different decisions after going through this difficult calculus, and the outcome has significant financial implications for any insurance company that’s agreed to provide coverage.

So now let’s consider the problem facing an insurance company, say a Blue Cross, that wants to offer coverage to cancer patients with similar diagnoses. While they may look similar to the company’s statisticians, different patients may choose very different courses of treatment. Some may decide to pursue aggressive options. Others may opt out of what’s expected to be a long and painful fight. The patients’ medical expenses will be drastically different, despite their similar prognoses.

Now suppose the Blue Cross offers them all the same policy for, say, $10,000 per year, based on data showing that the annual medical costs of cancer victims is about $8,000 on average. Who is going to take the insurer up on the offer? A patient who expects his expenses to cap out at just a few thousand dollars won’t sign up—for him, the coverage isn’t worth it. But the patients who have already decided that they’ll take advantage of aggressive and expensive treatments will enroll. The cost per person of all patients with a cancer diagnosis may be $8,000, but if the only patients who enroll are the ones who expect their costs to be more than $10,000, that’s a money-losing proposition for the insurer.

Suppose the Blue Cross goes through with higher-priced coverage for cancer survivors anyway, and finds that the policyholders end up with medical expenses of $15,000 per year on average—could it solve the problem simply by raising the price to, say, $20,000? It can’t, because that would only make the problem worse by getting rid of the relatively cheap-to-insure customers who were willing to pay $10,000 for coverage but no longer find it worthwhile at a price of $20,000. Each time it raises the price, the insurer gets stuck covering an ever more expensive set of cases. It’s a no-win situation for insurers, so they choose not to offer coverage at all.

Hendren didn’t invent the idea of markets falling apart because customers know something that companies don’t. Nobel prizes were awarded to a trio of economists in 2001 for developing this idea of adverse selection in the 1970s. But his use of the concept may at least partly resolve the puzzle of why those with pre-existing conditions can’t get insurance.

There are two critical ingredients to Hendren’s argument. First, as he puts it, there’s only one way to be healthy, but many different ways to be sick. As a result, there’s wide variability in the costs that someone with cancer, heart disease, and other uninsurable conditions will impose on an insurer. With a bit of luck, a heart attack victim who takes his medicine, watches his diet, and exercises regularly can stay healthy and out of hospital for a long time. Less diligent survivors are more likely to be in and out of the hospital and end up in the operating room for multiple bypass surgeries, running up a tab of hundreds of thousands in expenses. Similarly, a cancer sufferer who opts out of aggressive treatment won’t cost much to an insurer, while the monthly cost of many chemotherapy drugs run into the tens of thousands.

Equally important to Hendren’s argument is the idea that sufferers of heart disease and cancer have greater self-knowledge than healthy people in terms of what their likely medical care costs will be. The market for insurance unravels, in Hendren’s model, when patients have a clear view of their future health care costs and people who anticipate lower-cost futures self-select out of insurance coverage.

Hendren tests his assumption using the Health and Retirement Study, which has surveyed Americans 55 and older since 1992. The HRS asks a battery of questions about participants’ futures, including the probability that they’ll end up in a nursing home, be disabled, or dead sometime in the next decade. Crucially for Hendren, the survey also asks about existing medical conditions, so it’s possible to sort HRS respondents based on whether they have conditions which, according to insurance underwriting guidelines, would be grounds for rejection. For example, those with strokes or previous bouts of home care can’t get long-term care insurance that would cover nursing homes; people with back conditions or obesity are ruled out of disability insurance; and stroke and cancer victims aren’t eligible for life insurance.

Hendren then examines whether those suffering from illnesses that freeze them out of insurance markets are better at predicting their medical futures. Across all three markets, he finds this to be the case. On the prospect of nursing homes, insurable respondents’ predictions are no better than random guesses, after accounting for age, gender, and other things that an insurance company can use to calculate customer risk. They’re not much better at predicting future disability or the arrival of the Grim Reaper. By contrast, across all categories, predictions made by uninsurable respondents do much better than random, and always out-predict their insurable counterparts by significant margins. This backs up Hendren’s theory that the reason insurers won’t cover patients with pre-existing conditions isn’t that they’re too sick—it’s that they’re too knowledgeable about their likely future medical costs.

If a voluntary market for insurance for pre-existing conditions is doomed to unravel, what’s to be done to accomplish the Obama administration’s goal of accessible health care for all? For the time being, there’s a government-run Pre-Existing Condition Insurance Plan that covers those denied coverage in the recent past. But that program’s history hints at the enormous—and unexpected—costs that come from insuring people who have already had cancer, heart attacks, and strokes. The cost per participant of PCIP is projected to be nearly $30,000 in 2012, more than double what government actuaries projected. This is exactly what Hendren’s model would have predicted: Only the highest-cost cases choose to purchase the insurance.

Come 2014, the Affordable Care Act will prevent insurers from discriminating based on pre-existing conditions: cancer victims and stroke survivors will be able to buy insurance at the same price as otherwise similar applicants. Insurance companies may take a hit to profits, but part of the cost will surely be passed on to the lower-cost counterparts to this high-cost pool. Healthy people might be tempted to opt out, but under the new law, they’ll be required to have insurance. This individual mandate is a natural fix to the problem of adverse selection in health insurance: It keeps the lowest-cost participants from opting out, and as a result the market doesn’t unravel.

Perhaps unsurprisingly, these solutions rankle the likes of Ron Paul and other libertarians who see the heavy hand of government at work here. More thoughtful alternative proposals from the free-market-is-best school of thought suggest creating a market for the right to buy insurance in the future: That way, you could enjoy your individual liberty by not buying health coverage today, but still keep open the possibility of exercising the option to buy insurance in the future. This “forward contract” to buy insurance wouldn’t be undermined by the problems Hendren highlights, since purchasers would still be making the decision before they experience the stroke or heart attack that gives them inside knowledge on what their future costs will be.

But that’s asking for an awful lot of foresight for the average 20-year-old, who doesn’t really have much reason to think about his likely health status a few decades in the future. And indeed, the least forward-looking are also probably those that can’t or don’t buy insurance.

You can say that’s their own fault—if insurance contracts are available and people don’t buy them, that’s a choice and they should deal with the consequences. Or we can accept that there are some situations—health care being one of them—where the market doesn’t know best, and the guiding hand of government needs to step in to ensure fair treatment for all.