The Trump administration has produced a report on a key potential addition to the Senate health care bill, meant to bring conservatives on board. The report is at best opaque — and at worst deliberately misleading.

The report, obtained Wednesday by the Washington Examiner and labeled “preliminary,” purports to show that an amendment proposed by Sen. Ted Cruz (R-TX) would lower the cost of health insurance.

But when you actually sit down and spend some time with the analysis, as economists have now done, you find it doesn’t show that at all. Charts compare very different types of data, apples to oranges. And key details about how the agency produced the report are nowhere to be found.

vox-mark Sign up for the newsletter VoxCare Get our newsletter in your inbox once a week. Email (required) By signing up, you agree to our Privacy Notice and European users agree to the data transfer policy. For more newsletters, check out our newsletters page Subscribe

“I don’t see a reason to produce these graphs ever if the goal is to give good information about the change in premiums,” Craig Garthwaite, a health economist at Northwestern University, said of certain charts included in the report.

This report is one of the first we’ve seen from the Trump administration that analyzes Congress’s work to repeal and replace the Affordable Care Act. It suggests a partisan motivation in how it displays data and little interest in transparency.

There are three problems, in particular, that analysts have found with it.

1) The report explores what happens if you apply the Cruz amendment to the Affordable Care Act — a scenario nobody is considering

The Cruz amendment would allow health insurers to sell coverage that doesn’t hew to Obamacare regulations — plans that could reject sick people, for example, or cover few benefits — so long as they offer one plan that does hit those rules.

A version of the amendment was added onto the Senate health care bill late last week. But the HHS report doesn’t appear to analyze that scenario — what would happen if the Senate bill with the Cruz amendment became law. It analyzes what would happen if the Cruz amendment became law on its own.

In other words, this is a report on what would happen if you tacked the Cruz amendment onto the Affordable Care Act — a legislative scenario that nobody is debating. This means that the analysis assumes that Obamacare’s Medicaid expansion stays in place and the tax credits for private coverage remain constant. In actuality, the Senate bill would end the Medicaid expansion and cut its private-coverage tax credits.

It’s hard to see a clear reason for performing such an analysis. One expert I spoke with guessed that it could reflect the fact that it is much easier to analyze the Cruz amendment on its own rather than in the context of the whole Senate bill. So HHS may have bit off this one chunk to start because it wanted to at least offer some kind of analysis of this new legislative language.

Still, it’s unclear why a quick analysis of a policy that no one proposed would do anyone much good. It doesn’t tell us much at all about what would happen if the Cruz amendment became law in the way Republican legislators expect: as part of a larger Obamacare repeal bill.

2) This graph

This is a graph from page 5 of the report, and at first glance, this looks like good news for the Senate effort: Premiums appear to go down both for plans that do comply with the Affordable Care Act benefits and for those that don’t.

But the chart is misleading. It compares the “enrollment-weighted” average for premiums in the current marketplaces, where the average age is 48, to the average premiums for a 40-year-old under the Republican plan.

That means the premiums displayed for the Cruz amendment plans are for younger people than those displayed for the Affordable Care Act. So of course you’d expect those to be lower. (Hat tip to Loren Adler and Matt Fielder at the Brookings Institute who noticed this first).

One other thing: Those “non-compliant” plans — the ones all the way to the right of the graph, with very low premiums — are quite different from the compliant ones. Specifically, the HHS report estimates that these plans would have a $12,000 annual deductible. Premiums go down in those plans because the coverage is incredibly skimpy.

It is unclear whether the Congressional Budget Office would even count those plans as health insurance. The non-partisan agency has previously said it will only count plans as coverage if they provide “financial protection against high medical costs.” It’s uncertain whether plans that ask individuals to spend $12,000 before their benefits kick in would meet that threshold.

3) HHS isn’t transparent about how it drafted this report

Other health policy experts took issue with the lack of detail on methodology. This sounds wonky, but it matters a lot when you’re trying to figure out how individuals will react to changes in the price of health insurance — whether they’ll decide to purchase plans when those prices go up or down.

In particular, economists often go to painstaking lengths to explain how they determine “elasticities” in the insurance market. This is a measure of how economists expect behavior to change when insurance becomes more or less expensive.

Economists have to make “assumptions about how responsive people would be if they saw their premium change by $1,000, how they would respond and if they would still buy a policy,” says Jonathan Graves, a health economist at Vanderbilt University.

Economists often determine elasticities by combing through economics literature and seeing what other studies say about how people respond to changes in the health insurance market. They then build microsimulation models that use those elasticities to project how enrollment changes if premiums go up or down.

Other health care forecasters are quite transparent about how they determine elasticities in demand for health insurance. The Congressional Budget Office, for example, has a 29-page document that outlines its methods.

But the HHS report does something very different. It says that its methodology is “proprietary” and thus doesn’t share its assumptions about how people react to changes in the health insurance markets.

Without any further information from the HHS report, it’s awfully difficult to understand how they expect the health insurance marketplace to change — and how much stock to place in their projections.

Methodologically, what matters for the estimates here are the elasticity estimates, which the report says are “proprietary” https://t.co/Dbdy3DmFqA — John Graves (@johngraves9) July 19, 2017

This makes it really hard for an economist like Graves to make heads or tails of the HHS report. There aren’t any equations to read or literature reviews that help explain the ways in which this report expects the health insurance marketplace to change.

“On every dimension, it’s not well-documented,” Graves says. “It’s just really opaque what they did.”