The number one reason we don’t shoot down North Korea’s missiles is that we cannot.

Officials like to reassure their publics about our defense to these missiles. Japanese Chief Cabinet Secretary Yoshihide Suga told his nation after last week’s test, “We didn’t intercept it because no damage to Japanese territory was expected.”

That is half true. The missile did not pose a serious threat. It flew over the Japanese island of Hokkaido, landing 3700 km (2300 miles) from its launch point near North Korea’s capital of Pyongyang.

The key word here is “over.” Like way over. Like 770 kilometers (475 miles) over Japan at the apogee of its flight path. Neither Japan nor the United States could have intercepted the missile. None of the theater ballistic missile defense weapons in existence can reach that high. It is hundreds of kilometers too high for the Aegis interceptors deployed on Navy ships off Japan. Even higher for the THAAD systems in South Korea and Guam. Way too high for the Patriot systems in Japan, which engage largely within the atmosphere.

All of these are basically designed to hit a missile in the post-mid-course or terminal phase, when it is on its way down, coming more or less straight at the defending system. Patriot is meant to protect relatively small areas such as ports or air bases; THAAD defends a larger area; the advanced Aegis system theoretically could defend thousands of square kilometers.

But could we intercept before the missile climbed that high? There is almost no chance of hitting a North Korean missile on its way up unless an Aegis ship was deployed very close to the launch point, perhaps in North Korean waters. Even then, it would have to chase the missile, a race it is unlikely to win. In the only one or two minutes of warning time any system would have, the probability of a successful engagement drops close to zero.

“When over Japan, they are too high to reach,” tweeted astronomer Jonathan McDowell, in between tracking the end of the Cassini mission. “You’d have to put the Aegis right off NK coast to have a chance.”

“It’s actually virtually impossible to shoot down a missile on the way up,” adds Gerry Doyle, deputy business editor for Asia at The New York Times. “Midcourse or terminal are the only places you have a shot.” That would mean for a test missile shot towards Guam, THAAD would have a chance to engage, though it has only been tested once against a missile of this range. For the test flights over Japan that would mean the only engagements possible are to the east of Japan, when the missile was on its way down. But there is little reason and huge logistical difficulties in having U.S. Aegis destroyers and cruisers loiter in the ocean there, waiting for a possible test launch.

Related: Why Didn’t the US Shoot Down That North Korean Missile?

Related: The Technology Race to Build — or Stop — North Korea’s Nuclear Missiles

Trying to use missiles from Aegis ships “would be a highly demanding task and entail a significant amount of guesswork, as the ships would have to be in the right place at the right time to stop a test at sea,” explains Kingston Reif of the Arms Control Association. And that is if the systems worked as advertised. None of the theater systems have been tested under the stressful conditions of a real-world exchange. THAAD, Patriot and especially Aegis, have done fairly well in tests, but these have been tests designed for success, simplified, carefully staged and using mostly short-range targets. Aegis has only been tested once against an intermediate-range target says Reif, one of the leading experts on U.S. missile defense programs.

What about our long-range defenses, the Ground-Based Midcourse Defense, or GMD, interceptors based in Alaska and California? There the test record is even worse. Even under ideal conditions, where the defenders knew the time, direction and trajectory of the test target and all the details of its shape, temperature, etc., this system has only hit its target half of the time.

“The success rate of the GMD systems in flight intercept tests has been dismal,” says former director of operational testing for the Pentagon, Philip Coyle. Our chances of intercepting a threat missile, even under ideal conditions, are basically “at least as good as a coin toss,” says the former head of the Missile Defense Agency, retired Lt. Gen. Trey Obering.

Yet, reporters routinely use words like “shield” and “dome” to describe our supposed capability, giving us a false sense of security. Officials make the matter worse with exaggerated, if carefully constructed, claims. “The United States military can defend against a limited North Korea attack on Seoul, Japan and the United States,” said Chairman of the Joint Chiefs, General Joseph Dunford at the annual Aspen Security Forum in July.

Is this true? It depends what you mean by the word “limited.”

If North Korea cooperated and shot their new intercontinental ballistic missile, the Hwasong-14, at the United States with adequate warning so that we could prepare, and if the warhead looked pretty much like we expect it to look, and if they only shot one, and if they did not try to spoof the defense with decoys that looked like the warhead, or block the defense with low-power jammers, or hide the warhead in a cloud of chaff, or blind the defense by attacking the vulnerable radars, then, maybe this is true. The United States might have a 50-50 chance of hitting such a missile. If we had time to fire four or five interceptors, then the odds could go up.

But North Korea is unlikely to cooperate. It will do everything possible to suppress the defenses. The 1999 National Intelligence Estimate of the Ballistic Threat to the United States noted that any country capable of testing a long-range ballistic missile would “rely initially on readily available technology – including separating RVs [reentry vehicles], spin-stabilized RVs, RV reorientation, radar absorbing material, booster fragmentation, low-power jammers, chaff, and simple (balloon) decoys – to develop penetration aids and countermeasures.”

Our anti-missile systems have never been realistically tested against any of these simple countermeasures. This is one reason that the Pentagon’s current director of operational testing is much more cautious in his assessments than missile defense program officials. “GMD has demonstrate a limited capability to defend the U.S. Homeland from small numbers of simple intermediate-range or intercontinental ballistic missile threats launched from North Korea or Iran,” he reports. Moreover, it is impossible, he says, to “quantitatively assess GMD performance due to lack of ground tests” and “the reliability and availability of the operational GBI’s [Ground-Based Interceptors] is low, and the MDA continues to discover new failure modes during testing.”

Yet, we have spent $40 billion on the GMD system and over $320 billion on scores of missile defense systems over the past few decades. You have to wonder exactly what these tests are for: give the troops the protection they need or give the contractors the next program payment?

There is no need to rely on the word of missile defense boosters, or, for that matter, trust the analysis of jaded missile defense critics. We could stop testing for success and begin testing for actual performance, with “red team – blue team” tests, for example, to simulate a determined foe. We could also order an objective scientific assessment. For example, the American Physical Society could conduct a thorough examination of the feasibility and capability of kinetic missile defense weapons, just as they did for directed-energy weapons in 1987. That study popped the balloon of false claims about these weapons, the original basis for the “Star Wars” program begun by the Reagan administration, concluding that it would be decades before we would know if such weapons were even feasible.

North Korea’s ballistic missile threat is real. We need to know if our missile defenses are for real.

UPDATE: The comment from retired Lt. Gen. Trey Obering has been expanded to more accurately reflect his belief that missile defense interceptors could defend against a missile attack on the United States.