Like many medications, the wakefulness drug modafinil, which is marketed under the trade name Provigil, comes with a small, tightly folded paper pamphlet. For the most part, its contents—lists of instructions and precautions, a diagram of the drug’s molecular structure—make for anodyne reading. The subsection called “Mechanism of Action,” however, contains a sentence that might induce sleeplessness by itself: “The mechanism(s) through which modafinil promotes wakefulness is unknown.”

Provigil isn’t uniquely mysterious. Many drugs receive regulatory approval, and are widely prescribed, even though no one knows exactly how they work. This mystery is built into the process of drug discovery, which often proceeds by trial and error. Each year, any number of new substances are tested in cultured cells or animals; the best and safest of those are tried out in people. In some cases, the success of a drug promptly inspires new research that ends up explaining how it works—but not always. Aspirin was discovered in 1897, and yet no one convincingly explained how it worked until 1995. The same phenomenon exists elsewhere in medicine. Deep-brain stimulation involves the implantation of electrodes in the brains of people who suffer from specific movement disorders, such as Parkinson’s disease; it’s been in widespread use for more than twenty years, and some think it should be employed for other purposes, including general cognitive enhancement. No one can say how it works.

This approach to discovery—answers first, explanations later—accrues what I call intellectual debt. It’s possible to discover what works without knowing why it works, and then to put that insight to use immediately, assuming that the underlying mechanism will be figured out later. In some cases, we pay off this intellectual debt quickly. But, in others, we let it compound, relying, for decades, on knowledge that’s not fully known.

In the past, intellectual debt has been confined to a few areas amenable to trial-and-error discovery, such as medicine. But that may be changing, as new techniques in artificial intelligence—specifically, machine learning—increase our collective intellectual credit line. Machine-learning systems work by identifying patterns in oceans of data. Using those patterns, they hazard answers to fuzzy, open-ended questions. Provide a neural network with labelled pictures of cats and other, non-feline objects, and it will learn to distinguish cats from everything else; give it access to medical records, and it can attempt to predict a new hospital patient’s likelihood of dying. And yet, most machine-learning systems don’t uncover causal mechanisms. They are statistical-correlation engines. They can’t explain why they think some patients are more likely to die, because they don’t “think” in any colloquial sense of the word—they only answer. As we begin to integrate their insights into our lives, we will, collectively, begin to rack up more and more intellectual debt.

Theory-free advances in pharmaceuticals show us that, in some cases, intellectual debt can be indispensable. Millions of lives have been saved on the basis of interventions that we fundamentally do not understand, and we are the better for it. Few would refuse to take a life-saving drug—or, for that matter, aspirin—simply because no one knows how it works. But the accrual of intellectual debt has downsides. As drugs with unknown mechanisms of action proliferate, the number of tests required to uncover untoward interactions must scale exponentially. (If the principles by which the drugs worked were understood, bad interactions could be predicted in advance.) In practice, therefore, interactions are discovered after new drugs are on the market, contributing to a cycle in which drugs are introduced, then abandoned, with class-action lawsuits in between. In each individual case, accruing the intellectual debt associated with a new drug may be a reasonable idea. But intellectual debts don’t exist in isolation. Answers without theory, found and deployed in different areas, can complicate one another in unpredictable ways.

Intellectual debt accrued through machine learning features risks beyond the ones created through old-style trial and error. Because most machine-learning models cannot offer reasons for their ongoing judgments, there is no way to tell when they’ve misfired if one doesn’t already have an independent judgment about the answers they provide. Misfires can be rare in a well-trained system. But they can also be triggered intentionally by someone who knows just what kind of data to feed into that system.

Consider image recognition. Ten years ago, computers couldn’t easily identify objects in photos. Today, image search engines, like so many of the systems we interact with on a day-to-day basis, are based on extraordinarily capable machine-learning models. Google’s image search relies on a neural network called Inception. In 2017, M.I.T.’s LabSix—a research group of undergraduate and graduate students—succeeded in altering the pixels of a photograph of a cat so that, although it looked like a cat to human eyes, Inception became 99.99-per-cent sure it had been given a photograph of guacamole. (There was, it calculated, a slim chance that the photograph showed broccoli, or mortar.) Inception, of course, can’t explain what features led it to conclude that a cat is a cat; as a result, there’s no easy way to predict how it might fail when presented with specially crafted or corrupted data. Such a system is likely to have unknown gaps in its accuracy that amount to vulnerabilities for a smart and determined attacker.

As knowledge generated by machine-learning systems is put to use, these kinds of gaps may prove consequential. Health-care A.I.s have been successfully trained to classify skin lesions as benign or malignant. And yet—as a team of researchers from Harvard Medical School and M.I.T. showed, in a paper published this year—they can also be tricked into making inaccurate judgments using the same techniques that turn cats into guacamole. (Among other things, attackers might use these vulnerabilities to commit insurance fraud.) Seduced by the predictive power of such systems, we may stand down the human judges whom they promise to replace. But they will remain susceptible to hijacking—and we will have no easy process for validating the answers they continue to produce.

Could we create a balance sheet for intellectual debt—a system for tracking where and how theoryless knowledge is used? Our accounting could reflect the fact that not all intellectual debt is equally problematic. If an A.I. produces new pizza recipes, it may make sense to shut up and enjoy the pizza; by contrast, when we begin using A.I. to make health predictions and recommendations, we’ll want to be fully informed.

Building and maintaining a society-wide intellectual-debt balance sheet would probably require refining our approach to trade secrets and other intellectual property. In cities, building codes ask building owners to publicly disclose their renovation plans. Similarly, we might explore asking libraries or universities to accept, in escrow, otherwise hidden data sets and algorithms that have found a certain measure of public use. That would allow researchers to begin probing the models and underlying data on which we’re coming to depend, and—by building theories—make payments on our intellectual debt before it becomes due in the form of errors and vulnerabilities.

The growing pervasiveness of machine-learning models, and the fact that anyone can create one, promise to make this process of accounting difficult. But it’s vital. Taken in isolation, oracular answers can generate consistently helpful results. But these systems won’t stay in isolation: as A.I.s gather and ingest the world’s data, they’ll produce data of their own—much of which will be taken up by still other systems. Just as drugs with unknown mechanisms of action sometimes interact, so, too, will debt-laden algorithms.