Our new reliance on inscrutable models as the source of the justification of our beliefs puts us in an odd position. If knowledge includes the justification of our beliefs, then knowledge cannot be a class of mental content, because the justification now consists of models that exist in machines, models that human mentality cannot comprehend.

One reaction to this could be to back off from relying upon computer models that are unintelligible to us so that knowledge continues to work the way that it has since Plato. This would mean foreswearing some types of knowledge. We foreswear some types of knowledge already: The courts forbid some evidence because allowing it would give police an incentive for gathering it illegally. Likewise, most research institutions require proposed projects to go through an institutional review board to forestall otherwise worthy programs that might harm the wellbeing of their test subjects.

We have already begun to define the realms in which machine justification has too high a social cost. For example, Andrew Jennings, the Senior VP of Scores and Analytics at FICO, the credit scoring company, told me: “There are a number of long standing rules and regulations around credit scoring in the US and elsewhere as a result of legislation that require people who build credit scores to manage the tradeoff between things that are predictively useful and legally permitted.” Machine learning algorithms might discover, for example, that Baptists generally are good credit risks but Episcopalians are not. Even if this example were true, that knowledge could not be used in computing a credit score because U.S. law prevents discrimination on the basis of religion or other protected classes. Credit score companies are also prohibited from using data that is a surrogate for these attributes, such as subscribing to Baptist Week.

There are additional constraints on the model credit score companies can use to calculate credit risk. If a lender declines a credit application, the lender has to provide the reasons why the applicant’s score was not higher. To meet this requirement, FICO makes the explanations as actionable as possible by the consumer. For example, Jennings explained, an applicant might be told, “Your score was low because you’ve been late paying off your credit cards eight times in the past year.”

But suppose FICO’s manually created models turned out to be less predictive of credit risk than a neural network would be. In fact, Jennings says that they recently compared prototype FICO Scores derived via machine learning techniques with the results from the manual model, and found that the difference between those scores, using the same superset of input variables, was insignificant. But the promise of machine learning is that there are times when the machine’s inscrutable models will be far more predictive than the manually constructed, human-intelligible ones. In those cases, our knowledge — if we choose to use it — will depend on justifications that we simply cannot understand.

But, for all the success of machine learning models, we are now learning to be skeptical as well. The paradigmatic failures seem to be ones in which the machine justification has not escaped its human origins enough.

For example, a system that was trained to evaluate the risks posed by individuals up for bail let hardened white criminals out while keeping in jail African Americans with less of a criminal record. The system was learning from the biases of the humans whose decisions were part of the data. The system the CIA uses to identify targets for drone strikes initially suggested a well-known Al Jazeera journalist because the system was trained on a tiny set of known terrorists. Human oversight is obviously still required, especially when we’re talking about drone strikes instead of categorizing cucumbers.

Mike Williams, a research engineer at Fast Forward Labs, a data analytics company, said in a phone interview that we need to be especially vigilant about the prejudices that often, and perhaps always, make their way into which data sets are considered important and how those data are gathered. For example, a recent paper discusses a project that used neural networks to predict the probability of death for patients with pneumonia, so that low-risk patients could be treated as outpatients. The results were generally more accurate than those that came from handcrafted models that applied known rules to the data. But the neural network clearly indicated that asthmatic pneumonia patients are at low risk of dying and thus should be treated as outpatients. This contradicts what caregivers know, as well as common sense. It turns out that the finding was caused by the fact that asthmatic patients with pneumonia are immediately put into intensive care units, resulting in excellent survival rates. But obviously that does not mean they should be sent home. On the contrary. It takes a human eye to spot this sort of error.

Cathy O’Neill, author of the recent book Weapons of Math Destruction, points to implicit biases in the values that determine which data sets we use to train a computer. When I spoke to her, she gave me an example of someone looking for the best person to fill a job, with one desired trait being,

“someone who stays for years and gets promotions.” Using machine learning algorithms for this, you might end up always hiring men, since women tend to stay at jobs for shorter intervals. The same is true, she says, about blithely using machine learning to identify bad teachers in public school systems. What constitutes a bad teacher? Her class’s average score on standardized tests? How many students go on to graduate? Attend college? Make money? Live happy and fulfilled lives? Humans might work this out, but machine learning algorithms well may reinstitute biases implicit in the data we’ve chosen to equip them with.

So, we are likely to go down both tracks simultaneously. On the one hand, we will continue our tradition of forbidding some types of justification in order to avoid undesirable social consequences. Simultaneously, we are likely to continue to rely ever more heavily on justifications that we simply cannot fathom.