$\begingroup$

While I agree on ncasas answer in most points (+1), I beg to differ on some:

Decision Trees can be used as black box models, too. In fact, I'd say in most cases they are used as black-box models. If you have 10,000 features and a tree of depth of 50 you cannot reasonably expect a human to understand it.

Neural Networks can be understood. There are many analyzation techniques (see chapter 2.5 of my master thesis for some which are aimed at improving the model). Especially occlusion analysis (Figure 2.10), Filter visualization (Figure 2.11). Also the Why Should I Trust You? paper (my notes).

Explaining the prediction of a black-box model by fancy occlusion analysis (from "Why should I trust you?"):

I would like to point out The Mythos of Model Interpretability. It formulates some ideas about interpretability in a concise way.

Your question

Why are Machine Learning models called black boxes?

How people use it: Because they do not model the problem in a way which allows humans to directly say what happens for any given input.

Personal thoughts

I don't think this notion of a "black box model" makes much sense. For example, think of weather forecasting. You cannot expect any human to say which weather will be predicted if he is only given the data. Yet most people would not say that physical weather models are black box models. So where is the difference? Is it only the fact that one model was generated using data and the other one was generated using insights into physics?

When people speak of black box models they usually say it as if it is a bad thing. But humans are black box models, too. The critical difference I see here is that the class of errors humans make is easier to predict for humans. Hence it is a training problem (adverserial examples on the NN side) and an education problem (teaching humans how NNs work).

How the term 'black-box model' should be used: An approach which makes more sense to me is to call the problem a "black box problem", similar to what user144410 (+1) writes. Hence any model which only treats the problem as a black box - hence something you can put input in and get output out - is a black box model. Models which have insights (not only assume!) about the problem are not black-box models. The insight part is tricky. Every model makes restrictions on the possible function which it can model (yes, I know about the universal approximation problem. As long as you use a fixed-size NN it doesn't apply). I would say something is an insight into the problem if you know something about the relationship of input and output without poking the problem (without looking at data).

What follows from this: