Of my three primary interests — quantum computing, machine learning, and robotics — quantum computing has been monopolizing my time recently. However, I am still keeping my eyes and ears scanning the other two fields. And, I recently came across this fascinating article about how our brains work; it would seem to have major ramifications for the structure of neural networks.

When you start learning about neural networks, you quickly encounter the terms “weights” and “biases.” And, while you will always encounter weights, you won’t always encounter biases. For example, my first neural network was a very simple neural network that I translated from Python to C simply to maximize my understanding of how it worked; it had weights, but no biases.

So, let’s discuss weights first. Akin to the English expression, assigning weight to something assigns it a signficance, a degree of importance. Initially, weights are completely random. As you train your neural network, it updates the weights as it figures out, essentially, what is more important.

Biases are added for classification tasks, when you want to know, for example, if widget X is more like the widgets in category A or the widgets in category B. Think of a bias as a minimum threshhold of confidence that has to be reached in order for widget X to be considered for either category. If there is a 2% chance widget X fits category A and a 1% chance it fits category B, neither classification is convincing. Biases prevent such weak assignments.

The aforementioned article has intriguing implications, because researchers have shown that our brains filter sensory information before processing it. Instead of accepting inputs and then automatically assigning weights/significance to those inputs, it first filters out inputs that it considers to be completely irrelevant to the task at hand.

While one might argue that data should be preprocessed before training a neural network, that’s not what our brains do. We accept sensory inputs first, and then we do the equivalent of preprocessing. Only after that do we assign weights and maybe add biases.

Therefore, a neural network needs to accept inputs, train itself to filter out completely irrelevant inputs, and then train the model on the remaining, significant inputs to do whatever it is the model is intended to do. I don’t know much about the basal ganglia — it seems not many do — so it will be interesting to look into how a self-preprocessing neural network might work.