When do we use regularization ?

In Machine learning and statistics, a common task is to fit a model to a set of training data. This model can be used later to make predictions or classify new data points.

When the model fits the training data but does not have a good predicting performance and generalization power, we have an overfitting problem.

Regularization is a technique used to avoid this overfitting problem. The idea behind regularization is that models that overfit the data are complex models that have for example too many parameters. In the example below we see how three different models fit the same dataset.

We used different degrees of polynomials : 1 (linear), 2 (quadratic) and 3 (cubic).

Notice how the cubic polynomial "sticks" to the data but does not describe the underlying relationship of the data points.