$\begingroup$

I was trying to approximate a sine curve with a finite number of polynomials terms using linear regression (or the pseudo-inverse). I construct a Vandermonde matrix (or a Kernel polynomial feature matrix) and then solve the linear system (usually using the pseudo-inverse):

$$ y = \Phi(x)w$$

then I try to visualize the solution. For low degree polynomials the approximation seems fine but eventually when the degree of the polynomial is pretty high, there is a weird funky bit at the edge:

This reminded me of Gibbs phenomenon where at discontinuities there seems to be high oscillations near the jump. I know that Gibbs phenomena happens with a finite sum of Fourier series. However, this empirical observation really made me wonder. Is this the reason the edge of the approximation looks strange?

Also from a statistics/machine learning point of view it seems clear that if the solution is not regularized, then a high complexity model should overfit to noise. However, in this model there is no noise. Therefore, I was not quite sure what was going on and was wondering what it was.

Thus, my question is, is there a Gibbs phenomena for approximating sinusoidal with a finite number of polynomial terms? If yes, then what is it and what are its details?

Another observation that I found odd is that the high oscillation/jump only happened at the right discontinuity. I am not sure why that is but I thought it was quite puzzling.