I have found an answer to my question through narrowing down the naughty bit and googling. I have managed to reduce the lag 10 TIMES!!!! I mean....It is insane on how much computing power next-line was using to move a cursor down ?!?!

The fix:

Put this code into your init.el: (setq auto-window-vscroll nil)

The proof:

Now next-line does not trigger line-move-partial therefore reducing the lag. I do not remember setting up auto-window-vscroll to t . It wasn't anywhere in any of my .el files, I am not sure how it got set to t to begin with. So if anyone has an performance issues with the cursor movement, then the above fix will reduce the lag from about 50%-80% cpu time to 5% cpu time !!!

To quickly check if you are affected just run C-h v auto-window-vscroll . If it is set to t you might be having major performance issues. Verify with the Emacs profiler if the cursor movement does indeed cause a lag.

Best of luck fellow Emacs lovers!!!

Source of fix