Jaime has been used and marginalized by others his entire life. While he was never shunned as badly as Tyrion, he's still hurt by what others think of him, and how he's viewed by society. This is illustrated by his interactions with Brienne throughout the series, and also his exchange with Walder Frey in the season six finale—he lashes out with insults and angry words when Walder compares the two of them.

When he was 16, Jaime was named to the Kingsguard. It didn't take long for him to realize that King Aerys meant it as an insult to Jaime's father Tywin, to deprive him of an heir and use Jaime as a Lannister hostage. Jaime lost his house and inheritance, and after he killed Aerys, most of the realm dismissed him as an unfaithful/untrue knight. People started referring to him only as Kingslayer, instead of Jaime or Ser Jaime. He loses his name and honor.

When Jaime's right hand was cut off, he lost the only thing people still respected him for: his fighting prowess. Tywin tried to get him to return to Casterly Rock and take up the helm of House Lannister—not because he respected Jaime, but because he despised the idea of Jaime's brother Tyrion inheriting the family's wealth and power. Tommen later removed him from the Kingsguard, even though Jaime was trying to do the right thing for him and save Margaery from the Faith. He lost his occupation—and with it, his last shreds of self-respect.

His eldest son died, his father died, his brother left the continent, his daughter died, and now his youngest son has died. He never got a chance to be any sort of father figure to any of his children. He's lost his entire family except for Cersei (who's no longer the woman he knew and loved), and she's just shown him how little she really cares about him. Why let her live? Why go on living himself?