Today marks the anniversary of the passing of Andrew Johnson, perhaps the most-criticized president in American history.

Johnson died from a stroke on July 31, 1875 while visiting his daughter in Tennessee, just months into what he hoped was a political comeback. He had the unfortunate task of succeeding Abraham Lincoln after the 16th president’s assassination. Although popular at first, Johnson’s term in office went downhill quickly and he barely avoided conviction in the Senate after he was impeached by the House.

Johnson was born in 1808 in North Carolina. He served in the Senate and the House and as governor, and then military governor, of Tennessee. It was when he was serving as military governor that Lincoln asked him to replace Hannibal Hamlin as vice president in the 1864 campaign. Previously a Democrat, Johnson switched parties for the election.

Already an enigma because of his well-known independent streak and his earlier support of slavery, Johnson turned heads when he was possibly drunk at his own inauguration in 1865. (Another theory was that Johnson was ill, since he hadn’t been known for drinking in public.)

Johnson suddenly found himself as president when Lincoln died just after starting his second term. Tasked with the job of Reconstruction, Johnson’s policies were controversial and caused his new-found party to impeach him in 1868. Johnson kept his job by one vote in a Senate trial.

In his official biography on the White House website, Johnson’s term is summed up politely. “Although an honest and honorable man, Andrew Johnson was one of the most unfortunate of presidents. Arrayed against him were the Radical Republicans in Congress, brilliantly led and ruthless in their tactics. Johnson was no match for them,” says a bio prepared by Michael Beschloss and Hugh Sidey.

Looking back at how historians have ranked Johnson (and other presidents) since 1948, he appears to be the one president who has suffered most at the hands of revisionist history—either because more facts are available about his term, or because his place in the history of race relations has been re-evaluated.

In 1948, historian Arthur M. Schlesinger, Jr. ranked Johnson as a middle-of-the-pack president: a respectable 19th out of 29 presidents. Since then, some presidents have risen in the eyes of historians, like James K. Polk and Andrew Jackson while Andrew Johnson’s ratings have plummeted like a rock. In a 2010 Siena College survey, Johnson was called the worst president in history. A 2011 survey from a British academic institute ranked Johnson as 36th out of 40 presidents. And a C-SPAN academic survey had Johnson ranked as the second-worst president, just ahead of James Buchanan.

Lincoln was a tough act to follow, and Johnson’s failed attempt in obstructing much of the Republican Party’s Reconstruction plan has been a tough pill for historians to swallow. After becoming president, Johnson fought with his own Cabinet and party members over the scope of readmitting secessionist states and the voting rights of blacks.

Johnson favored a very lenient version of Reconstruction and state control over voting rights, and he openly opposed the 14th Amendment. Although Johnson had supported an end to slavery in the 1860s, he was a white supremacist. “This is a country for white men, and by God, as long as I am President, it shall be a government for white men,” he wrote in 1866.

In the end, the Radical Republicans won control over Reconstruction and Johnson became a pariah. Johnson vetoed the Civil Rights Bill, but Congress overrode the veto in an unprecedented move. Somehow, Johnson survived the impeachment trial, possibly because there was no vice president to replace him, and moderates feared Benjamin Wade, the Senate president pro tempore who would have replaced Johnson.

Johnson is also known for another historical footnote: He was the only former president to be elected to the U.S. Senate. In 1875, Johnson ran for his old Senate seat from Tennessee (after failing in tries for the Senate in 1869 and the House in 1872). He won after a state convention that featured 56 ballots. Former President Johnson was sworn in by the same Senate that almost impeached him a few years earlier. After serving about four months in the Senate, he passed away.