When Barack Obama unveiled his official presidential portrait at the Smithsonian National Portrait Gallery on Monday, his response was gracious, if self-deprecatory. That combination has become something of a norm since the museum began commissioning portraits of presidents in the 1990s. Obama praised the likeness, but joked that artist Kehinde Wiley had denied his request to be painted with smaller ears and less gray hair; in 2008, George W. Bush praised college classmate Bob Anderson’s portrait as “fabulous” but quipped that he knew a sizable crowd would turn up “once the word got out about [his] hanging.” Even Abraham Lincoln poked fun at his own looks, despite his savvy use of portraiture as political message.

But not all presidents’ reactions to their official portraits have been so joyful. When he first laid eyes on the painting that was to be his official White House portrait, Lyndon B. Johnson disgustedly called painter Peter Hurd’s work “the ugliest thing I ever saw” and refused to accept it. Hurd was already decades into his successful career as a painter, specializing in portraiture and landscapes of the American Southwest. Arrogant enough to be unaffected by the comment and eager to publicize the president’s “very damn rude” behavior, he readily responded to press curiosity about the incident. Americans were sympathetic toward the scorned artist and increasingly skeptical of the president’s character—a slight that Johnson, who was already seen as short-tempered, could hardly afford. After displaying the piece at a Texas museum in retaliation, Hurd later donated his painting to the Portrait Gallery, which agreed to not display it until after Johnson's death.

“It’s a mystery to me,” says David C. Ward, former senior historian at the National Portrait Gallery and author of the new release America’s Presidents: National Portrait Gallery. “It’s a good 20th-century ceremonial portrait, and he hated it.”

America's Presidents: National Portrait Gallery A striking collection of presidential portraits from the National Portrait Gallery, this volume encapsulates the spirit of the most powerful office in the world. Buy

Unlike Obama’s portrait, which has received praise for its departure from the photorealistic tradition of presidential portraiture, Hurd’s portrait of Johnson wasn’t radical and on its face seemed quite similar to those of his predecessors (Elaine de Kooning’s portrait of John F. Kennedy being a notable exception.) A tall, broad-shouldered, determined-looking Texan in a dignified black suit, Johnson is imagined atop the roof of the Library of Congress, holding a heavy-looking U.S. history book, as the dwarfed U.S. Capitol building lights up Washington, D.C. in the twilit background. Like Wiley, Hurd didn’t shrink the president’s ears, blur the lines in his face or darken his gray, slicked-back hair; he portrayed Johnson flatteringly, powerfully, but he portrayed him as he was.

“If you just forget [Johnson’s] opinion—it’s a really good portrait of [him],” Ward says. “The fact that you’ve got Lyndon Johnson in this fictitious space, elevated above the entire landscape of the nation’s capital, I think that’s interesting… That’s what Johnson was. He was master of the Senate and then an extremely important president.”

Despite his power and prominence, though, Johnson was often overcome with insecurity. As a Texan, he saw himself as something of an outsider, according to Ward, and was often paranoid that more refined politicians aimed to take advantage of him. This unease was especially obvious in his relationship with the Kennedys: while they were wealthy, conventionally attractive and largely seen as classy and distinguished, Johnson grew up in poverty and was sometimes thought of as a “crude, kind of buffoonish outsized Texan,” according to Ward.

“He’s a major consequential figure, and we’ve tended to forget about him,” Ward says. “He’s still overwhelmed—and this would drive him crazy––by the glamour of [John F.] Kennedy.”

That tension might explain Lady Bird Johnson’s critique that the portrait of her husband didn’t properly depict his “gnarled, hardworking” hands. Though Johnson’s family was poor, he was no farmhand. He became a teacher right out of college and transitioned quickly to life in politics. Ward theorizes that perhaps Lady Bird felt the portrait didn’t adequately differentiate him from genteel New Englanders like Bobby Kennedy.

“Johnson always thought that people were looking down on him,” Ward says. “I wonder if there isn’t this uneasiness on the part of Johnson that somehow the city-slickers are taking advantage of him.”

But it’s possible—even probable, according to Ward—that Johnson’s disapproval of the portrait had less to do with him being fraught with feelings of self-doubt than it did with him being something of a bully himself. He is known to have driven an aide and a plumber to mental breakdown during his time as a politician (though the aide later said that Johnson was very conscious of his staff’s welfare.) He had a habit of applying the descriptor “piss-ant” to his adversaries, from “piss-ant” reporters to the “damn little piss-ant country” of Vietnam. And upon rejecting Hurd, Johnson arrogantly showed the artist his portrait created by the renown Norman Rockwell, which he claimed to prefer despite later getting rid of that painting as well.

“If he felt that you didn’t have any power, I don’t think he’s somebody you’d want to spend any time with,” Ward says. “He liked bullying people. It was like this compulsion to dominate people.”

But couldn’t his caustic personality simply be a byproduct of his insecurity? Ultimately, the discussion of Johnson’s shocking reaction to his presidential portrait couldn’t be more burdened than the legacy of the man himself. Once a celebrated liberal politician, Johnson championed progressive economic causes, access to education and racial equality with his dream for a “Great Society” at the height of the Civil Rights era. But his disastrous approach to the War in Vietnam—which led to the deaths of more than 58,000 Americans—practically precludes remembering him as a great president. The question of how to remember Lyndon B. Johnson in portrait and in policy doesn’t have a simple answer.

“He’s an increasingly tragic figure,” Ward says. “But on the other hand, the point of being a tragic figure is that you bring about your own demise.”