Abraham Lincoln was our tallest president. At 6-foot-4, he would stand out even today, and he certainly towered over the men and women of his era. The top hat he habitually wore in public made him taller still. You couldn’t miss him in a crowd. The 16th president wore the top hat in war and peace, on the stump and in Washington, on occasions formal and informal. He wore it the night he was assassinated.

Likely a descendant of the 17th-century steeple, or sugarloaf, hat, which was in turn influenced by the headgear worn by soldiers, the top hat gained in popularity until, by the early 1800s, says Debbie Henderson in her book The Top Hat: An Illustrated History, “it had become the irrepressible symbol of prestige and authority.”

Nowadays, we have trouble envisioning Lincoln without his top hat, but how he began wearing it remains unclear. Early in his political career, historians tell us, Lincoln probably chose the hat as a gimmick. In those days he was rarely seen without his stovepipe, the traditional seven- or even eight-inch-high hat that gentlemen had been wearing since early in the century. True, Lincoln’s version was often battered a bit, as if hard worn, an affectation perhaps intended to suit his frontier image. The reformer Carl Schurz later recalled his first meeting with Lincoln, in a railroad car in the 1850s, on the way to one of the future president’s debates with Stephen Douglas. Schurz described Lincoln’s tailcoat as shabby and his top hat as crumpled, giving him what one historian has called a look “of unassuming simplicity.” So ubiquitous is the image of the battered stovepipe that the playwright John Drinkwater, in his popular Abraham Lincoln (1918), has Mrs.Lincoln saying, shortly after her husband is nominated for president, “I’ve tried for years to make him buy a new hat.”

When Lincoln gave his famous speech at the Cooper Institute in New York in February of 1860, some observers were quoted as saying that his hat looked bashed in. But this is unlikely. As the biographer Harold Holzer points out, Lincoln, the very day of his speech, bought a new top hat from Knox Great Hat and Cap at 212 Broadway. His suit fit poorly, his boots hurt his feet, but when he gave his speech in his stovepipe, says Holzer, “at least he would look taller than any man in the city.”

Lincoln’s top hats were not always of the same design. At his first inauguration in 1860, he wore the lower silk plush hat that had by that time come into fashion. By the start of his second term in 1864, he was again wearing a stovepipe, following (or perhaps ushering) a style that would continue for a good decade or more after his assassination.

Lincoln’s stovepipe made him an easy mark for caricaturists, and many drawings have survived in which the hat is the viewer’s means for identifying him. But the cartoonists are not the only ones who found it easy to spot the 16th president in his hat.

In August of 1864, Lincoln was on horseback, on his way to the Soldiers’ Home, about three miles northeast of the White House, where he had the use of a stone cottage in the summer months. A would-be assassin fired from near the road, shooting the stovepipe off Lincoln’s head. Soldiers who found it said there was a bullet hole through the crown. This incident gave rise to the popular notion that the hat saved Lincoln’s life.

The better surmise is that the hat made Lincoln easy to spot in a crowd. In July of 1864, at the Battle of Fort Stevens, he stood in the battlements wearing his trademark hat—making him, in Carl Sandburg’s phrase, “too tall a target” for the Confederates—until warned by a Union officer to get down.

On the night Lincoln died, he dressed for the theater in a silk top hat, size 7-1/8, from the Washington hatmaker J. Y. Davis, to which he had added a black silk mourning band in memory of his son Willie. When Lincoln was shot, the hat was on the floor beside his chair.

No other president is so firmly connected in our imaginations with an item of haberdashery. We remember Franklin D. Roosevelt’s cigarette holder and John F. Kennedy’s rocker, but Lincoln alone is remembered for what he wore. Harold Holzer says, “Hats were important to Lincoln: They protected him against inclement weather, served as storage bins for important papers he stuck inside their lining, and further accentuated his great height advantage over other men.”

Lincoln’s taste for hats also gave us a remarkably durable image of our most remarkable president. Lincoln remains a giant in our memories, and looms even taller in his stovepipe hat.

Years into his career as a Yale University law professor, Stephen L. Carter began writing best-selling novels, including 2012’s The Impeachment of Abraham Lincoln.