In an era when our wars seem too often fought far away, literally and figuratively, from the majority of Americans’ lives, Memorial Day serves a vital purpose: providing cultural space for those who have lost loved ones in war. It offers us images and commemorations of fallen soldiers—a communal chance to remember and honor the lost.

Yet Memorial Day’s original meanings and narratives are significantly different from, and would add a great deal of complexity and power to, how we see them nowadays. The holiday was first known as Decoration Day, and (per thorough histories by scholars like David Blight) was originated in 1865 by a group of freed slaves in Charleston, South Carolina. The slaves visited a cemetery for Union soldiers on May 1st of that year and decorated their graves, a quiet but very sincere tribute to what those soldiers have given and what it had meant to the lives of these freedmen and women.

The holiday quickly spread to many other communities of former slaves, and then to the North and nation as a whole. Yet just as quickly, national Decoration Day celebrations began to focus on less potentially divisive perspectives of former soldiers, not to mention ones far less focused on slavery. Indeed, by the 1870s the commemorations featured veterans from both sides, as illustrated by prominent former Confederate Roger Pryor’s 1877 Decoration Day address in New York.

Yet despite this telling national shift, former slaves continued to honor the holiday in their own way, as evidenced by a powerful scene from Constance Fenimore Woolson’s local color short story, “Rodman the Keeper” (1880). Woolson’s protagonist, himself a Union veteran living in the South, observes a group of ex-slaves leaving their decorations on the graves of the Union dead at the cemetery where he works as a caretaker.

On one hand, these ex-slave memorials to fallen soldiers offer a parallel to the family memories that now dominate Memorial Day, and serve as a beautiful reminder that the American family extends to blood relations of very different and equally genuine kinds. On the other hand, the ex-slave memorials represent far more complex and communally significant American stories and perspectives than any individual familial memory. These Decoration Day commemorative acts offered a vital acknowledgment of both some of our darkest histories and the ways in which we had overcome them at great but necessary cost.

I’m not trying to suggest that current celebrations of Memorial Day are anything other than genuine and powerful. I’ve heard the eloquent words of my grandfather, Art Railton, about what experiences with his fellow soldiers meant to him; he even commandeered an abandoned bunker and hand-wrote a history of his Company after the war. But as with so many of our national histories, what we’ve forgotten is just as genuine and powerful, and often a lot more telling about who we’ve been and thus who and where we are. The more we can remember those histories, too, the more complex and meaningful our holidays, our celebrations, our memories and our futures will be.

Ben Railton is an Associate Professor of English at Fitchburg State University and a member of the Scholars Strategy Network.