Bettmann / CORBIS Grand Army of the Republic veterans at the annual Memorial Day Parade in New York City, May 1922

Memorial Day marks the unofficial start of summer, conjuring images of picnics, barbecues or just a lazy day off. But originally the holiday was charged with deeper meaning  and with controversy.

The exact origins of Memorial Day are disputed, with at least five towns claiming to have given birth to the holiday sometime near the end of the Civil War. Yale University historian David Blight places the first Memorial Day in April 1865, when a group of former slaves gathered at a Charleston, S.C., horse track turned Confederate prison where more than 250 Union soldiers had died. Digging up the soldiers' mass grave, they interred the bodies in individual graves, built a 100-yd. fence around them and erected an archway over the entrance bearing the words "Martyrs of the Race Course." On May 1, 1865, some 10,000 black Charleston residents, white missionaries, teachers, schoolchildren and Union troops marched around the Planters' Race Course, singing and carrying armfuls of roses. Gathering in the graveyard, the crowd watched five black preachers recite scripture and a children's choir sing spirituals and "The Star-Spangled Banner." While the story is largely forgotten today, some historians consider the gathering the first Memorial Day. (See the photo essay "Our World at War.")

Despite scattered celebrations in small towns, it took three more years for the holiday to become widely observed. In a proclamation, General John A. Logan of the Grand Army of the Republic  an organization of former soldiers and sailors  dubbed May 30, 1868, Decoration Day, which was "designated for the purpose of strewing with flowers or otherwise decorating the graves of comrades who died in defense of their country during the late rebellion." On Decoration Day that year, General James Garfield gave a speech at Arlington National Cemetery. Afterward, 5,000 observers adorned the graves of the more than 20,000 Union and Confederate soldiers entombed at the cemetery. (Read a TIME cover story on how not to lose in Afghanistan.)

At the outset, Memorial Day was so closely linked with the Union cause that many Southern states refused to celebrate it. They acquiesced only after World War I, when the holiday was expanded beyond honoring fallen Civil War soldiers to recognizing Americans who died fighting in all wars. It was also renamed Memorial Day. Some critics say that by making the holiday more inclusive, however, the original focus  on, as Frederick Douglass put it, the moral clash between "slavery and freedom, barbarism and civilization"  has been lost. Most Southern states still recognize Confederate Memorial Day as an official holiday, and many celebrate it on the June birthday of Jefferson Davis, the President of the Confederacy. But Texas, for one, observes the holiday on Robert E. Lee's birthday, Jan. 19  which also happens to be Martin Luther King Jr. Day.

The long-cherished Memorial Day tradition of wearing red poppies got its start in 1915. While reading Ladies' Home Journal, an overseas war secretary named Moina Michael came across the famous World War I poem "In Flanders Fields" by John McCrae, which begins, "In Flanders fields the poppies blow/ Between the crosses, row on row." Moved, she vowed always to wear a silk poppy in honor of the American soldiers who gave up their lives for their country. She started selling them to friends and co-workers and campaigned for the red flowers to become an official memorial emblem. The American Legion embraced the symbol in 1921, and the tradition has spread to more than 50 other countries, including England, France and Australia. (Watch TIME's video "An 'Honor Flight' for WWII Vets.")

With the National Holiday Act of 1971, Congress moved Memorial Day from May 30 to the last Monday in May. But critics say guaranteeing that the holiday is part of a three-day weekend promotes relaxation instead of stressing the holiday's true meaning. In 1989, Senator Daniel Inouye of Hawaii introduced a bill to move the holiday back to the fixed date of May 30. He has reintroduced it in every Congress since then  with no success.

While traditional Memorial Day rites have dwindled in many towns, they remain strong at Arlington National Cemetery. Since the 1950s, on the Thursday before Memorial Day, soldiers of the 3rd Infantry Division have placed American flags at each of the more than 260,000 graves there. During the weekend, they patrol around the clock to make sure each flag remains aloft. On the holiday itself, every year about 5,000 people turn out to see the President or Vice President give a speech and lay a wreath at the Tomb of the Unknown Soldier. And other Americans are encouraged to observe in a more solitary fashion. At 3 p.m. local time, according to the 2000 National Moment of Remembrance Act, which was passed to emphasize the meaning of Memorial Day, all Americans should "voluntarily and informally observe in their own way a moment of remembrance and respect, pausing from whatever they are doing for a moment of silence or listening to 'Taps.'"

Read "How to Leave Iraq."