Americans are likely to think of New Year’s Eve and New Year’s Day as a time to celebrate the fresh start that a new year represents, but there is also a troubling side to the holiday’s history. In the years before the Civil War, the first day of the new year was often a heartbreaking one for enslaved people in the United States.

In the African-American community, New Year’s Day used to be widely known as “Hiring Day” — or “Heartbreak Day,” as the African-American abolitionist journalist William Cooper Nell described it — because enslaved people spent New Year’s Eve waiting, wondering if their owners were going to rent them out to someone else, thus potentially splitting up their families. The renting out of slave labor was a relatively common practice in the antebellum South, and a profitable practice for white slave owners and hirers.

“‘Hiring Day’ was part of the larger economic cycle in which most debts were collected and settled on New Year’s Day,” says Alexis McCrossen, an expert on the history of New Year’s Eve and New Year’s Day and a professor of history at Southern Methodist University, who writes about Hiring Day in her forthcoming book Time’s Touchstone: The New Year in American Life.

Some enslaved people were put up for auction that day, or held under contracts that started in January. (These transactions also took place all year long and contracts could last for different amounts of time.) These deals were conducted privately among families, friends and business contacts, and slaves were handed over in town squares, on courthouse steps and sometimes simply on the side of the road, according to Divided Mastery: Slave Hiring in the American South by Jonathan D. Martin.

Accounts of the cruelty of Hiring Day come from records left by those who secured their freedom, who described spending the day before January 1 hoping and praying that their hirers would be humane and that their families could stay together.

“Of all days in the year, the slaves dread New Year’s Day the worst of any,” a slave named Lewis Clarke said in an 1842 account.

“On New Year’s Day, we went to the auctioneer’s block, to be hired to the highest bidder for one year,” Israel Campbell wrote in a memoir published in 1861 in Philadelphia, in which he describes being hired out three times.

“That’s where that sayin’ comes from that what you do on New Year’s Day you’ll be doin’ all the rest of the year,” a former slave known as Sister Harrison said in an interview in 1937.

Get your history fix in one place: sign up for the weekly TIME History newsletter

Harriet Jacobs wrote a particularly detailed account in “The Slaves’ New Year’s Day” chapter of her 1861 autobiography Incidents in the Life of a Slave Girl. “Hiring-day at the south takes place on the 1st of January. On the 2[n]d, the slaves are expected to go to their new masters,” she wrote. She observed slave owners and farmers renting out their human chattel for extra income during the period between the cotton and corn harvests and the next planting season. From Christmas to New Year’s Eve, many families would “wait anxiously” to find out whether they would be rented out, and to whom. On New Year’s Day, “At the appointed hour the grounds are thronged with men, women, and children, waiting, like criminals to hear their doom pronounced,” Jacobs wrote.

On one of these fateful days Jacobs saw “a mother lead seven children to the auction-block. She knew that some of them would be taken from her; but they took all.” The slave trader who took the children wouldn’t tell her where he was taking them because it depended on where he could get the “highest price.” Jacobs said she would never forget the mother crying out, “Gone! All gone! Why don’t God kill me?”

Enslaved people who attempted to resist going to their new masters were whipped and thrown in jail until they relented and promised not to run away during the new arrangement. Older slaves were also particularly vulnerable, as Jacobs describes one owner trying to hire out a frail roughly 70-year-old woman because he was moving away.

But the history of New Year’s Day and American slavery is not all horror. The holiday was also associated with freedom.

The federal ban on the transatlantic slave trade went into effect on New Year’s Day in 1808, and African-American communities did celebrate, but the festivities were short-lived.

“Different slave-trade abolition commemorations took place between 1808 and 1831, but they died out because the domestic slave trade was so vigorous,” says McCrossen. The risk of violence was also too great. For example, on New Year’s Eve in 1827, in New York City, a white mob attacked African-American congregants and vandalized their church.

The holiday became more associated with freedom than slavery when Abraham Lincoln signed the Emancipation Proclamation, freeing slaves in Confederate states on New Year’s Day in 1863. Slaves went to church to pray and sing on Dec. 31, 1862, and that’s why there are still New Year’s Eve prayer services at African-American churches nationwide. At such “Watch Night” services, congregants continue to pray for more widespread racial equality 157 years later.

Correction, Dec. 27

The original version of this story misstated the year of an attack on an African-American church on New Year’s Eve in New York City. It was 1827, not 1927.

Get our History Newsletter. Put today's news in context and see highlights from the archives. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Write to Olivia B. Waxman at olivia.waxman@time.com.