When First Lady Michelle Obama took the stage during the first night of the Democratic National Convention, she talked about how it felt to be a black woman waking up in the White House every morning—a building constructed with slave labor. It was a powerful moment in her speech, hearkening back to the generations of African-Americans forced into bondage in this country. Up until a few decades ago, little attention was paid to looking into who actually laid the foundations and put up the walls of the White House. But what documentation exists today shows that many of Washington, D.C.’s most iconic government buildings, including the White House, were built by slaves.

In 2005, Congress put together a task force to shed light on the subject. After months of research, the commission announced that while it would never be able to tell the full story of the slaves who built these buildings, there was no doubt that they were intricately involved in the work, Alexander Lane reported for PolitiFact.

“Indifference by earlier historians, poor record keeping, and the silence of the voiceless classes have impeded our ability in the twenty-first century to understand fully the contributions and privations of those who toiled over the seven decades from the first cornerstone laying to the day of emancipation in the District of Columbia,” Senate Historian Richard Baker and Chief of the House of Representatives Office of History and Preservation Kenneth Kato wrote in a foreword to the report.

From a geographical standpoint alone, it should come as no surprise that slave laborers were used to build the nation’s capital. Washington, D.C., was built on landed ceded to the federal government by Virginia and Maryland, and at the time the Potomac region was home to almost half of the country’s 750,000 slaves, Lane reports.

While the White House Historical Association reports that the D.C. commissioners originally tried to bring cheap workers over from Europe to build the new capital, their recruitment efforts fell short. As a result, they forced local slaves to provide the labor, often renting workers from their masters for year-long periods of time.

“Slaves were likely involved in all aspects of construction, including carpentry, masonry, carting, rafting, plastering, glazing and painting, the task force reported,” Lane writes. “And slaves appear to have shouldered alone the grueling work of sawing logs and stones.”

In addition to constructing the buildings, slaves also worked the quarries where the stones for the government buildings came from. Ironically, the Statue of Freedom that sits atop the Capitol dome was made with the help of Philip Reid, a man enslaved by sculptor Thomas Crawford, who was commissioned to build the statue. According to the Architect of the Capitol, Reid was paid $1.25 a day by the federal government for his contributions.

“There is no telling how many stories that have been lost because, as a country, we didn’t value these stories,” historian and reporter Jesse J. Holland tells Smithsonian.com. “We’re always learning more about the presidents as we go forward and we’ll also learn more about the people who cooked their meals and dressed them.”