Obama’s invocation of the slaves who built the White House was rhetorically powerful because the building is a symbol of the United States; to say slaves built it is to imply that they built the United States. Pointing out that it was not only slaves who built it (in essence, debunking an assertion that Obama did not make) undercuts the point, and in turn connects to the long tradition of arguing that however bad slavery may have been, it was only a small, regional phenomenon in the South. But as my colleague Ta-Nehisi Coates has written, slavery was the engine of the entire antebellum American economy: “In the seven cotton states, one-third of all white income was derived from slavery. By 1840, cotton produced by slave labor constituted 59 percent of the country’s exports.” Coates quotes historian David Blight, who wrote, “Slaves were the single largest, by far, financial asset of property in the entire American economy.”

More immediately controversial was O’Reilly’s statement that slaves working at the White House “were well-fed and had decent lodgings provided by the government.” The idea that slavery was, if not a good thing, not such a bad thing overall has a long, if hardly venerable, intellectual history.

This defense of slavery as essentially benign runs, for example, through Senator John C. Calhoun’s 1837 speech defending the “peculiar institution” from the criticism of northerners. In a speech in which he labeled slavery “a positive good,” the South Carolinian contended that enslaved blacks actually had it good:

Compare his condition with the tenants of the poor houses in the more civilized portions of Europe—look at the sick, and the old and infirm slave, on one hand, in the midst of his family and friends, under the kind superintending care of his master and mistress, and compare it with the forlorn and wretched condition of the pauper in the poorhouse.

Some 20 years later, another senator from South Carolina, James Henry Hammond, made the same point. Speaking to Northerners, he contended:

The difference between us is, that our slaves are hired for life and well compensated; there is no starvation, no begging, no want of employment among our people, and not too much employment either. Yours are hired by the day, not cared for, and scantily compensated, which may be proved in the most painful manner, at any hour in any street in any of your large towns. Why, you meet more beggars in one day, in any single street of the city of New York, than you would meet in a lifetime in the whole South.

Of course, that wasn’t quite all of it. Hammond added that the manifest inferiority of blacks further not only justified but practically sanctified the endeavor:

We do not think that whites should be slaves either by law or necessity. Our slaves are black, of another and inferior race. The status in which we have placed them is an elevation. They are elevated from the condition in which God first created them, by being made our slaves. None of that race on the whole face of the globe can be compared with the slaves of the South. They are happy, content, uninspiring, and utterly incapable, from intellectual weakness, ever to give us any trouble by their aspirations.

It is not hard to find contemporary accounts from slaves who were neither happy nor content—and whose eloquent testimony accounts to their inspiration, capability, and intellectual powers. If many whites like Hammond were unable to see this, however, others were not so blinkered. As O’Reilly noted, Michelle Obama’s predecessor as first lady, Abigail Adams was living in the White House at the time when slaves were building it, and she recorded her observations of those working on landscaping the grounds.