Think of Dixie, and your mind probably conjures something like "Duck Dynasty" — bearded men bouncing along dirt roads in pickup trucks, raucously waving rebel flags.

You probably wouldn’t think of black-tied bankers cavorting in the plush ballroom of Manhattan's St. Regis hotel. But were you to peek inside the recent gathering of a secret Wall Street society, you’d have witnessed investment banking tycoon Warren Stephens taking the stage in a Confederate flag hat, performing an ode to finance to the tune of “Dixie." "In Wall Street land we’ll take our stand,” you would have heard him croon, as 200 of his well-heeled brethren nibbled foie gras.

Is this linkage of Wall Street and the slaveholding South merely a coincidence?

Actually, there's a very old affinity. Throughout its history, Dixie has often worn a suit.

Cotton Kings and Sugar Daddies

We tend to think of America as separate regions: the South where slavery happened, and the North that opposed it. But this regional distinction is an illusion in many ways, and one the money men gladly encourage. The truth is that without Wall Street's vigorous support and financing, the system of slavery could not have thrived. Much of the North's economic prosperity rested on the blood-soaked foundation of Southern cotton production. The curators of a fascinating and extensive exhibition series at the New York Historical Society, "Slavery in New York," put it this way: "New York was…the capital of American slavery for more than two centuries." In her book Disowning Slavery, Joanne Melish notes that one of the North’s greatest cultural achievements in the Civil War was to effectively erase its ties to slavery in the public imagination.

New York City’s slavery connection goes way back to its colonial past. Wall Street was the location of New York’s first slave market, established in 1711. New York never developed a big cash crop like cotton, so it didn’t have plantations on the scale of states further south. But you didn’t have to be a planter to benefit from the business of slavery. Many of New York’s prominent 18th-century families were deeply involved in the slave trade. Philip Livingston, a signer of the Declaration of Independence whose wealth flowed to Yale University, was a major slave trader.

New York passed a gradual emancipation act in 1799, and slavery was officially abolished in the state in 1827— but only officially. As historian Eric Foner has noted, the port of New York continued as the financial center of the illegal transatlantic slave trade up until the 1860s.

New York controlled the South’s cotton trade, which is why most of the city’s merchants and bankers supported slavery during the 1830s, 40s and 50s. Over and over, they used their influence to get concessions for the South in order to maintain their access to cotton: white gold. The South’s cotton production was a key source of profit and employment for the shipping, banking, insurance and textile industries.

As political scientist Thomas Ferguson has explained in his book Golden Rule, the Democratic Party of the 1830s was one in which elite merchants, New York City bankers, and planters like U.S. treasury secretary Robert Walker could find common ground on many items, including their support of free trade and slavery. (Originally from Pennsylvania, Walker moved to Mississippi to become a cotton, slave and land speculator and a politician who staunchly defended slavery on both economic and moral grounds.)

Money flowed into the New York financial world not just from cotton, but from another form of white gold: sugar. Businessman William Havemeyer, elected mayor of NYC in 1845 and 1848 and again in 1872, was from a sugar-refining family that relied on raw materials derived from the plantations of the Deep South and the Caribbean.

Perhaps the biggest sugar daddy of all was New York financier Moses Taylor, a sugar merchant who became a member of the Chamber of Commerce and key player in firms that evolved into familiar names, like Citibank. (Think of that next time you see a jaunty blue public bicycle cruising by in New York bearing the Citibank logo.)

Financiers like Taylor got a piece of the Southern action in numerous ways: they earned commissions from brokering sales, supervised the investments of planters in banks and financed the purchases of land and slaves. They got very cozy with their planter friends down South, even shepherding their children when they came to New York to study or serve as apprentices.

Today, if you were to take a stroll through Charleston, South Carolina and then hop a plane to Newport, Rhode Island (a satellite of the New York financial community up through the Gilded Age), you would notice a striking similarity in the style of houses, and you might even observe some of the same names. These two affluent port cities shared extensive connections both through slave trading and also through their elite social world. Southern planters and northern financiars and merchants worked together to protect their mutual interests in commerce and slavery. Starting around the mid-19th century, wealthy Southern planters took to building houses in Newport to escape the summer heat. Upper-class Northerners and Southerners became bound together in marriages and business dealings in resorts like Newport and universities like Yale, and later Princeton (nicknamed the “Southern Ivy"), which became the favorites of Southern planters.

The Civil War and After

Shipping merchant Fernando Wood, New York’s mayor in the 1840s, was a staunch supporter of the city’s ties to the slave South. On Jan. 8, 1861, just before the outbreak of the Civil War, the New York Times published the transcript of a report in which Mayor Wood called on the city to declare independence so it could continue trading with the slave South. Many in the New York media and business community agreed with Wood, lamenting what a war was going to cost in lost cotton profits. The New York Democratic Party machine objected to the idea of secession, but only because members tended to prefer keep close ties to the South without violating the Constitution.

Toward the end of the war, Wood went to Congress where he fought the 13th Amendment (the anti-slavery amendment), on the grounds that it violated private property rights.

The war may have put a temporary strain on relations between northern financiers and their planter friends, but the romance was quickly reignited. As Thomas Ferguson explains, the Democratic Party rose anew after the Civil War “with a coalition of bankers, merchants, and some (not all) important railroad men.” President Andrew Johnson "treated the South rather like another group of similarly connected business leaders treated Germany 80 years later, and began reinstalling the old leadership of the defeated country into power.” New York bank attorney Samuel Tilden, who ran for president in 1876, reasserted the Democratic Party’s ties to finance and Dixie, helping to organize efforts that led to the end of the South’s occupation.

During the Gilded Age, Northern and Southern elites enjoyed cozy relations, continuing on through the Roaring Twenties, as immortalized in F. Scott Fitzgerald’s The Great Gatsby, in which Southern belle Daisy Buchanan frolics in Long Island with her wealthy husband Tom and New York’s financial elite.

The Banker Dreams of Dixie

As Jack Temple Kirby points out in his book Media-Made Dixie: The South in the American Imagination, something called "Dixie" emerged in U.S. popular culture the 1930s. Leading up to that period, the South was more often than not depicted as brutal and backward, but by 1939, the year of the release of Gone With the Wind (courtesy of a Jewish filmmaker from Pittsburgh) the nostalgic version of a glorious, fallen South became popular in everything from whiskey advertisements to the music of Tin Pan Alley. For the most part, it was not the creation of Southerners, but rather the purveyors of popular culture in the emergent media and advertising industries located in other parts of the country.

The South became idealized in travel literature as the languid escape from hectic urban life in the North. The Carolina Lowcountry, in particular, was sold as a place where aristrocratic men endowed with paternalistic wisdom presided over genteel settings where people knew their socioeconomic place.

Dixie was a big seller. It still is. Flip through a recent issue of Forbes Life, the style handbook of the New York financier, you’ll notice advertisements for all the accoutrements of power expected of the banker-elite: exquisite watches, golf equipment, high-end matchmaking services, and…Dixie. An ad for Brays Island Plantation, located in Sheldon, South Carolina, features golden-hued hunting scenes, horseback riders in Spanish moss-draped settings, a porticoed great house. “One perfect plantation,” beckons the copy. The website invites you to come on down and enjoy “the sporting lifestyle of a traditional Lowcountry plantation.”

Cherokee Plantation, located between Charleston and Savannah, is an even more rarefied vision of the Old South sold to elites from elsewhere. Forbes 400 member Dirk E. Ziff, a New York media magnate, is part owner of Cherokee, which is often touted as the most expensive private club in America. A friend’s father who worked in real estate once invited me to see it. When a mule-drawn cart driven by a black employee appeared to transport our group to breakfast, I marveled at the completeness (and insensitivity) of the fantasy. But this place was not really created for the scions of Southern planters, but rather those who wish to play-act.

It is the Northern financier for whom a nostalgic marketing of plantation life often holds the most allure. Price of membership at Cherokee? A million bucks — and that won’t cover the yearly fees of $85,000. (That’s not the kind of cash most descendents of southern planters have lying around these days.) Here’s what you get, according to Forbes magazine:

“…Members who hunt quail can use a $100,000 Purdey house gun and enjoy an outdoor banquet—complete with tablecloths and candelabra. In the evening, members and guests are invited to pluck a stogie from the walk-in humidor in the grand plantation house. When guests climb into bed, they find a flannel-covered hot water bottle tucked between the sheets, even in summer.”

General Sherman burned down the original plantation house at Cherokee, but it and the entire estate were rebuilt as a sort of Southern Never-Neverland in 1930 by the same firm responsible for designing Central Park. The membership roster of Cherokee Plantation is a closely guarded secret, but it has been speculated to read like something “cribbed from the latest Fortune 500 list.” Here’s betting that the roster of Kappa Beta Phi, the Wall Street secret society that hosted the infamous shindig at the St. Regis hotel, boasts a few members. Maybe more than a few.

When Wall Street bankers express their fondness for the Old South, they’re not just whistling Dixie.