On a Sunday morning in 2017 I took down his picture, and by afternoon it was in the alley with other rubbish awaiting transport to the local landfill for final burial. Hardly a hero’s end.

The painting had no monetary value; it was really just a print of an original overlaid with brushstrokes to appear authentic. But 40 years earlier it had been a gift from a young Army wife to her lieutenant husband when the $25 price (framed) required juggling other needs in our budget.

The dignified likeness of General Robert E. Lee in his Confederate Army uniform had been a prized possession of mine. I’d grown up not far from the Custis-Lee Mansion, and at West Point, Lee, the near-perfect cadet, Mexican War hero, academy superintendent, and, finally, the commander of the Confederacy’s Army of Northern Virginia, cast a long, ever-present shadow. Later, in Army quarters from Fort Benning, Georgia, to Fort Lewis, Washington, the painting reflected my fascination with leadership, and it spoke of duty and selfless service.

Although it was a portrait of a man, to many it evoked wider ideas and emotions. For like an object bathed in the light of the setting sun, Robert E. Lee’s shadow took on exaggerated size and grew steadily as America’s Civil War retreated ever further into the softer glow of history.

A mythology grew around Lee and the cause he served. For many, Lee’s qualities and accomplishments, already impressive, gained godlike proportions. This was the Lee I first came to know: a leader whose flaws and failures were sanded off, the very human figure recast as a two-dimensional hero whose shadow had eclipsed the man from whom it came.

But as time passed, the myth was reexamined. The darker side of Lee’s legacy, and the picture in my office, now communicated ideas about race and equality with which I sought no association. Down it came.

Read: Lee’s reputation can’t be redeemed

It was not a simple decision. For almost 150 years, Lee had been a subject of study, and of admiration, not only for his skill, but also as a symbol of stoic commitment to duty. And while I could appreciate the visceral association with slavery and injustice that images of the Confederacy’s most famous commander evoke, for a lifetime, that’s not the association I’d drawn. I’d read and largely believed Winston Churchill’s statements that “Lee was one of the noblest Americans who ever lived and one of the greatest captains known to the annals of war.”

At age 63, the same age at which Lee died, I concluded I was wrong—to some extent wrong about Lee as a leader, but certainly about the message that Lee as a symbol conveyed. And although I was slow to appreciate it, a significant part of American society, many still impacted by the legacy of slavery, had felt it all along.

Most accounts of Lee as a man, and a leader—his physical presence, demeanor, valor, and apparent serenity—reflect almost quintessentially desirable leadership traits. But staring into a bright light makes it hard to see clearly. More than most, Lee is portrayed either in a glare of adulation or, more recently, under a dark cloud of disdain.

At West Point, Lee and the other Southern heroes became icons whom other cadets and I instinctively sought to emulate. In a painful contradiction, they also betrayed the oath we shared, took up arms against their nation, and fought to kill former comrades—all in the defense of a cause committed to the morally indefensible maintenance of slavery.

In terms of Lee’s character, in some ways he was a good man, and in other ways a bad one. But leadership itself is neither good nor evil. Malevolent leaders emerge as often as those we judge to be good. Leadership is better judged as either effective or not. Was Lee effective? In large ways yes, and in many ways no. It is difficult to separate Lee the leader from the mythology that has grown around him. If we look more closely, we’ll find that reality pushes back on the myth.

Ta-Nehisi Coates: What this cruel war was over

Institutions have tremendous influence on the leaders that emerge within them. In the mid-19th century, Lee made himself into one of the most respected members of the United States Army. One way to understand the first 54 years of Lee’s life, which included a 32-year career in the United States Army, is to understand why, in 1861, he was offered high-ranking command on opposing sides in a civil war.

As a cadet at West Point, Lee set a rarely achieved record of zero demerits and enviable academic marks. More fundamentally, he seemed to internalize the academy’s values captured in its motto of “Duty, Honor, Country.” Fellow cadets, who included a number of future comrades and battlefield opponents, gave their charismatic yet serious comrade the moniker of “Marble Man,” as though anticipating the role he would play for the last decade of his life, and for the first 150 years following his death.

For 31 years following his graduation from West Point, Lee’s reputation as a soldier continued to rise. Entering a peacetime Army, he spent the first 17 years of his career working on projects fortifying America’s extensive coastline and improving navigation on the Mississippi River. Dignified and reflexively courteous, Lee exuded quiet professionalism, acting out a part he’d written for himself. The examples of those he admired, like George Washington, the values he had inherited from the society he came from, the history he read, and his incubation at West Point shaped the image of the leader he wanted to be, and the leader he molded himself into.

Like many soldiers of his era, Lee first saw action in the Mexican War. Major General Winfield Scott, who commanded that war and was the most important Army officer of his era, frequently mentioned Lee in his dispatches, judging him to be “the very best soldier I ever saw in the field.” Within the Army, Lee had been marked as a man to watch.

In the years following the Mexican War, Lee remained in uniform. He took a high-profile post as the superintendent of West Point in 1852 and, in 1855, he received a promotion to lieutenant colonel and a transfer to the cavalry. But his personal life increasingly intervened.

The death in 1857 of his wife’s father, George Washington Parke Custis, caused Lee to take extended leave from his unit in order to settle family affairs. That process involved more than executing Custis’s last will and testament. The slave-worked estates were poorly run and heavily in debt, and the professional soldier found himself in an active role within the landed, slave-owning gentry for which the South was known.

Lee’s own statements on slavery are conflicting, but his overall record is clear. Although he repeatedly expressed his theoretical opposition to slavery, he in fact reflected the conventional thinking of the society from which he came and actively supported the “peculiar institution” of slavery. Well before joining the Confederacy, Lee loathed abolitionists, and his feelings hardened as the Civil War dragged on.

Read: The myth of the kindly General Lee

From as far back as 1859, Lee’s personal treatment of slaves has been a public issue. Although accusations that he beat his slaves are impossible to prove after 150 years, their veracity is arguably beside the point. Lee was a willing and active participant in a society and economy that rested on slavery, and he fought ferociously to defend it. Lee was a Southerner, and efforts to depict him in opposition to slavery run contrary to his actions.

The road to America’s Civil War, and Lee’s fateful decision to join the Confederacy, was a long one. Pressure between the North and the South grew over decades, but the final straw for most in the Deep Southern states was the November 1860 election of the Republican candidate Abraham Lincoln of Illinois. For Lieutenant Colonel Robert E. Lee, then commanding the 2nd United States Cavalry Regiment at Fort Mason, Texas, the immediate challenge was to lead, and to carefully manage, a mass of officers and soldiers who were individually challenged to choose between loyalty to their state, their nation, and the new Confederacy that might arise. Outside events threatened to rupture long-held loyalties and produce an assault on the very premise upon which the United States had been created some eight decades earlier.

As pressures mounted, Lee read Edward Everett’s The Life of George Washington as if for guidance from the nation’s first president, spending these months meditating on where his own obligations lay. Was his duty, as he had sworn, to the United States, or, more basically, to its Army? Or did he owe allegiance to older ties to his beloved Virginia and, if it should secede, to the South? For Lee, the choice could not have been entirely the product of political analysis; that’s not how he functioned. Family, friendships, and visceral ties to the land and society from which he came all entered the calculus.

Everett’s tome made it no easier. Washington’s legacy seems to have reaffirmed to Lee the magnitude of his impending decision: “How his great spirit would be grieved if he could see the wreck of his mighty labors,” he wrote on January 23, 1861. These times of solitary reflection likely brought more questions than answers.

When cadets of South Carolina’s military college, The Citadel, fired upon a Union steamer which was attempting to resupply the Federal-held Fort Sumter in Charleston Harbor, the march to war quickened. When Texas voted to secede and joined six other states of the slaveholding Deep South to form the Confederate States of America, the United States Army garrison at Fort Mason was now in potentially hostile territory—a delicate position for any commander. But Lee was spared the task of navigating this uncertainty alongside his troops when he was ordered to “report in person” to Washington, D.C.

Events would soon force a personal decision. Although Lee saw secession as tragic, he had confessed to a friend that he felt his “loyalty to Virginia ought to take precedence over that which is due to the Federal government.” Reaching Arlington on March 1, 1861, the dutiful Lee looked to Virginia to guide his choice.

Read: Why I changed my mind about confederate monuments

Virginia, however, had not yet decided to join the Confederacy, and its choice, along with that of three other states of the Upper South—Arkansas, Tennessee, and North Carolina—appeared to depend upon the turn of fast-moving events surrounding besieged Fort Sumter and the willingness and ability of Lincoln’s United States government to assuage Southern fears over the future of slavery under a Republican administration.

Apparently firm in his conviction to attach his loyalty to Virginia but awaiting the South’s most populous and powerful state’s decision, Lee sent a somewhat mixed signal on March 28 by accepting a promotion to colonel in the United States Army.

In Virginia, which had rejected a proposal for secession on April 4, opinion shifted after Lincoln’s call on April 15 for 75,000 soldiers to be raised to put down the growing rebellion in the South. Virginia’s departure from the United States was put into motion, with the state’s legislature conditionally approving secession.

Against this backdrop, on the morning of April 18 Lincoln requested the highly regarded Lee to remain loyal to the Union and offered him command of the army of Federal volunteers being raised to put down the rebellion. Lincoln’s overture to secure Lee as a leader of Union military forces was a shrewd move. The new president had been told of Lee’s capability as a soldier, but he was also acutely aware of the cultural significance of having Lee, the Virginian, lead Union soldiers, a premonition that would hold true in converse as the dual weight of the Lee name and Virginia’s swing to the Confederacy became a tipping point for the secession of another three states. Lincoln’s cautious use of an intermediary in extending this invitation, to avoid embarrassment to his administration should Lee reject the Union’s overtures, was warranted. Lee refused “the offer he made me to take command of the army … though opposed to secession and deprecating war, I could take no part in an invasion of the Southern States.”

Winfield Scott told him that he must now formally resign.

On April 20, 1861, Colonel Robert E. Lee tendered his resignation from the United States Army. “You have made the greatest mistake of your life, and I feared it would be so,” General Scott, his mentor since the Mexican War, told him.

Virginia’s decision followed a month later when, in a referendum on May 23, 1861, 128,884 Virginians voted for secession, against the 32,134 who voted to remain in the Union. Lee the Virginian now became a Confederate.

Read: Local officials want to tear down confederate monuments—but states won’t let them

Although many historians view Lee’s loyalty to Virginia, and therefore his decision to fight for the Confederacy, as preordained, evidence and human nature suggest how excruciatingly difficult it actually was. Lee’s loyalties remained conflicted. He’d written extensively on his patriotism and faith in his nation: “There is no sacrifice I am not ready to make for the preservation of the Union, save that of honor,” but more fundamentally, Lee defined himself by duty.

From his earliest days, Lee’s conduct, his diligence, and his willing sacrifices were rooted in fulfilling responsibilities he set for himself, and in meeting the expectations of others. It was a persona he crafted carefully and projected intentionally. It was not a false depiction, but instead it was remarkably accurate in reflecting the very essence of the man. For Lee, the torture came when the institutions and values to which he felt obligations came into conflict. For the first time in his life, he could not simultaneously meet all the commitments he’d made. In simply tying his decision to the course chosen by his native Virginia, he essentially passed the most important moral decision of his life to the popular vote of others. Soon he would find himself supporting the greatest evil in American history, slavery, and not only opposing, but ultimately trying to destroy, some of the very institutions and ideas he’d held dear.

On April 22, 1861, when Lee accepted command of Virginia’s forces, he did it inside the state capitol at Richmond, which housed Jean-Antoine Houdon’s iconic statue of George Washington. As a boy in northern Virginia, Lee had walked the same streets as Washington; Lee’s wife was Washington’s step-great-granddaughter; and Lee had referenced the definitive biography of Washington when considering his loyalties at Fort Mason. In the Virginia statehouse in 1861, Lee was quite literally standing in his hero’s shadow. When he was named commander of Virginia’s forces, the president of the state convention even handed Lee one of Washington’s swords. In accepting, Lee would eventually commit himself to tearing asunder the nation that his role model had spent a life creating.

Lee’s decision to abandon both the Army and the nation to which he had sworn allegiance and dedicated his life after being offered command of soldiers on opposing sides of the Civil War was a Plutarchian moment in American history if there ever was one. That is to say, it was a moment of historical significance when a leader had to choose between competing values that could not be resolved in the abstract. The soldier for whom the concept of loyalty and the obligation of duty were sacred found himself in a complex collision of competing ethics and responsibilities. The decision to join Virginia, and ultimately the Confederacy, resulted in contradictions Lee spent the remainder of his life trying to rationalize, and admirers have attempted to ignore or justify.

Lee had some notable Civil War triumphs: Among the most memorable was Chancellorsville in 1863, in which he dispatched his intensely aggressive subordinate, Thomas “Stonewall” Jackson, on a daring march around the Union flank to win a famous victory. His “audacity” became the stuff of lore.

Still, he lost.

In April 1865, General Robert E. Lee put on his finest remaining dress uniform and rode his horse, Traveller, to meet the fellow West Pointer and Mexican War veteran General Ulysses S. Grant at Appomattox Court House, a small Virginia village, to discuss terms for the surrender of Lee’s army. The meeting, more than just the ending of the Civil War, was the beginning of the next chapter of the Lee legend.

What Lee did following the war was less important than the emergence of the mythology of the Lost Cause. The Southern war to defend the right to hold other human beings in slavery was recast as a struggle to defend Southerners’ freedom to maintain a way of life and to safeguard the work of the founding generation—as they defined it. As the objectives were redefined, the war itself was also given a new narrative—that of an outnumbered, poorly supplied band of heroes who courageously and stoically fought until overwhelmed by the industrial North. And for that matter, even Northern politicians venerated the man. In 1936, in a tribute while unveiling a Lee statue, President Franklin D. Roosevelt said:

All over the United States we recognize him as a great leader of men, as a great general. But, also, all over the United States I believe that we recognize him as something much more important than that. We recognize Robert E. Lee as one of our greatest American Christians and one of our greatest American gentlemen.

Lee personified a need for many across the country: He made the South’s cause seemingly noble, and gave a North seeking reconciliation a man-shaped olive branch. And so the statues came. Although he never sought the role, and while living played no part in its development, no leader better fit the Lost Cause narrative than Robert E. Lee. More than anyone, it was Lee the patrician hero, Lee the principled Southern patriot, and Lee the stoic warrior (rather than Lee the slaveholder, Lee the rebel, or Lee who had lost the Civil War) who fit the model in character and persona. Long after his death, he became the icon of the movement. As decades passed, Lee’s name and likeness spread, and took on whatever messages and meanings were desired by the observer.

Read: Take the confederate statues down

How do we judge Robert E. Lee—a leader I’d been raised to admire? The contradiction between the soldier whose qualities were held up for veneration and his effort to maintain slavery and divide the nation is clear. But apart from that, as a leader, what difference did he really make? How do we judge any leader? And what does our selection of leaders and heroes say about us?

For me, as for many others, assessing Lee is particularly difficult. From one angle, his stature is simply too big, his memory too venerated. Four years after his death, a Southern congressman, Benjamin Harvey Hill of Georgia, eulogized the soldier from Virginia:

He was a foe without hate; a friend without treachery; a soldier without cruelty; a victor without oppression, and a victim without murmuring. He was a public officer without vices; a private citizen without wrong; a neighbor without reproach; a Christian without hypocrisy, and a man without guile. He was a Caesar, without his ambition; Frederick, without his tyranny; Napoleon, without his selfishness, and Washington, without his reward.

But another angle, a bronze Lee on horseback as depicted in one of the many statues of the man, seemingly leading the South’s successful resistance to equality and change, blurs our ability to assess. We know the reality that neither image is an accurate reflection of the man or the leader, but mythology overpowers reason.

The picture of fellow soldier Robert E. Lee that hung in my home and inspired me for so long is gone, presumably crushed and buried with the other detritus of life. But the memory remains. The persona he crafted of a disciplined, dutiful soldier, devoid of intrigue and strictly loyal to a hierarchy of entities that began with God and his own sense of honor, combined with an extraordinary aptitude for war, pulls me toward the most traditional of leadership models. I try to stand a bit straighter. But when I contemplate his shortcomings, and admit his failures, as I must my own, there is a caution I would also do well to remember.