“Optimism is the faith that leads to achievement; nothing can be done without hope,” Helen Keller wrote in her 1903 treatise on optimism. But a positive outlook, it turns out, isn’t merely an intellectual disposition we don — it’s a deep-seated component of our evolutionary wiring and the product of powerful, necessary delusions our mind is working around-the-clock to maintain. At the root of that mental machinery lies what psychologists have termed the self-enhancement bias — our systematic tendency to forgo rational evaluation of our own merits and abilities in favor of unrealistic attitudes that keep our ego properly inflated as to avoid sinking into the depths of despair.

The self-enhancement bias, which has significant overlap with the optimism bias neuroscientist Tali Sharot has studied, is one of the seventeen psychological phenomena David McRaney explores in You Are Now Less Dumb: How to Conquer Mob Mentality, How to Buy Happiness, and All the Other Ways to Outsmart Yourself (public library), which also illuminated why we have a hard time changing our minds and how Benjamin Franklin handled haters.

The mind’s delusory tendencies, McRaney explains, are just as vital as the automatic self-preservation processes of the body. Much like the respiration inhibition function of the brain prevents us from damaging our lungs by consciously deciding to stop breathing, the psyche employs a sort of “despair-inhibition module” of positive illusions constantly running in the background to power our self-enhancement bias — those rose-colored glasses we reserve exclusively for viewing ourselves, without which we might be blinded by life.

Citing several studies, McRaney writes:

Your wildly inaccurate self-evaluations get you through rough times and help motivate you when times are good. [Research shows] that people who are brutally honest with themselves are not as happy day to day as people with unrealistic assumptions about their abilities.

In other words, not only was Hunter S. Thompson right about journalism when he wrote that “there is no such thing as Objective Journalism” and that “the phrase itself is a pompous contradiction in terms,” but he was also right about the human condition at large — we are wildly unrealistic about ourselves, and that’s a good thing. Still, our self-perception — or explanatory style — exists on a spectrum, and different people fall at different spots along it. McRaney explains:

At one end is a black swamp of unrealistic negative opinions about life and your place in it. At the other end is an overexposed candy-cane forest of unrealistic positive opinions about how other people see you and your own competence. Right below the midpoint of this spectrum is a place where people see themselves in a harsh yellow light of objectivity. Positive illusions evaporate there, and the family of perceptions mutating off the self-serving bias cannot take root. About 20 percent of all people live in that spot, and psychologists call the state of mind generated by those people depressive realism*. If your explanatory style rests in that area of the spectrum, you tend to experience a moderate level of depression more often than not because you are cursed to see the world as a place worthy neither of great dread nor of bounding delight, but just a place. You have a strange superpower — the ability to see the world closer to what it really is. Your more accurate representations of social reality make you feel bad and weird mainly because most people have a reality-distortion module implanted in their heads; sadly, yours is either missing or malfunctioning.

A subset of three positive illusions powers our self-enhancement bias. One is the illusory superiority bias — our tendency to judge ourselves less harshly than we do others and to see ourselves as unique, special individuals amid a homogenous, dull crowd. Another is the illusion of control — our hindsight’s inclination to attribute our successes to ability and our failures to luck. McRaney writes:

The illusion of control persists like the other positive illusions because you need to feel as though you can push against the world and notice it move. Without that belief, your spirit dwindles quickly…

(It’s worth noting that the workings of this particular cognitive curiosity get significantly warped due to social and cultural biases that render some individual privileged and some discriminated against — over time, the former begin to see their fortunate circumstances as a reflection of their innate ability and merit, and the latter come to see themselves as the cause of their own misfortune, further internalizing the social trauma.)

Another vital positive illusion is the optimism bias:

Optimism bias [is] mental construct that provides smokers the belief they’ll be among those who escape cancer, motorists the confidence they can speed during rainstorms, couples the certainty they will die hand in hand behind a white picket fence, and immigrants the beamish tenacity to open a new business in a down economy. No matter the statistical odds, no matter how many examples to the contrary you’ve seen in your life, you have a tendency to believe everything will work out in the end, and it is hard to argue with this approach to life when you consider the alternative. The bias, however, disappears when you observe others. You believe your heart will stay strong until you are in your nineties, but that your cousin who buys chicken-fried steaks in bulk is headed for an early grave. The bias also prevents you from buying a fire extinguisher for your kitchen, or going to get a regular checkup. Your optimism bias keeps you looking to the horizon with growing expectation and glee.

Still other illusions underlie the trifecta of our self-illusory positivity — confirmation bias, which leads us to notice more of the information which confirms our beliefs and less of that which contradicts them, hindsight bias, which causes us to retroactively revise our own predictions in the face of new information and claim that we always saw it coming, and self-serving bias, which lets us take credit for all the good stuff that happens to us but blame the bad on external circumstances or other people. McRaney summarizes the formidable alchemy of these conspirers in forming the master-delusion of our self-enhancement bias:

The positive illusions and their helpers form a supercluster of delusion that thumps in the psyche of every human. Together, illusory superiority bias, the illusion of control, optimism bias, confirmation bias, hindsight bias, and self-serving bias combine like Voltron into a mental chimera called self-enhancement bias. It works just as the name suggests — it enhances your view of your self.

To be sure, this isn’t just some abstract theory psychologists concocted while twiddling their thumbs. McRaney cites a number of studies that illustrate in nearly comical detail the living manifestation of self-enhancement bias. One of the best was conducted at UCLA in 2010:

Researchers conducted a survey of more than 25,000 people ages 18–75 and found that the majority rated their own attractiveness as about a seven out of ten. This suggests that the average person thinks he is a little better looking than the average person. About a third of the people under 30 rated themselves as somewhere around a nine. […] You don’t have to be a mathematician to see a major problem here. Every person’s assumptions about being above average can’t be true. There can’t be an average unless some people sit in the middle of a bell curve and others fall to either side. Statistically speaking, if you had a perfect measure of your abilities you would see that you fall into the average category for most things, but you have a very hard time believing this is true.

So how, in the grand scheme of Earth history, did we end up so gloriously misguided about ourselves? McRaney points to evolutionary biologists, who have suggested that “the overconfident invaders of the jungles and savannah may have been so bold and intimidating that when they charged into the camps of their enemies, they tended to do better than the more timid and shy among them.” (The neuropsychology of the winner effect further advances this theory.) Similarly, some psychologists have posited that confidence is the deciding factor in survival:

There are psychologists who believe that morale is nothing more than a cluster of positive illusions; and morale is generally considered more important in combat than anything else. Confidence in battle and in courtship is certainly an important starting point for understanding where self-enhancement bias came from. These, though, may just be variations on a more fundamental truth. The general speculation is that over the last few million years, the primates who survived long enough to become your grandparents were the ones who didn’t give up when all hope was lost.

Fast-forward to modernity, and the same tendency toward doggedness and optimism in the face of resistance can be seen in the stories of our most celebrated modern heroes.

McRaney sums up the vital role of our self-enhancement bias:

You have the capacity to rationally judge the risks and benefits, the costs and rewards, of complex systems, but in a pinch you can fall back on a simple and reliable shortcut: just be slightly and blindly overconfident. The best bluff, it turns out, is the one in which even the bluffer is unaware of the cards he is holding. If you could accurately assess the odds against you—whether those odds took the shape of a hunting expedition, a one-on-one fight, or the job market for philosophy majors — you would probably turn away from the struggle more often than not. There is always plenty of evidence that the odds are not in your favor, enough to deter you from trying just about everything in life. Luckily for you, most of the time you have no idea what you are getting into, and you greatly overestimate your chances for success. It makes sense that primates like you would have evolved a fondness for delusions of grandeur. That’s the sort of attitude that gets you out of caves and beds. The relentless bombardment of challenges and tribulations makes it very difficult to be a person, whether you must fend off rabid beavers or ravenous bill collectors. Those who tried just a few percentage points harder, who persevered just a smidge longer, defeated nature more often than the realists. You’ve inherited a tendency to thrash against the odds, to be optimistic in the face of futility.

For a more mundane manifestation of our positive illusions, McRaney points to social media, where we carefully manicure and manipulate the image of Self we broadcast to others. (After all, Dani Shapiro put it perfectly: “Is there anything less revealing of Self than a selfie?”) But despite how off-putting such practices can be, it’s perhaps assuring to remember that they spring from an essential evolutionary feature. McRaney’s words are ultimately ones of assurance:

The desire to see yourself as better than average and more competent, skilled, intelligent, and beautiful than you truly are is likely embedded in your psyche as a by-product of millions of years of forging ahead against the same odds of survival that have erased 99 percent of all species that once roamed this planet.

He is careful to point out, however, that nurture aids nature in sculpting our levels of confidence as our early experiences have the power to either enhance or suppress those inherent tendencies. For a glaring example, look no further than the question of women in science, where we’ve made paltry progress since the days of pioneering astronomer Maria Mitchell and the gender gap in modern science education — in a culture that questions girls’ competence in science, is it any wonder that women are far less likely to be confident, let alone overconfident, in such careers?

And yet, large-scale social biases and limitations aside, we’re extraordinarily good at cultivating the very conditions that would contribute to our own confidence. McRaney points to the work of legendary psychologist B.F. Skinner, who believed that our core personality is shaped by small, everyday “experiments” we conduct in childhood, designed to foster our self-enhancement, by putting ourselves in situations where we are competent and thus grow increasingly confident:

Over time, [Skinner] believed, you learn that a wide variety of situations and behaviors will get you attention and praise or some other reward, and you begin to position yourself to always be in situations that allow for such an exchange with the outside world. You build a sense of self-confidence around those actions and situations you can be fairly certain will provide you a return or, as he put it, a reinforcer. This is why, he said, you decide to skip some gatherings and attend others. This is why you become fast friends with some people, and others turn you off within seconds. You tend to protect a bubble you’ve created and nurtured your entire life, a bubble of positive illusions that make you feel good about yourself. Those good feelings bleed into your sense of control and your general attitude when facing unfamiliar problems. Self-esteem and self-efficacy work together to get you out of bed in the morning and keep you going back for more punishment from the unforgiving world.

This model offers the psychological basis for Anna Deavere Smith’s poetic definition of self-esteem as “that which gives us a feeling of well-being, a feeling that everything’s going to be all right.” But perhaps the greatest gift of our self-enhancement bias, McRaney argues, is bestowed upon us precisely when we feel that things are not going to be all right. In a sentiment reminiscent of Viktor Frankl, he concludes beautifully:

Throughout human history there have been periods in which people bore tremendous burdens and slogged through what seemed like insurmountable misery. From concentration camps to death marches, to plagues and wars, people who share the same basic mind as you have suffered and survived horrific events. Likewise, you share something amazing with those who live daily under the yoke of terrible oppression. Should you be plucked from your cozy place in this world and assume their plight, should your will be tested at the intensity of so many before you, one constant is sure: You will be resilient. You won’t give up.

You Are Now Less Dumb is enormously illuminated in its entirety and comes on the heels of You Are Not So Smart, one of the best psychology books of 2011.

* For more on depressive realism, see Jonathan Rottenberg on the evolutionary origins of depression