He always knows how to hit me where it hurts the most.

“It was all for nothing.”

According to one of the more annoying voices in my head, if I don’t remember something, it was wasted.

Perhaps you know the feeling: you’ve lost your memory of an awesome event or an instructive book. You feel stupid and guilty and doubt whether there is a point in making any effort in the first place.

“If you don’t know what was in that book anymore, you might as well not have read it.”

“If you have no recollection of the content of last night’s conversation, the evening was wasted.” (Korsakoff has already started, I’m afraid.)

Gone with the wind

Realizing that I can’t recall a moment of experience makes me sad.

I’ve lost something that I didn’t want to lose and now it’s gone forever.

It seems that, for me, a moment of experience needs to become a part of my long-term memory to really matter.

If some moment of experience becomes part of your memory, you know what happened, you can refer to it, describe it, think back to it, talk about, relive it. If some moment of experience is not thusly registered, you don’t have these options.

That feels like a genuine loss.

For example, at the beginning of this summer, I had an amazing evening with a friend at a bar. We laughed a lot, but the majority of the moments of experience of that night didn’t stick and I’m unhappy about that.

Unfortunately, I forget moments of experience all the time, but I don’t think that’s on me.

Can you recall what you and your family talked about over dinner last Tuesday? Or what you even had for dinner?

That’s right — I didn’t think so.

In a very real sense, memories are all we get to keep.

Fewer memories, then, means that a lot is lost for me: I was there, but are these experiences truly mine?

Experience versus memory

At this point, another voice in my head enters the scene: “Dude, you had a lot of fun, with one of your best friends. Who cares about memory-containment?”

This conflict gets at something very deep:

We remember fundamentally different than how we experience.

For one, there are many cognitive biases — the peak-end effect, for example — that distort the accuracy of our memories.

But what I’m getting is that for our “remembering self”, other things matter than for our “experiencing self”.

“Most of them don’t leave a trace”

The remembering self maintains the story of our life. It’s the part of us that our doctor assesses when he asks how we’ve been feeling lately.

The experiencing self, on the other hand, lives life continuously. It’s the part of us that our doctor assesses when he asks if it hurts when he hits you there.

This experiencing self has moments of experience. One after the other.

What happens to these moments? Nobel Prize-winning psychologist Daniel Kahneman studies the differences between our experiencing and remembering selves, and according to him most of these moments are lost forever:

“Most of them don’t leave a trace. Most of them aren’t picked up by the remembering self.”

I don’t like this. I’m going to stamp my feet and insist that they should count for something. What happens during these moments is my life. It’s the finite resource that I’m spending while I’m on this earth.

That was the voice of the remembering self.

The experiencing self doesn’t get what he’s babbling about:

“Why should [memory] be necessary? Everything — our friendships and hatreds, the way we look, our handshakes, our books, our handwriting — bears witness to our being [whether we recall it or not].” -Friedrich Nietzsche, Untimely Meditations (1876)

This is what I meant when I said that different things matter to the experiencing self and the remembering self.

As such, when we reflect on what’s important in our lives and how happy we are, we can do so from two distinct perspectives.

Two kinds of happiness

The experiencing self cares about being happy in your life, whereas the remembering self cares about being happy with your life.

The first notion is about how happy a person lives. The second notion is about how satisfied the person is when that person thinks about her life. Here’s Kahneman summarizing the latest research:

“The main lesson we’ve learned is that they are really different. You can know how satisfied somebody is with their life, and that really doesn’t tell you much about how happily they’re living their life, and vice versa. The correlation is about .50.”

And a correlation of .50, it turns out, means that there’s quite the gap:

“What that means is if you met somebody, and you were told that his father is six feet tall, how much would you know about his height? Well, you would know something about his height, but there’s a lot of uncertainty. You have that much uncertainty. If I tell you that somebody ranked their life eight on a scale of ten, you have a lot of uncertainty about how happy they are with their experiencing self. So the correlation is low.”

Why is the correlation so low? It’s because we do not attend to the same things when we think about life and when we actually live.

When I reflect on how happy I’ve been, I’m evaluating — I conjure a picture of the past and process it. But my evaluation diverges from my experience — life as I live it from moment-to-moment.

When you think about how happy you’d be living in Budapest, for example, you’re likely to make these judgments based on factors that the reflective self deems important but which are not very important to the experiencing self.

For instance, one such consideration, says Kahneman, is the climate:

“It turns out that when people move to California in the hope of getting happier because of the nice weather, their experiencing selves don’t get happier. But they think they are happier! Because when they think about they’ll be reminded of how horrible the weather was in the previous place they lived and they will feel they’ve made the right decision.”

Adjudicating the conflict

How should we adjudicate the conflict? Which voice in my head should I listen to?

Does a lack of memories matter when the experience was worthwhile or when, as Nietzsche writes “everything bears witness to our being” anyway?

For something to matter, we must have reasons to care about it.

Let’s begin by asking whether we have reasons to care about the experiencing self.

The Pixel Theory

When you think about it, this is a silly question.

Of course we have reasons to care about our experience of actual life.

Consider, for instance, the ‘Pixel Theory’, coined by Tim Urban over at WaitButWhy.

If we focus too much on the remembering self, we start to see our lives as a picture depicting an epic story, assuming that the key to happiness lies in the broad components of the image. That’s a mistake, Urban argues, because it overlooks the experiencing self: we don’t live in the picture’s broad strokes. We live at all times in a single pixel of the image — a single Today:

“So while thousands of Todays will, to an outsider from far away, begin to look like a complete picture, we spend each moment of actual reality in one unremarkable Today pixel or another. It’s an error to brush off a mundane Wednesday and to focus entirely on the big picture, when in fact the mundane Wednesday is the experience of actual life.”

OK, I get it, life isn’t about the memories, but about the experiences.

But I still don’t want my memories to leave me.

Who I am

Do we, likewise, have reasons to care about the remembering self?

I believe that we do.

Philosophers of love tell us that one of the aspects that makes a long-term love relationship so special is that it emerges from a unique shared history between two persons. Those we love are irreplaceable because of their relation to the past.

We don’t want to go home to fully functioning but memory-free clones or duplicates of our loved ones. Rather, we want to go home to the loved ones with whom we have a shared history. If a loved one doesn’t remember our shared history, then our relationship has lost something very deep.

Memory is connected to identity more broadly. The ability to recall past moments of experience is crucial for me to be me.

When people can no longer remember many of their experiences — as in cases of dementia — we say that they are “no longer themselves”. In fact, some philosophers argue that when dementia takes someone’s memories, the previously healthy person and the demented patient should be regarded as two different people entirely.

If personal identity and ability to remember are so closely related, we would be wrong to dismiss the remembering self’s concerns as irrelevant because ‘life is about the experiences, dude!’.

Stop fighting, you kids

We have reason to tend to both our experiences and our memories.

A good life requires a satisfied experiencing self and a satisfied remembering self.

Therefore, we should not choose one to discard the other.

After all, Kahneman’s research indicates that their happiness is relatively independent.

The voices in my head, then, while taking themselves to be battling for overall domination, are merely arguing about priority.

It’s up to me, I suppose, to keep them both well-fed.

In that case, they’re gonna have to take turns 😃.