Transcript

Robert Wiblin: Hi listeners, this is the 80,000 Hours Podcast, the show about the world’s most pressing problems and how you can use your career to solve them. I’m Rob Wiblin, Director of Research at 80,000 Hours.

I listen to about 20 hours of audio on my phone each week, so I care a lot about making sure I’m absorbing what I’m hearing as efficiently as possible. If you subscribe to podcasts on your phone app, you’ll be able to customise the speed at which it plays each show, so it’s not so slow you get bored, but not so fast you can’t keep up. Many apps also let you chop out any silences, though rather than do that I prefer to just play it a bit faster.

The densest podcasts I listen to at 1.2x and the fluffiest at 2.3x or so.

Personally I use BeyondPod Podcast Manager on Android and I hear the best podcast app for iPhones is called Overcast.

I also use Pocket, an app which grabs articles from the internet for you to read on your phone later. Fortunately it’s able to read it to you from your phone, because my hands get sore if I have to scroll down my phone too much. It takes a while to get used to its style, as its pronunciation isn’t perfect, but after a few hours I didn’t have trouble following any more.

For audiobooks I use Audible, but you probably already know about that one.

Regardless of what I’m listening to I pay close attention to the speed so I don’t waste time listening to something less quickly than I could handle. You might like to do the same!

I should quickly apologise about the audio quality at the start of this episode which is bit worse than usual. Fortunately I noticed I was recording on the wrong microphone a quarter of the way through, so it’s back to normal quality from 36 minutes in.

Alright, here’s Spencer.

Robert Wiblin: Today I’m speaking with Spencer Greenberg, whose reviews the first time around were so positive that he has the privilege of being the first guest on the show to be interviewed twice. Spencer is still an entrepreneur, and he founded Spark Wave, a startup foundry which creates novel software products designed to solve problems in the world such as scalable care for depression and tools for improving social science.

He also founded ClearerThinking.org, which offers free tools and training programs that have been used by over 150,000 people which are designed to improve decision making and reduce biases in people’s thinking, which is going to be a big topic of our conversation today.

His background is in mathematics and he has a PhD. in Applied Math from NYU with a specialty in machine learning. Spencer’s work has been featured in major media outlets such as the Wall Street Journal, the Independent, Lifehacker, Gizmodo, Fast Company, and the Financial Times, and I could have said more, but I won’t. So, thanks for coming back on the podcast, Spencer.

Spencer Greenberg: It’s great to be here. Thanks so much for having me.

Latest projects

Robert Wiblin: So, today we hope to talk about how to reason through difficult questions more accurately and assess evidence and also when we should expect to be overconfident and when to be underconfident. But first, what have you been focused on in the ten months since we last spoke?

Spencer Greenberg: Yeah, so things have been moving along with Spark Wave. As you mentioned, we build new software products to try to solve problems in the world. If it’s sufficiently promising after we build the first version of the product, we’ll recruit a CEO, ultimately the goal is spinning out a new company. And so, we actually have four portfolio companies right now where we built the first version of the product, we’ve recruited a CEO. So, one of them is UpLift which is our automated program for helping people with depression. We’re trying to reduce depression symptoms. And that’s run by Eddie Liu. And then we have Mind Ease which is trying to be the best software product in the world for calming you down when you’re having significant anxiety. It’s run by Peter Brietbart. We have Clearer Thinking run by Aurora Quinn-Elmore. And Positly which is our recruitment platform for recruiting people for studies. It’s run by Luke Freeman.

Robert Wiblin: Cool. So, I think last time we talked about UpLift. How’s that going? And when might people expect to be able to use it?

Spencer Greenberg: Things are going really well. You know, we have people using UpLift every week, but we’re still in a closed beta, because we have some things we know we need to improve in that system. Almost done with that. We’re going to release it really soon.

Robert Wiblin: Okay. Nice. And how are the other projects coming along? Will we be able to see the results any time in the next year or two?

Spencer Greenberg: Yeah, definitely. I mean, I’m really excited about all four of these products. So, Positly, we actually just got Positly out on positly.com. Basically the idea is trying to accelerate social science research by helping you recruit the right participants the right time, get them to do the right thing. So, let’s say you’re running a study, a scientific study and you need 100 women age 20 to 24 and maybe you want to divide them into groups and have them do different things, and track them over time, have them fill out surveys at different points. That kind of research can be very complicated, expensive, time consuming and difficult to get right, and we want to make things like that much, much easier to help accelerate people’s work. And then also on the side of developing products, you know, if you look at why companies fail, very commonly, probably the most common reason is they build a product that people really don’t need or want. So, we think there’s huge room for helping companies be better at releasing new, better products by getting the data back from potential customers much earlier and more robustly. And so we want to help with that as well.

Robert Wiblin: How are you recruiting these people? Is this all Mechanical Turk?

Spencer Greenberg: So, the idea of the system is essentially to have multiple back ends to help you get just the right participants you need. Whether you’re weighing I want the cheapest sample possible really fast, or I want a representative national sample, or I want a sample from a specific place, so we’re going to be building out these different back ends. So the first back end we have built out is mechanical torque where we can layer a lot of really nice features on top of it to make it really nice for researchers. But then when we have these different back ends coming for just getting the perfect sample for you.

Robert Wiblin: Yeah, and what’s new with like Clearer Thinking?

Spencer Greenberg: So, Clearer Thinking we got a lot of real exciting stuff happening. We had some long term studies that we’ve been working on. One is actually on creating happiness habits, and this one I’m really excited about ’cause I literally think it increased my happiness five percent. I know that’s a bold claim and it stuck now three or four months now, you know? Fingers crossed. I don’t know whether that will last forever. But the idea of that project is imagine that you had let’s say 30 happy thoughts a day that you wouldn’t normally have, like what would that do to you? I think that’s a reasonable hypothesis that if people actually had 30 extra happy thoughts throughout the day, it would make you a happier person. So, how do we make that happen? And so, our idea was try to make that happen in the simplest way possible. By installing a trigger in your life that already happens, let’s say 20 or 30 times a day and making that trigger so strongly associated with the happy thought that it actually triggers the happy thought.

So for me, when we were developing the study and I was testing it, the trigger I set for myself was before I just check social media, when I have the thought of checking social media, I would think of something I’m grateful for, and it just worked to a crazy degree. I’m just way more grateful than I’ve ever been in my life. It’s just really been a wonderful experience.

Robert Wiblin: Is this a bit temptation bundling? So, maybe checking social media is a bit bad habit, but then you’re bundling it with this thing that’s helpful to you?

Spencer Greenberg: Well, I think of temptation bundling a little differently. I do use temptation bundling.

Robert Wiblin: Yeah?

Spencer Greenberg: It’s also one of my favorite techniques. I use it to work out, where basically there are TV shows that I only watch if I’m working out while I’m doing it. Choose a motivator that makes the working out more feel more pleasant. But this is a little bit different. This is really about creating a trigger in your environment. The trigger’s a certain type of thought. And really the whole thing is that at first it would only happen when I would go check social media. Now it happens whenever I think about social media and I often don’t even go check it. I just have a thought, oh, wait, then I feel grateful, and then, you know, I don’t even need to check the social media. It’s really cool.

Robert Wiblin: Okay, so the goal is to think of something you’re grateful for. Now do you ever have difficulty doing that?

Spencer Greenberg: So, this is another funny thing that happens psychologically. So, at first I would every time think of something I’m grateful for, but then it started getting like almost shorter and shorter until almost just like a happy mental maneuver. It’s a little hard to describe. But it’s just sort of like this happy feeling that I produce at that moment. After having done it hundreds of repetitions of it. We’re going to be running a study on this technique. It’s coming up soon, and then if we have good results, the goal would be to turn it into a tool so anyone could use the tool to try to create these happy associations to you know, generate 20 or 30 more happy thoughts every day.

Robert Wiblin: Yeah. What kind of happy thoughts do you have? Are you able to share them?

Spencer Greenberg: For me?

Robert Wiblin: Yeah.

Spencer Greenberg: Really for me I used gratefulness, so trying to think about all the things I have in my life. I have a funny way of doing gratitude that is especially effective for a certain type of people. Where I imagine myself not having the thing, and then I remember that I have it, and I feel really great about that. But that technique doesn’t necessarily generalize, ’cause I’ve showed that technique to some other people, and they say when I imagine not having it, it’s like so crushing that it doesn’t work. But that’s how I do it. So, I’ll be like,”Oh wow. Tea exists in the world. It’s such a wonderful thing.” And then I’ll contrast that. Imagine tea didn’t exist and it was impossible to get tea, and then I’m like, “Oh, but there is tea.” And that makes me feel really good.

Robert Wiblin: Yeah, after we spoke last time I tried out the UpLift app. I managed to get a link to it. It was closed beta. It was really good. Unfortunately, I became happier after a couple of weeks of using it, so I didn’t manage to complete the course.

Spencer Greenberg: I’m really glad to hear that. Not necessarily causal, but regardless whether it’s causal, I’m really glad to hear that you’re feeling happy.

Robert Wiblin: Yes. I mean, I thought it was very good material and I was thinking easy to motivate myself to do it ’cause you had all of these entertaining explanations and examples through it. So, as soon as that goes public, I think I’ll announce it on the show and encourage people to give it a crack.

Spencer Greenberg: That’s great.

Intrinsic and instrumental values

Robert Wiblin: All right. Let’s move on to a talk you gave at EA Global last week on intrinsic values and instrumental values where you did a big survey at trying to figure out what do people actually value. For its own sake, what did you find out?

Spencer Greenberg: This is a topic I’ve been thinking about a lot lately. So, first let me try to define an intrinsic value. So, there are many things we all value like money, food, et cetera. But most of the things we value we only value for their effects. So, imagine something and say, “Do I value it?” If the answer’s yes, you can say to yourself, “Well, would I continue to value it even if it got me nothing else? Even if there were no effects?” If so, it’s an intrinsic value. Or another way of putting that is an intrinsic value is something you value for its own sake not [only] for the effects that it has. So for many people like their own pleasure is an intrinsic value, but money is not an intrinsic value, because money if it couldn’t get us any effects, like say, there was hyperinflation, it would be worthless, like we wouldn’t care about money at all.

Robert Wiblin: Okay, so who did you survey?

Spencer Greenberg: We ran a survey where we got a bunch of different participants. Some of them in fact effective altruists and fully identify, some of them partially identify as effective altruists, and some of them non effective altruists. We looked at four different variables: at age, how liberal or conservative people are, gender, and whether they identify as an effective altruists, and we looked at what intrinsic values people report having. And this was a very challenging study to run, because it’s very hard first of all to get people to understand exactly what an intrinsic value is, but it’s even harder to get them to kind of do the proper mental maneuver of assessing whether they value a thing even if it doesn’t have any effects.

So, all those results, you definitely have to take them with a grain of salt keeping in mind that it’s tough to really get people to do this properly.

Robert Wiblin: Yeah, so you had some checks at the beginning of the survey, right? To see whether people had understood, then you tried to exclude people who didn’t get what the questions actually meant.

Spencer Greenberg: Exactly. We excluded a lot of participants because basically we taught to them about the idea, then we gave them a quiz to make sure they understood it, and if they got more than one question wrong on the quiz, we excluded them, and then we actually had them state in their own words what an intrinsic value is and we read all of those and if they seemed incompatible with our definition, we also excluded them. So, we exclude a lot of participants before analyzing the data, but even still, you still have to take it with a grain of salt. But yes, we try hard to make the data as valid as we could.

Robert Wiblin: So, what were the main finding from the study?

Spencer Greenberg: Yeah, so we looked at which intrinsic values were associated with different groups of people. To give you some examples, so for conservatives, we found they tended to report an intrinsic value around religion and things like retribution where those who’ve done bad things get punishment for it. And also the preservation of existing values, which makes a lot of sense, ’cause to some people that’s really what conservative means. Kind of preservation values. On the other hand, people who were reporting liberal, they tended to value things like animal well-being, nature, and happiness of strangers.

Now, I should point out what we’re looking at here are the things that differentiated each group. In other words, not the things that necessarily conservatives valued highest and liberals valued the highest, but the things that differentiated them from all other groups.

Robert Wiblin: What was distinctive.

Spencer Greenberg: Exactly. Exactly. So, we also looked at females versus males. Females reported intrinsic values around kindness or caring, diversity and human freedom more often. Males reported ones around their own selfish interests, the interests of people they know personally, and also around pleasure of strangers. When going to older versus younger, we found that older people reported intrinsic values around being cared about or trusted also about society’s morality in general. Younger people around things like animal life-spans and that they themselves are admired and also that the people they know have pleasure. And finally, looking at effective altruists versus non-effective altruists. There really we found a very clear pattern, effective altruists tended to value the happiness and suffering of all conscious beings, and that was really the big differentiator from other groups.

Robert Wiblin: Yeah, I think you broke them into universal and non-universal values, is that right? Things that are like not specifically about your life or about people you know and ones that were specifically either you or people you know or things about your own life.

Spencer Greenberg: Exactly. I like to categorize intrinsic values in three groups. “Self” ones that are like, “my own pleasure”, things like that. Then there’s community ones which are about people that are special to you. So either people from your in-group or people who are your friends or whatever. And then universal ones are all the others. They’re not necessarily about people, but they could be about people. They could say I care about the suffering of all conscious beings or I could be I care about there being beautiful things in the world. Right?

Robert Wiblin: Yeah. And what did you find there?

Spencer Greenberg: Well, one of the interesting things that most people actually did report having at least one universal intrinsic value. I think that’s kind of interesting, because universal intrinsic values are reasons for people who don’t know each other, essentially people even on different sides of the world from each other, to cooperate and work together because there’s something they care about that’s not about themselves, not about the people they know personally, and they can kind of collaborate. I think one way to think about effective altruism is it’s people that have universal intrinsic value around reducing suffering around the world, increasing well-being, getting together to figure out how do we do that. How do we all work together to increase that.

Robert Wiblin: Yeah, so I guess we’ve talked about some of the differences, but just like what are the just boring findings? What are the [things] most people care about?

Spencer Greenberg: So, if we look at people who don’t identify as effective altruists we found that 82% of them reported that “I love other people” as an intrinsic value. Kind of interesting. “That I myself feel happy” was 74%, not too surprising. “That I continue to care about other people” 74%. “That beautiful things continue to exist” 71%. “That people I know personally feel happy” 69%. So, nothing like super surprising, but there’s a long list of these that a lot of people report as intrinsic values. Maybe they’re making a mistake. Maybe they’re misunderstanding their intrinsic values, but I think it’s at least worth keeping an open mind that maybe people actually do kind of intrinsically value these things.

And one way to think about that, like what is really going on here is that if we think of our brain as a machine, one of the operations that our machine brains have is this, “I value this thing” operation. Right? And so, it isn’t necessarily that strange or surprising to say there might be quite a lot of different ideas or concepts or things that our brain does this “I find this value” operation even when you remove all the effects of that thing.

Robert Wiblin: Yeah, so for example, we’ve got 71% of people who didn’t identify as effective altruists valuing that beautiful things continue to exist. Do you think that you’d get that high number if you said that beautiful things continue to exist, but nobody sees then, which kinda seems like it should be a requirement if you’re ruling out any impacts of that.

Spencer Greenberg: That’s a really good question. I imagine that probably would reduce the number ’cause it would cause people to like second guess and double think about that as an intrinsic value, but interestingly enough I know two people who are very savvy philosophically and introspective and they both claim beautiful things existing if nobody sees them are actual intrinsic values. I had this talk with them, they’re like, “No, no, no.” I get it. I actually value these things even if it’s on an alien world that no-

Robert Wiblin: That no one ever sees.

Spencer Greenberg: … conscious beings will ever see. So, you know, I think what we’re really getting at here is a psychological thing, right? When I’m talking about intrinsic values, I’m talking about a psychological phenomenon. I’m not talking about like some universal truth, and so there’s isn’t necessarily a wrong answer to what someone’s intrinsic values could be.

Robert Wiblin: Yeah, so what did effective altruists tend to value?

Spencer Greenberg: Yeah, so as you could very much imagine effective altruists tend to value things around suffering and happiness of conscious beings, but they also, about 84% of them said that people I know personally feeling happy is an intrinsic value. It makes a lot of sense. 81% the animals feel happy. 80% that I suffer less than I do normally. So, I think this is really interesting, because I think as a community if you look at kind of what is unique about the effective altruistic community with regard to intrinsic values, like there’s lot of unique things about it, but with regard to intrinsic values I think it really is about how much the effective altruism community values the suffering and well-being of all conscious beings, but if you look at effective altruists individually, many of them at least report having other intrinsic values. In fact, many intrinsic values. So, you get a much broader view that maybe you lose when you’re kinda staring at the community as a whole.

Robert Wiblin: Yeah, were there any non-welfare things that effective altruists were particularly interested in?

Spencer Greenberg: I think that effective altruists, quite a few of them said that they continue to learn being an intrinsic value. 40% said that “people I know are able to get the things they want”. That’s a little bit of a preference satisfaction thing. 33% said “that humanity does not engage in immoral acts”. So, that’s kind of interesting.

Robert Wiblin: Interesting. So there they might have kind of a justice conception, so even if injustice is being perpetrated, even if they are in effect welfare, they might be against it. Yeah, so were there any surprising results maybe that made you question whether the methodology worked?

Spencer Greenberg: Well, you know, I definitely, like I said, definitely take this with a grain of salt. Like, I was pretty surprised by the number of people reporting beauty being a value. And you know, it does make me wonder whether people are fulling interpreting it properly. But I think this exercise of trying to figure out your own intrinsic values is a really useful thing to do. I actually see five reasons to try to figure your own intrinsic values out.

Robert Wiblin: Yeah, go for it.

Spencer Greenberg: If you wanna jump into that. Yeah, so the first reason I think it can be useful to figure out your own intrinsic values has to do with what I call “value traps” and trying to avoid them. A value trap is when you associate something with an intrinsic value, because it used to be associated or maybe you just had the false belief that it was and then you pursue the thing without actually getting the intrinsic value out of it. So, an example of this might be maybe when you were young like not having that much money reduced your autonomy. So, you associate money with autonomy. So, you end up getting a career where you make tons of money, but you work so many hours you actually have very little autonomy. But you continue doing it, because you have inertia and you don’t really pay attention to the fact that it’s not getting you the intrinsic value you were seeking. And I think this is actually shockingly common how often we kind of do these things that were like vaguely associated with intrinsic value, but don’t get the intrinsic value out of it.

Robert Wiblin: Yeah, do you think actually people by association end up valuing the thing terminally. So, you wanted to get this job because you thought it would have pretty good consequences but then it just becomes so hooked in with your mind as something that you desired and worked towards that in fact you do just end up valuing it for its own sake?

Spencer Greenberg: Because it’s a psychological phenomenon, I can’t for sure that people couldn’t come to value it intrinsically, but I think usually what happens is they just aren’t making a careful distinction between intrinsic and-

Robert Wiblin: Instrumental values.

Spencer Greenberg: Yeah, instrumental values, and so what happens is because they’re not making that distinction, they just keep pursuing the thing ’cause they think it’s valuable, it’s valuable, it’s valuable, but if you actually force them into facts mode, they’re like, “Really? Do you think that money is inherently valuable even if it’s useless?” They’d be like, “No, probably not.” And then they’d start to separate out that value and realize it’s not intrinsic value.

Robert Wiblin: Do you have any examples from your own life where you confused terminal and instrumental value?

Spencer Greenberg: Well, you know, I think that as someone who loves learning, I think it can be easy, and I think I’ve made this mistake of like viewing learning as good in and of itself. And I think learning can actually be kind of a dangerous trap in that way, because there’s something about the wiring of many people that where you’re learning and you feel like you’re making progress, right? But maybe you’re learning something that’s useless. Maybe you’re learning something that’s even wrong. But your brain doesn’t care, right? You’re learning and you’re feeling good. You feel like you’re improving.

Robert Wiblin: It’s stimulation.

Spencer Greenberg: It’s stimulation, but it’s also like if you’re playing a video game at least you know you’re not doing something productive. When you’re learning, you kind of like, you may be doing this thing, I’m productive, because it feels productive. That’s actually the worst kind of non-productive activity, right?

Robert Wiblin: Yeah, that’s interesting. I do know some people who said they value learning for its own sake even if it has no positive consequences.

Spencer Greenberg: Absolutely. And quite a few people in the survey said that. I don’t think that I value it for its own sake. I think I value it because of the effects it has.

Robert Wiblin: Yeah, I’m the same and I put to them, what if you just had a bunch of true facts that were stored in a hard disk somewhere that no one ever actually plugged in? Like in a sense there’s more like knowledge in the universe encoded somewhere … and I guess it’s very exquisite that it has not effects on any [crosstalk 00:19:19].

Spencer Greenberg: Yeah, I imagine that’s good intuition pump. And this is the thing about intrinsic values, we have to use these intuition pumps to like do these thought experiments carefully, and our intrinsic values will kind of shift as we do thought experiments, and so for example, when you start considering scope, right? You imagine one person suffering, you’re like, “that’s bad”, and you imagine ten people suffering, “that’s worse” a thousand people suffering, a million people suffering. At first I think people will be like, “Yeah, that’s worse and worse.” But their intuitive feeling will almost be as though a million people suffering is only slightly worse than one person suffering, which is clearly, you know. When you think about it, you’re like, “No, no, no, it’s way, way worse.”

And that is a intuition pump and I think it’s a lot of people caring more about suffering and realizing that value of suffering is kind of almost sort of an unbounded value in the sense that l there can be so much potential for suffering that it can be a value that becomes extremely important. Whereas, other ones like maybe about personal suffering are more bounded in a way.

Robert Wiblin: Yeah, okay so that’s the first reason to have to worry about this distinction. What’s number two?

Spencer Greenberg: So, number two is helping us plan. So, something when we think about something as being like the good thing to do, like something we want, but we don’t carefully think about what is the intrinsic value we’re getting out of it and we make a really kind of inefficient plan to get the intrinsic value. So, for example, imagine that you really want to understand the universe. That’s a kind of intrinsic value of yours and you associate that with being a tenured professor ’cause your professors can sit around and think a lot of about the way things work, and so you decide to become a tenured professor and you have this really crazy like 15 year plan of how you’re gonna get there and you start pursuing it. But if you’d notice that the main reason you were drawn to that was for the intrinsic value, you may have been able to make a much more efficient plan. Like just spend your free time studying the way things work, and maybe that would’ve achieved most of the intrinsic value without this tenure plan. And this goes to the idea of like goal factoring, it’s sometimes taught in applied rationality type workshops.

Robert Wiblin: Yeah, do you just wanna quickly describe goal factoring?

Spencer Greenberg: So, the idea of goal factoring is that if you think about how you’re going to achieve a goal and then you think about why you want to achieve the goal, once you’ve broken that down, you can start considering, “Okay, I’m trying to achieve this goal for reasons A, B, and C.” Can I come up with another plan that might get me A, B, and C also? Maybe there’s a better way to do it than my original plan of how to get there.

Robert Wiblin: Yeah, I guess shockingly often you find that you weren’t accomplishing the goal in the most direct way.

Spencer Greenberg: Exactly. Exactly. So, I think that’s one of the benefits of understanding intrinsic values, ’cause the intrinsic values are sort of like the end nodes of your value system. So, they’re like you know, those are the things you’re in some sense trying to get according to your own value system, so if you don’t know what those are, it can be hard to make efficient plans.

Robert Wiblin: Yeah, why do you think it is that humans kind of by design seem to get fixated on intermediate nodes rather than thinking through about what they’re trying to accomplish ultimately and then going directly there?

Spencer Greenberg: It’s an interesting question. I think that the human brain just doesn’t clearly differentiate between value and intrinsic value. So, it blurs those two things together and it kind makes sense, because … okay, getting food … very, very useful for survival, right? The fact that it’s not an intrinsic value if that was demotivating to humans that might not be great for survival, right? It’s like very important we try to get food. Food is not the end goal, but like you know, the fact that the brain treats I as an end-goal a lot of the time would kinda make sense, right?

Robert Wiblin: Yeah, so a lot of the time it doesn’t matter too much, but then sometimes it causes you to go really astray when you have-

Spencer Greenberg: Yeah, every once in while it causes you to like spend your entire life doing something pointless, ’cause you never clearly separated your intrinsic values from your other values, right?

Robert Wiblin: Yeah, I wonder if … well, one thing would just be our brains are just like not that great and they make kind of random errors or they just don’t reflect us sufficiently. Another one might be that the environment’s changed such that our ancestors they were around what was more simple and they didn’t have to spend a lot of time reflecting on terminal or instrumental and intrinsic values, whereas these days our plans tend to be more complicated, have more steps, more intermediate outputs and so now we need to reflect more on this system to kind of level about whether we’re actually accomplishing the thing that we ultimately want.

Spencer Greenberg: That’s a good point and I would also say there are a lot of things that cause human behavior that are not intrinsic values, right? Habits, for example. Or reward and punishment through the mechanism of operative conditioning. Automatic biological responses. So, there are many things that drive our behavior. I think of intrinsic values as like one of these drivers of behavior. And the metaphor I like to use is intrinsic values are kind of like a beacon shining off in the distance. Like most of the time you’re in your boat and you’re just focused on rowing and you’re trying to dodge the waves that are hitting the boat, but like every once in a while you kinda look out and you’re like, “Where am I trying to get to again? Why am I rowing in the first place? Am I headed in the right direction?” That’s the kind of role intrinsic values play.

Robert Wiblin: All right. That’s number two, what’s number three?

Spencer Greenberg: So, number three is thatI think understanding your intrinsic values can help you better understand and handle a kind of social guilt that’s pretty common where you know, we all are exposed to the values of other people. You know, our parents growing up, our community, our current friends, and so on, and when their intrinsic values are different than our own, it can create these really weird feelings. It’s like everyone else around you really values this thing, and you don’t really value it that much. Maybe you really value this other thing that they don’t value. And so you start feeling like oh, there’s something wrong with me. I’m an imposter. Maybe I’m a bad person. Maybe you feel guilt.

An example of this might be like you know, your parents expect you to have such and such career, but that career doesn’t get you the things that you intrinsically value, and so, like what’s wrong with you? You know? Why are you a bad child or whatever. And so, I think once you reframe this in the view of intrinsic values, and you’re like, “Wait a minute. So, my parents have these intrinsic values. My friends have these and I have these.” It just helps you understand it and potentially feel much less guilty and just kind of be like, “Okay, this is what’s happening.” I think it can help you relate better to those communities as well.

Robert Wiblin: Okay, yeah. I completely agree. What’s number four?

Spencer Greenberg: So, number four, this one’s subtle, is I think understanding your intrinsic values can help you avoid a kind of strange double think that I think sometimes occurs. The double think is around, it’s really the last one actually, it’s when you think that you’re only supposed to have certain intrinsic values, but you actually have others. So, I’ve seen this happen in two ways. The first way is that you believe there’s an objective truth about what’s valuable. Like maybe you’re convinced on some philosophical theory of morality and you think the only thing that is valuable is what that says, like utilitarianism, for example.

The other way I see it happen is the people around you in your social circle, they only accept as a community, accept certain intrinsic values, right? So, you think I’m only supposed to have certain intrinsic values, but you actually have other ones. So what you try to do is you try to recast your other intrinsic values in terms of these ones that are acceptable or valid. An example of this would be when someone says something like, “Well, the reason I cultivate friendship is because it makes me more effective.” I think what’s going on there very often is kind of self-deception where people feel like they’re not supposed to have these intrinsic values, so they kind of trick themselves into thinking that they don’t have other intrinsic values.

This leads to what I think is a really important and subtle point that whatever you believe is objectively true about value in the universe, and whatever you believe is like the right values you’re supposed to have according to your social group, those things are independent from what your current intrinsic values are. Your current intrinsic values is like a psychological fact. A scientist could study your intrinsic values and answer the questions. It’s a fact about yourself. It’s not a fact about the universe. It think it’s very important to draw that distinction and say you might believe in objective moral truth and you might believe you figured out what it is, but it doesn’t mean that’s what your intrinsic values are right now. Maybe you aspire to make your intrinsic values match them more closely, but they’re probably not there yet, and if you don’t draw that distinction, you might end up having this very bizarre doublethink where you basically deceive yourself and create these weird psychological effects potentially harmful.

Robert Wiblin: Yeah, so I think I often hear people say that the reason I’m just having fun or I’m going on holiday, I’m not actually working to improve the world right now is just-

Spencer Greenberg: So I don’t burn out.

Robert Wiblin: Yeah. So, one thing is yes, so you get these explanations like ’cause I don’t want to burn out, so it’ll be like I’ll be able to work even harder later on. Sometimes that’s true, sometimes it’s not. But every so often I hear people say I know, it’s because of weakness of will or it’s like, on some level I wish that I wasn’t doing this, but in practice I can’t actually make myself work all the time or I just actually just don’t care enough about the world to work that hard. But it sounds like you’ve seen a lot of people in your social group just try to trick themselves every time that it’s always for some greater good.

Spencer Greenberg: I definitely have seen this and I think that some of the EA community is especially susceptible to is this,not drawing clear distinctions between these like my intrinsic value,with what I think my intrinsic value should be, versus what I think the universal truth about intrinsic values are, and those are different things and you should understand that. And like one way to recast that, like, oh, I’m going to go on vacation, is like, oh I have an intrinsic value of my own happiness and that’s okay. There’s nothing wrong with that. Being in denial of that fact doesn’t make it go away. If you actually want to change your intrinsic values, you still need to know where you’re at and then think about what you want them to be, rather than pretend that they’re already there.

Robert Wiblin: Yeah, I mean, I would say I do as a brain, I value my own welfare more highly than other people, and I don’t think that that’s good; nonetheless, I make peace with that and pursue my own interests to a reasonable degree and I don’t feel bad about it ’cause I don’t think that’s gonna help.

Spencer Greenberg: Yeah, and personally, I think it’s healthy to say, “I have a bunch of intrinsic values. Some of them are about helping the world. Let me support those values by devoting a certain amount of my efforts to really doing the best I can at promoting that universal intrinsic value of reducing suffering or increasing well-being.” Or whatever it is. Then I also have these other intrinsic values which I balance against that, you know? So, I’m not willing to like necessarily, utterly destroy myself for my universal intrinsic value, but I’m willing to work really hard for a long time and make it a big part of my life and that kind of a thing. I think that’s the healthy way to balance it.

Robert Wiblin: Yeah, I agree. All right. What’s number five?

Spencer Greenberg: So, the fifth and final reason why I think it can be useful to understand your intrinsic values is I think it can help us when we’re thinking about the vision of the world we want to create, because it’s pretty easy to say like on the margin ways we can make the world better, like if there was less disease, less poverty, less suffering, I think most people would agree, that’s good. But when we actually start thinking about what world do we want to make. Like, in hundreds of years if humanity makes a new world, what do we want that world to look like? As soon as you start trying to describe that world, weird things happen. First of all, if that world is built on just like one or two intrinsic values, a lot of times, that world will sound unappealing, even to yourself. Even if you think that those are your only or prominent intrinsic values. Second of all, it will probably sound even less appealing to other people, and so-

Robert Wiblin: If you don’t share those values or weight them as highly.

Spencer Greenberg: Exactly, and so I think if we’re trying to make a world that like generally is broadly appealing and lots of people will want to be in and we ourselves even will want to really be in and then we’ll think of it as optimal. We have to really consider multiple intrinsic values and try to build a world that like has this complex set of intrinsic values that it supports. Otherwise, you’re building a world for just like a small subset of people. And a classic example of this is that if you’re hedonic utilitarian, you know, by some forms of logic people come to the conclusion that the best world is like just hook everyone up to a happiness machine, right? I think a lot of people even people who think of themselves as hedonic utilitarians actually don’t think that that’s the best world. Right, like even according to what they think of as their value system. Right?

Robert Wiblin: Yeah, I guess I do or I would put myself on a happiness machine if there wasn’t like anything useful that I could do and I would recommend that to other people.

Spencer Greenberg: What if some people didn’t want to be in it though?

Robert Wiblin: Yeah, so then I think I wouldn’t force them for like pragmatic reasons potentially.

Spencer Greenberg: But not because you value their preferences.

Robert Wiblin: No, that’s right. Yeah. I suppose moral uncertainty is another thing. I think the second thing that I care the most about after welfare is autonomy.

Spencer Greenberg: Well, then so autonomy is a reason not to push people to be in some situation they don’t want.

Robert Wiblin: Absolutely. I’d recommend it to them, but I wouldn’t ever require anyone to, but it’s interesting that if you do have this view that there are objectively valuable things, and that people could be mistaken about what’s valuable, then you might want to go for quite an extreme future or a future that’s not appealing to everyone. Then I guess in reality you’d want to compromise with everyone, because you don’t have like total power over the future and everyone gets more if you compromise rather than fight.

Spencer Greenberg: Absolutely. So there’s the pragmatic reasons to compromise so we can all work together. There’s also moral uncertainty reasons. Okay, so you think this is the only good thing, but are you sure?

Robert Wiblin: Yeah, right.

Spencer Greenberg: Philosophers don’t think so. I think, what, like a quarter of them are consequentials or something like that? I don’t remember the exact number.

Robert Wiblin: Yeah a quarter or a third or something like that.

Spencer Greenberg: Yeah, so there’s a lot of disagreement about these things. So, I think that’s a really good reason. So, compromise is a good reason to include multiple intrinsic values, moral uncertainty is a good reason to include multiple intrinsic values, but also, if you don’t believe in objective moral truth, yet you still claim that the only thing that matters is, let’s say utility, well, I’m a little confused. Because if you don’t think there’s objective moral truth, in what sense is that the only thing that matters? Well, you’re definitely not describing your brain, because that’s not the way our brains work. Human brains have multiple intrinsic values.

Robert Wiblin: So, it seems like there are some people who don’t believe there are objective answers to what’s intrinsically valuable. And it’s true that in their own life they will weight their own welfare more highly and things like that just because of, you know, like how humans are as a species, but then when it comes to like much more abstract cases or cases where kind of resources aren’t limiting or they’ve changed their own … they’ve like make their own life and their friends’ lives kind of go in the normal way off of humans which isn’t only considering welfare, then in a much broader picture, they’re willing to I guess work towards a much stranger world where you optimize only for particular things that I think are valuable in the abstract. I guess for those people it’s not clear that they’re making a mistake. It’s true, yeah, if they think about their own future, what future would they create for them personally. It would be probably a lot more human than like a happiness machine. But then when they’re thinking about you know, what should I turn rocks into it might look more like a happiness machine.

Spencer Greenberg: Yeah, well, you know the way I explain this is well, first of all, people can be convinced that certain things are objectively valuable. And if they’re convinced of that, they might try to maximize that thing. I worry about that, because I think there’s, for the reasons mentioned, that that’s potentially dangerous to just focus on one intrinsic value. But if they don’t believe in objective moral truth, then really, what are they doing? Like, in some sense what is there to do other than reflect on your own intrinsic values [in that case], because you don’t believe there’s anything out there, so all there is is to like examine what’s here in your mind. When you do that, I think that, and I think my data also hints at this, you get really a complex system of intrinsic values. There’s a lot of different things that people seem to value and they include things like people getting what they want, people not suffering, justice and all kinds of other things.

And I think you know so, not all of those intrinsic values will drive people to the same degree, right? One person may be more driven by their own happiness. Another person may be more driven by the happiness of their community and the third person might be more driven by the happiness of all beings, right? So, even if all three of those people actually had the same intrinsic values they might have different strengths of like how driven they are by those values.

Overconfidence

Robert Wiblin: Okay, let’s move on to another study that you’ve done about overconfidence this time. What was that about?

Spencer Greenberg: Yeah, so this is a really new result. We’ve been working on it for a while, but kind of just coming together in a final result. We’re looking at whether we can predict on what sort of skills people tend to be overconfident when rating themselves relative to others. And we’ve actually done a series of three studies to try to get this result. It’s pretty complicated to try to produce. In one of the studies, we had each person evaluate out of a hundred people of their own age and gender, who live in their area, how many of those hundred people do they think they’d be better than at some particular skill?

So we had them rate a hundred such skills and then we can look for each skill, what is the average rating people give? So on average, they think they’re better than 30 out of 100, that suggests that maybe people will be under-confident at that skill whereas on average they rate each other at 70 out of 100 maybe people are overconfident at that skill.

Robert Wiblin: Okay, so you were looking at in which cases they were overconfident and which cases they were underconfident?

Spencer Greenberg: Exactly, we want to be able to predict. And so I want to give you guys a little quiz now, I’m going to read you a few skills and these are some where we found pretty strong findings in either overconfidence or underconfidence, see if you can guess which ones are overconfident versus underconfident.

Robert Wiblin: I actually don’t know the result so I can actually guess blind, yeah.

Spencer Greenberg: Great, okay, you ready?

Robert Wiblin: Yeah, go for it.

Spencer Greenberg: All right. So “knitting a sweater”? People are overconfident or underconfident?

Robert Wiblin: Yeah, I’m going to guess that few people have done that and they’re going to think, “Well I don’t know anything about knitting,” and they’re probably going to be underconfident actually.

Spencer Greenberg: That’s right, we found that they were underconfident. How about “thinking critically”?

Robert Wiblin: I think that too many … almost everyone thinks that they’re more rational or more intelligent than other people so they’re going to be overconfident.

Spencer Greenberg: Yeah, we’re way overconfident so people rating themselves better than about 71 out of 100 people about that on average. Okay, how about “lifting 10 pounds”? So this was a bit vague, we didn’t specify what it means to be lifting 10 pounds, but “lifting 10 pounds”.

Robert Wiblin: I think people will be appropriately confident because they’re just going to be baffled by what this is.

Spencer Greenberg: People would pride themselves being better than 71 out of 100 people at lifting 10 pounds.

Robert Wiblin: What does that even mean?

Spencer Greenberg: There is one theory that when there’s ambiguity, people tend to use the ambiguity to their advantage to interpret it in a way that makes them look good. We didn’t necessarily find that as a result, but that is one theory that’s out there in the academic world about how people deal with ambiguity. Here’s another one, “running a marathon”.

Robert Wiblin: I think most people are going to imagine that running a marathon is very unpleasant and that they haven’t done it. On the other hand, they might think that everyone else is terrible at running marathons too. All right, appropriately confident?

Spencer Greenberg: We found people were underconfident.

Robert Wiblin: Under?

Spencer Greenberg: This is, I think an interesting topic, because a lot of people have heard that people are overconfident, people are overconfident, but actually it’s not true universally. There are things people seem to be underconfident in and so we did find that out of these 100 skills that people tended to be overconfident not underconfident, but there still were quite a few [where] people [were] underconfident and so we thought that was pretty interesting.

Robert Wiblin: Yeah was there any unifying theme for what things people were overconfident and underconfident about?

Spencer Greenberg: Yeah, so we actually looked at what traits of a skill are predictive of people rating themselves as being overconfident or underconfident in that skill and we did this by having people rate the specific … how many people out of 100 they’d be better than at the skill, but we also actually ran another study where we had people do this skill and so we could actually see whether they truly were overconfident or underconfident compared to their prediction.

So we analyzed all that data and we actually looked for variables about a skill that were predictive in both cases. In other words, when people were rating themselves in the abstract relative to other people, but also when they were actually then going to then do this skill and we could actually measure it. We actually were able to find five different traits about a skill that seemed predictive in both cases of whether someone’s going to be over or under confident.

Robert Wiblin: What were they?

Spencer Greenberg: So the first one is how good people think they are at the skill on average. So if it’s a skill that a lot of people say they’re good at, then people tend to be overconfident in our data. Our second one is whether people feel that the thing’s something that’s a matter of personal opinion. Like whether you’re good at it is a matter of personal opinion. So maybe writing a novel, how good you are might be a matter of personal opinion whereas maybe throwing a dart at a dart board, maybe there’s more objective matter of whether you’re hitting the dart board.

Robert Wiblin: In that case, actually everyone could say that they’re better than average and be right by their own lives because they have a different sense of what it is to be good at that particular skill.

Spencer Greenberg: Yeah and I think that’s right and that goes to that ambiguity point. When it’s ambiguous, maybe people are using that to say that they’re good at the thing. Third trait we found is how experienced people say they are on average doing the thing. Strangely, potentially, you might think it’s strange, as people reported being more experienced we found that they can be more overconfident in the thing.

Fourth, how much people say that the activity reflects someone’s personality or character. So maybe the skill is making friends and people might think that really has to do with your personality and character so they may be more overconfident. Whereas maybe rolling a bowling ball, maybe people think that’s less to do with your personality and character, maybe they’re not so overconfident.

Robert Wiblin: So if it’s about their personality, then they tend to be more overconfident. If they view it as more a core aspect of who they are.

Spencer Greenberg: Exactly. And finally, how difficult people think the thing is. So if they rate the thing as being very difficult, this is the one that goes in the reverse direction. If they think the thing is very difficult, they tend to be underconfident [crosstalk 00:39:59].

Robert Wiblin: Which kind of explains the marathon.

Spencer Greenberg: Yeah exactly. So if you think about the marathon one, people might think it’s difficult, which could cause them to be underconfident, they may not view it as part of their personality or character potentially, so that could explain why they might be underconfident. People are not experienced so that could explain why they’re underconfident. They might say it’s not a matter of personal opinion so that could explain why they’re underconfident and so on. So you see, the marathon one lines up pretty well with a bunch of these traits actually to explain why people might be underconfident.

Robert Wiblin: What about the knitting one?

Spencer Greenberg: Yeah it’s an interesting question. I don’t know. I’ve gone through and for a bunch of the ones that had the most extreme results, scored them on these five traits, and just from that very simple way of doing it, the five traits do a pretty good job at the extremes. I’m not going to say that if you know these five traits of the scale you’re going to be super accurate but they do seem to be significantly correlated with whether people are overconfident or underconfident from our data at least.

Robert Wiblin: I guess, the knitting one people don’t have much experience with it, maybe they regard it as kind of hard because they wouldn’t know how to begin and they also … presumably most people would not be knitting as a core part of their personality. So those things push in the direction of being underconfident.

Spencer Greenberg: Right.

Robert Wiblin: So there’s a pretty big literature on overconfidence and overplacement right?

Spencer Greenberg: There is.

Robert Wiblin: So how does this gel with the broader literature?

Spencer Greenberg: Well we started out looking at literature and trying to take traits that they found but we weren’t getting the R squared as we wanted it. So we weren’t getting … so in other words, we took a few traits from the literature that people had said were correlated with whether someone’s over or underconfident, and then we ran our first study and we just didn’t get the strong correlations we were hoping for. So then we went back and we just cast a really wide net. We tried to come up with all the traits we could through brainstorming, we actually crowdsourced other people’s ideas about what traits might predict over or underconfidence, we came up with a list of, I think, 21 different traits and then we just ran them all and tried to use large enough data sets that we could really afford to check all of them and that’s where we got this list of five. So that’s where it comes from.

Robert Wiblin: Must have had a huge sample to have enough … and also a lot of different tasks that … to get enough variety across these five characteristics.

Spencer Greenberg: Well yeah. So in one of our studies, we had 10 different tasks that people were doing and we actually had them do the task so we could measure how performant they were and we had them predict their performance relative to others before they did it. Yeah, we used large samples because it’s really hard to enter discussions about it.

Robert Wiblin: Yeah, what were some of the things you expected to have an effect but didn’t?

Spencer Greenberg: Well, it’s tough because a lot of the things are somewhat ambiguous. For example one of the things we expected to have an effect is how much it relates to your ego. We thought that if you would feel bad about not being good at it, maybe that would affect it and we just didn’t find a very strong effect there but then again, we did find that if it relates to your personality or character … so that’s kind of like being relative to your ego but not quite the same thing.

Robert Wiblin: Yeah, were you running a regression where you did all of these things at once and so maybe one of them was cannibalizing the effect of the other?

Spencer Greenberg: Exactly, maybe once you took into account personality and character, it kind of sapped the power out of ego or something like that. But these were the five … what we were looking for here is stability. We wanted to find the ones that through the different studies we ran, we got a consistent effect in the same direction, so it didn’t hinge on the details of the way we did the study and that’s how we came to these five. So we definitely wouldn’t claim these were the only five, you could also carve up the space of traits differently, right? For any given trait, you might well find other traits are correlated with it you could have used instead but these five were kind of robust ones that we were able to uncover.

Robert Wiblin: Yeah, so what are the broader lessons we could take away from this? I suppose one is people are always told that everyone’s overconfident all the time or that’s a simple story, but it’s much more complex than that.

Spencer Greenberg: Yeah, although I would say it is a general rule, it’s often true, there are domains that people tend to be underconfident in and these five traits can actually help you pick out what those domains might be. We’re exploring now building a little tool that you can put down a skill and we want to help you be able to actually make a prediction about whether people will be under or overconfident. So hopefully that will come out ClearerThinking.org in a few months.

Robert Wiblin: I guess there is this other concept of overconfident which I think is more technically called over precision, which is if you ask someone to say, “What’s the population of China? Give me a range that it’s 90% likely to fall into that range.” They tend to give too narrow a range. Have you studied that at all? Do you know what the results are there?

Spencer Greenberg: Yeah absolutely, people tend to give narrow ranges on these things and so that’s another form of overconfidence, it’s actually very confusing in the literature because people use these words interchangeably but they’re actually different. There’s a third type of overconfidence which is your absolute performance, not your relative performance to other people. When it comes to this kind of making a prediction interval of the likelihood of the number of people in China being between certain number and certain number, there that gets into the idea of calibration training which is that you can actually learn to make those interval estimates more accurate.

We’ve actually done a bunch of work on this. We’re working on a project where we’ll help train people on calibration and actually we made a tool before which you actually can find on ClearerThinking.org where we give you 30 things where half of them are common misconceptions and half of them are things that sound like common misconceptions but they’re actually true. For each of them you have to say whether it’s true or false and then you have to make a prediction how confident you are. At the end, we analyze it and give you an indication of whether you tend to be over or underconfident and teach you about your predictive capabilities.

Robert Wiblin: Yeah I did that one last year, it’s really fun to pick out which are the misconceptions and which are not and which are the fake ones that you guys have planted in there. Not to brag, I was pretty well calibrated.

Spencer Greenberg: Nicely done. Well you know, a funny thing about that, it was surprisingly hard to figure out what was true with regard to these common misconceptions and we actually made a few mistakes which we only learned over more than a year of people using the tool, we’d occasionally have someone who’s an expert in some really tiny domain be like, “Actually, I’m an Egyptologist and I know that this is not true.” So it took us probably a year to get all the bugs out of there. I think, fingers crossed, they’re all correct now but to me, that was really a lesson in how difficult it can be to figure out the truth about things and we would cite things from sources we thought were super reputable and it turned out they were citing someone, who was citing someone who was citing someone. You just go through the whole chain of citations and it bottoms out nowhere. You know?

Robert Wiblin: Yeah, I guess maybe if you’d had the right answers at the time, my calibration would have been much worse. I guess there is this phenomenon where you get myths and then you get myths about the myths so you get like a mythical refutation of it.

Spencer Greenberg: Oh, interesting.

Robert Wiblin: There’s a thing about spinach I think, like the amount of iron in spinach. So the original myth, I think is that there’s lots of iron in spinach and then there was this mythical refutation at how it was a decimal point error. So it’s like, oh no it’s a myth that spinach has lots of iron and the original one is false for a totally different reason, the decimal point thing is just also itself like an urban legend.

Spencer Greenberg: Oh my gosh, it’s terrible. Truth is really complicated to figure out and these are around things that people don’t generally have really strong burning opinion about, right?

Robert Wiblin: Yeah.

Spencer Greenberg: Think about how bad it is when people have a bias where they really want a certain answer to be right. It’s hard to figure out the truth even when people can be pretty dispassionate.

Robert Wiblin: Yeah I’ll see if I can find out the true story about iron in spinach. It does make you despair if we can’t even figure that one out, then what hope is there for anything else?

Spencer Greenberg: But on the plus side, putting it out there in the world and having people comment on it, I think it eventually debunks things and if you’re willing to listen to people’s opinions, you can make it incrementally more accurate.

Robert Wiblin: Yeah it’s interesting, I guess the internet has made it a lot easier to spread myths but also made it a lot easier, probably to correct them. Or you can find the person in the world who knows the most about this issue in Egypt and he can set you straight, whereas otherwise you just never have a hope of connecting with them.

Spencer Greenberg: Exactly.

Bayesian updating

Robert Wiblin: All right, so I guess this raises the general issue of how you can decide what’s evidence for a claim and what’s not and how much you should change your belief based on what you observe. You’ve spend a lot of time, I guess, thinking about this question of how to update accurately. Do you want to give an intro to this topic?

Spencer Greenberg: Absolutely, yeah, so this ties into the idea of Bayesianism which is probabilistic, mathematical theory of how much to change your beliefs based on evidence. The way I like to think about this is actually using an English language phrase that we like to call the Question of Evidence and we actually have a module on ClearerThinking.org that teaches you how to use the Question of Evidence to evaluate the strength of evidence. So if you’re interested, go check that out. But the way the Question of Evidence works is it asks the English language question, “how likely would I be to see this evidence if my hypothesis is true, compared to if it’s false?”

So let’s say if you got a three to one ratio, like you’re three times more likely to see this evidence if my hypothesis is true than if it’s false, that gives you moderate amount of evidence. If it’s 30 to one, you’re 30 times more likely to see this evidence if your hypothesis is true than if it’s false, that’s really strong evidence. If it’s just one, you’re as likely to see this evidence if your hypothesis is true than if it’s false, that’s no evidence, it actually doesn’t push you in anyway and then if it’s one in three, one third, then that pushes you in the opposite direction, it’s moderate evidence in the opposite direction. One in thirty would be strong evidence in the opposite direction.

So, I think what a lot of people don’t realize is all these equations and so on can be very confusing but there’s the English language sentence which is the only way to say how strong evidence is. That is the right sentence. Other sentences that sound similar, actually are not the right way to quantify evidence.

Robert Wiblin: Yeah, what other expressions do we use that you don’t like?

Spencer Greenberg: Well informally, people often think like, “Oh if this thing seems likely to occur if my hypothesis is true, then that’s strong evidence.” Well not necessarily, because again, you have to say-

Robert Wiblin: It’s the ratio.

Spencer Greenberg: It’s the ratio. How likely is this evidence to occur if my hypothesis is true compared to if it’s not true? So they might leave out the, ” … compared to if it’s not true.” So there’s a lot of ways that our brains can not quite use the right formulation of this and you have to kind of go back to that sentence and say, “Huh, okay but let me go back to the sentence and estimate that.” You’re generally not going to get a hard number. It’s not like you’re going to say, “The number’s actually 3.2,” but you often will have a gut feeling that, “Oh yeah, this evidence is actually quite a lot more likely if my hypothesis is true than if it’s not.”

Robert Wiblin: What’s that called?

Spencer Greenberg: So that’s called the Bayes factor. So the Bayes factor is the likelihood of seeing this evidence if the hypothesis is true divided by the likelihood of seeing this evidence if the hypothesis is not true. That quantifies the amount of evidence but then the question is what do you with the Bayes factor?

Well, it tells you how much evidence you have for or against something, for or against a hypothesis, then you have to think about, well what did I believe before that? That’s called your prior. So you kind of have this prior belief about how much more likely your hypothesis is than not your hypothesis and then you use the Bayes factor to get your new probability of how likely your hypothesis is relative to not your hypothesis.

Robert Wiblin: Okay, so you’ve got your original probability of the claim being true and then you multiply it by the probability of seeing the data if it is true over the probability of seeing that data if it was false?

Spencer Greenberg: Close. Okay, so it seems a little funny to work in terms of odds, but turns out that the math works much more nicely in terms of odds. That’s why we always do it relative. Like the probability of seeing this evidence the hypothesis is true, relative to the probability of evidence if the hypothesis is not true. That’s kind of an odds ratio of how much more likely something is than something else. Your prior is how much more likely is your hypothesis to be true versus not true before you looked at the evidence? So if you thought it was three times more likely that your hypothesis was true than that it wasn’t true before you saw the evidence, that was a three to one odds, now you get some evidence and you think that it’s 10 times more likely to see this evidence if your hypothesis is true than not true, then you’re going to take 10 times 3, now-

Robert Wiblin: It’s 30 to one.

Spencer Greenberg: 30 to one odds, right. So you start with certain odds and you’re adjusting your odds as you get evidence.

Robert Wiblin: Then you can convert that back to a probability if you like.

Spencer Greenberg: You can refer it back to probability if you like or you can just work with odds if you get comfortable doing that.

Robert Wiblin: Yeah, do you prefer using odds or percentages?

Spencer Greenberg: It’s much nicer using odds when you’re doing Bayesian updating, when you’re trying to figure out the strength of the evidence, because the formula works out really nicely. It’s simple multiplication.

Robert Wiblin: Yeah, okay. Do you want to go through a simple example? I guess other than the diagnostic test for breast cancer or something like that?

Spencer Greenberg: Yeah. The classic one. Is there a particular topic you want to do it around?

Robert Wiblin: Trying to think. Elections or soccer matches? I suppose the World Cup’s on right now, I’m tracking that a lot.

Spencer Greenberg: Oh no.

Robert Wiblin: Maybe I can walk you through it. You start out with two teams, maybe that are similarly matched and perhaps you originally think there’s a one in three chance that your favorite team will win, say, I guess … I don’t know, backing Spain or something like that. Okay, so the odds of winning there would then be one to two. And then say in the first minute, for simplicity, they score a goal. How might we update there? I guess then we need to look at … I guess it gets complicated pretty fast, right?

Spencer Greenberg: Right, so then the question you want to ask yourself is how much more likely would I be to see this evidence that they scored a goal if they are going to win compared to if they aren’t going to win and there it’s going to be a subjective assessment based on your intuition about soccer. Someone scoring a goal early in the game, is that much, much more likely to occur if the team ends up winning relative to if they don’t or only a little bit more likely to occur? Whatever, you probably have more soccer intuition than I do and this is where your experience is going to come in.

Robert Wiblin: Okay, interesting. Is it just subjective judgment at that point? I suppose you could have a more explicit model of a lot of the games and-

Spencer Greenberg: You could absolutely, yeah, if you had a data set of all the games that are played and let’s say they got a goal in the first 10 minutes, you could look at the data of all the times that one team gets a goal in the first 10 minutes before the other team scored, what percentage of the time did they beat that team, that could actually give you this evidence. You could literally do a calculation around it but if you’re just hanging out with your friends and you’re trying to see what’s the chance your team’s going to win that you prefer, you’re going to do a more subjective judgment of how strong is that evidence.

Robert Wiblin: Okay, so let’s say team scores in the first minutes, I’m going to say that I’d be three times as likely to see that if they are going to win as if they’re not, if that’s the right phrasing?

Spencer Greenberg: Yeah.

Robert Wiblin: So it’d be a likelihood ratio then of three to one times by one to two-

Spencer Greenberg: Right because the prior odds were one to two that they would win because out of the three possibilities, only one of them involves them winning, right?

Robert Wiblin: Mm-hmm (affirmative).

Spencer Greenberg: There’s one to two, exactly and then you multiply that by the Bayes factor, which is the new evidence.

Robert Wiblin: So then I’ve got three over two, so we’ve gone from a 30% chance of winning, or 33% chance of winning to a 60% chance of winning based on that goal?

Spencer Greenberg: Yeah.

Robert Wiblin: Yeah, okay. I’ve been aware of Bayes rule for ages, but because I’ve always been thinking in terms of probabilities, I’ve found it very hard to apply it, or I’ve been able to do it reasonably when the odds are close to 50/50 because then you don’t run up against this upper bound or lower bound of zero or 100%. But yeah, with the odds I can do it, when it’s at the extremes as well.

Spencer Greenberg: Yeah, the odds way of thinking about it, although it’s a little annoying to convert between odds and probabilities and you have to wrap your mind around that, it turns out to just be much simpler to think about. So I always like to think of it in odds but I’ll tell you the way that I use the most in life is I have some belief about a thing, I get some evidence and I want to know, “Wait a minute, is this a really … ” Let’s say it’s evidence against my belief and I want to say to myself, “Is this weak evidence or is this strong evidence, or is this kind of moderate?” I like to do that double check of, “Well how likely would I be to get this evidence if my hypothesis was true relative to if it’s not, get a kind of intuitive feeling, is it 30 to 1, is it 3 to 1, is it more like 1 to 1?” And then that sort of tells me intuitively should I downgrade my belief in the theory or not and how much I should downgrade it.

So it’s not so much I’m doing an explicit calculation but it’s really interesting to see that surprisingly often, your brain won’t necessarily automatically do the right thing. Your brain won’t necessarily automatically realize that, “Oh this is strong evidence,” or, “This is only very weak evidence.”

Robert Wiblin: Okay so what are some takeaways from this?

Spencer Greenberg: So this way of thinking, this Bayesian way of thinking is the formal mathematically correct way of thinking about how to adjust probabilities when you get evidence. So it’s interesting to say, “Well what does it tell us about how evidence works?” One thing it tells us is that evidence is always probabilistic, we start with some probabilistic belief about the world like a hypothesis with some amount of probability we assign to it and then we have to adjust that probability. And I think this lesson, while very basic is one of the most important lessons, that everyone of your beliefs could turn out to be false.

Now maybe some of them, you’re 99.99% confident, but that’s not the same as being 100% confident. In fact, one of the things that Bayesianism tells us, this idea of multiplying by the Bayes factor, is that you can never get to 100%. So if you start with like a three to one odds and you start accumulating evidence, you’re going to keep multiplying by different numbers, but it’s never going to go to infinity. You’re never going to get an odd of infinity to one. In other words … and that’s equivalent to saying, “You’re never going to get 100% belief.” So unless you started with 100% belief, somehow when you were born, if you’re a Bayesian updater, you’re never going to end up with 100% belief. So I think this is also an important lesson.

Another lesson I think we can take away from this that’s very important is that if you ignore small amounts of evidence, it can really lead you in the wrong direction. So say you’re very confident, you’re 95% confident something’s true and then you get a trickle of evidence that’s slightly against it, so it says it’s not true, and then you get another trickle of evidence, another trickle of evidence, all saying it’s not true, but none of them are that strong. Well, if you don’t update each time, then you could just say, “Oh well I’m 95% confident, so I’m almost sure this thing’s true and this evidence is very weak so I’m just going to throw it away.” Then the next evidence comes in you’re like, “Oh this is very weak,” and you throw it away. But if you throw it away enough times, actually it could turn out you should now believe the opposite, but because you kept dismissing the evidence because no single packet of evidence was so strong to change your mind, well now you end up with the wrong belief.

So one thing we learn from this is that really the way evidence should work is that it’s smooth, gradually adjusting all the time. We’re getting a little more confident we get evidence in favor, we’re getting a little bit less confident we get evidence against.

Robert Wiblin: Interesting. So you mentioned earlier that often when people present evidence against your view, you don’t automatically figure out what is the right Bayes factor. Are there any systematic errors that you think you’re making there or is it just that you don’t really intuitively … or humans don’t instinctively think in terms of what is the likelihood of seeing this evidence if it’s true versus if it’s false?

Spencer Greenberg: Yeah, one area we talked about is they might say, “Oh this evidence is likely if my hypothesis is true,” but they also don’t realize it also might be likely if the hypothesis is not true. Another thing is that something people will get fairly weak evidence and they interpret it as much stronger evidence than it really is. So they’re looking at how compatible that evidence is or something, but they’re not actually evaluating the formal strength of the evidence, which is what the Bayes factor tells you. So it can be very confusing.

Another error people make is they don’t take into account the prior probability. So they’re just evaluating the evidence as though they didn’t know anything previously. Example of this that I see my own brain do a lot is let’s say I’m traveling and I’m walking in a foreign city and I see someone that vaguely looks like a friend of mine. My brain will immediately be like, “Oh that’s so and so.” But then I do this mental correction of being like, “Okay, but the prior probability of so and so being in this random city that I’m in right now is so low, even if that looks quite a bit like the person, it’s probably not them,” unless it really looks exactly like them then okay, it probably actually is them.

Robert Wiblin: Whereas if you saw the same thing in the city that you both live, then you’d be much more likely to think it was them.

Spencer Greenberg: Exactly, much more likely.

Robert Wiblin: Because it’s much more likely to begin with.

Okay so this raises a whole bunch of issues that hopefully we’ll be able to get to over the next 40 or 50 minutes. So we’ve discussed here priors, so what’s your pre-existing belief, and then how to update. But we can discuss how often should you stick to common sense, so when should you really believe your prior, or put a lot of weight on it, versus updating based on the things that you see? There’s where does this prior come from? So what class of things should you be considering and including when you’re trying to assess, I guess the base rate of the thing that you’re evaluating actually occurring?

Then there’s how much weight to give to explicit models and theories that people present to you? So if they have some kind of micro-economic model of how something works, under what circumstances should you update a lot versus a little? Then there’s direct empirical evidence, so if you get a new study on nutrition that says that some food is especially healthy or unhealthy, how much do you update on that? Then there’s other kinds of evidence like heuristics that we use, which is a lot of rules of thumb that seem to guide people well in general, even if they’re not explicitly quantitative in form.

Then often at the end, I think, when we’re evaluating a lot of evidence together, or there’s been a particularly strong argument that’s tried to move our belief a lot in one direction, I imagine both of us do error checks so we think, “Well if this was true, what would that imply, and is it showing something else that we don’t observe that contradicts then or that suggests that we’ve made a mistake?” Maybe we can take some of these things in turn.

Spencer Greenberg: Sounds great.

Robert Wiblin: All right, so something I have a big interest in is when should we trust common sense and give a lot of weight to our priors and when should we not? Do you have much of a view on this?

Spencer Greenberg: I think it’s important to note that we all intuitively form beliefs all the time, based on just the things we see around us, and that makes a lot of sense. When we have a direct perception of the world and that gives us a ton of evidence. Where it gets a little sketchier is when there’s things that we’re not directly witnessing over and over and over again. So it’s either something that we witness rarely and we try to make some inference from it, or when it’s something we don’t witness at all and we just hear it through someone else. It’s filtered through other people. Those are the times when our intuitive beliefs start getting a little hairy and we might begin to doubt them.

When it comes to common sense, there’s certain kinds of things you would expect common sense to be effective at. Maybe people’s common sense about how to stay safe, you might expect that humans are pretty good at that sort of thing. But people’s common sense about really difficult philosophical problems, or difficult problems in computer science or something like that, if they’re not someone who’s trained specifically in that thing, I don’t know why you would expect common sense to be particularly good at those kinds of problems.

Robert Wiblin: Yeah, I think that that’s how I think about it, is looking at different fields and thinking, “Do humans get feedback on whether they’re right about these questions?” Like feedback personally and also feedback, potentially through evolution that humans who got these things wrong in the past tended to die more frequently, or at least find out about it and correct it culturally.

Spencer Greenberg: That’s a really excellent point, the feedback issue, and it’s actually really interesting to think about, when can a human learn to do something? Another way to rephrase this is when can we trust our intuition? If we’re in a situation where we do something over and over again, there’s some variability in the domain, but not too much, we get feedback on how we did, but the feedback’s not too noisy and we get a lot of repetition, we can learn to predict all sorts of things accurately.

For example, imagine you’re a psychologist, you see patients every day, one thing you get feedback on is whether the patient gets upset in your office. So you could imagine a therapist actually getting really good at preempting, “Oh this patient seems like they’re about to get really upset.” Maybe getting really good at predicting what they can do to help a patient calm down, because they have very rapid feedback on that. But they might have much less rapid feedback on whether the person’s doing well a year later, or something like that. So that’s a much less tight feedback loop, that might be harder to build their intuition and there they might have to rely more on studies or theories rather than just what their gut tells them about what’s going to make this patient better in a year.

Robert Wiblin: Yeah, okay so how about I rattle off different kinds of questions and you can tell me whether you think you should give a lot of weight to your prior or not? Okay so you’re talking to someone and you’re trying to evaluate how they feel about you, or whether they’re happy or not.

Spencer Greenberg: Yeah, I think here people actually differ a lot in their ability to make this kind of prediction. So really you have to have some self-awareness and say, “Am I the sort of person that’s good at reading social information, or maybe I’m not so good at it?” There are some people that are incredible at this, where they can really read things and people’s faces and people are like, “What? How did you know that I’m feeling that way?” Subtle emotions. Other people are just really not good at it and they can’t read even emotions that would often be obvious to others. So there, a lot of variability individually.

Robert Wiblin: Yeah, I guess it’s true that there’s a huge amount of variability here which is kind of interesting because you’d think evolution would push us pretty strongly to being good at this because people who made big social mistakes, or couldn’t read other people would have been at a big disadvantage in social situations in the ancestral environment.

Spencer Greenberg: You’d think so. It could be that some ways of being low social skill could be associated with other positive benefits, it’s a possibility, but it’s tough to say for sure.

Robert Wiblin: It’s also just a super difficult problem. Which could explain, yeah, why people make mistakes. I guess the people who are very good at this are reading tiny cues, computers can’t do this at this point even though they’re able to see things and replay them and analyze them in great detail.

Spencer Greenberg: Although increasingly, machine learning algorithms are beginning to be able to read facial emotions. So for example, I saw this machine learning system where it had people watch advertisements and then machine learning would try to actually measure their emotion throughout the ad and then see, “Oh, people are getting excited and now they’re feeling surprised,” or whatever, that kind of thing.

Robert Wiblin: Each I expect they’re going to be beating us pretty soon. All right, philosophy.

Spencer Greenberg: So I think philosophy is some of the hardest stuff out there and our intuitions are just not well honed. Often philosophers use intuition in their arguments, this is sort of an interesting debate. A lot of philosophers will acknowledge this and say, “Yes, intuition is part of our argumentation technique.” Other philosophers, I think a minority of them, deny this and say actually they don’t use intuition. Then there’s an interesting debate of is a philosopher’s intuition about philosophical problems actually well-honed?

I think that you can go back and forth on this and one reason potentially though to doubt whether you can have a well-honed intuition on these kind of things is it’s not clear we ever find out the right answer. Sometimes we find out this thing’s the wrong answer because we find it’s inconsistent, or there’s a problem, but rarely do we like, “Oh yeah, we solved that part of philosophy, now we know that we were right on that.”

That being said, some of philosophy is really about trying to figure out things like your own values or what things mean to you and there, you really are reflecting on your own internal systems and if you’re doing that, if you’re trying to … like our discussion of intrinsic values, if you consider that part of philosophy as figuring out your intrinsic values, well you have no choice but to use your intuition because your intuition is the system that tells you what you value. So you’re out of luck figuring it out another way.

Robert Wiblin: Yeah, okay, cooking?

Spencer Greenberg: As someone who’s a terrible cook, I don’t trust my intuition in cooking at all, but when you cook a lot, you certainly develop intuition, no question. You watch people who are really good chefs, they’re not reading the recipe, they’re just like, “Oh I think it needs a little more of this, a little more of that,” and they’re probably right because they’ve cooked so many recipes and tested them throughout the process of cooking them, that their gut system is very, very good.

Robert Wiblin: Okay, macroeconomics.

Spencer Greenberg: Oh, so macroeconomics is really tricky because I think a lot of it is actually very counterintuitive. Where people will kind of expect a certain thing to happen and because of weird second order effects or because of the way incentives work or just subtle things about supply and demand, it won’t work the way you expect. So I think actually a lot of times, our intuitions are just not useful in that domain.

Robert Wiblin: Yeah, I think they’re actually worse than random in that case, which is interesting.

Spencer Greenberg: They might be. There seems to be all these surprising ways where the first order effect is a certain way and so we just have a really hard time believing that that’s not the way that the final effect goes. You know what I mean?

Robert Wiblin: Yeah, oh you mean like on a small scale it’s one way and on a big scale, it’s actually almost the reverse?

Spencer Greenberg: Exactly, it’s like, “Well if you pay one person more money, clearly that’s good for them, so why don’t we force all companies to pay people more money?” Then you’re like, “Wait, but if every company was forced to pay everyone more money-”

Robert Wiblin: Wouldn’t that just raise the prices of everything and then it cancels out?

Spencer Greenberg: Yeah so you get this really counterintuitive effects when you start trying to take your local intuition about everyday life and globalize it to a whole economy.

Robert Wiblin: Yeah, I think it’s often called the fallacy of aggregation. All right, what’s a different domain? Human social psychology, the kind of thing that you’re studying.

Spencer Greenberg: So human social psychology, that’s a really interesting one because I think for a lot of us, we actually have pretty good intuition about a bunch of things about psychology. Certainly not all things, but a bunch of things. Most people could tell you, if you described a story, they could tell you will someone be sad if that happened to them, or angry and people will be pretty good at that. There’s many social things about the way people relate to each other where we’re pretty good predictors.

So that actually raises the bar pretty high because if you’re a psychologist trying to discover some new thing about psychology, you’re competing against people’s pretty well-honed, intuitive psychology detectors that they have really … Not only are they pretty well-honed, but they are getting feedback all the time. I’m like, “Oh, I mis-predicted my friend, and now my friend’s angry at me,” and that kind of thing. That being said, there certainly are some findings in psychology that people would not have predicted that you wouldn’t just expect them to be true automatically.

Robert Wiblin: Yeah. I guess it seems like pretty often those things … They’re not replicating super well, so I wonder whether in fact our intuition was better when we … A lot of these psychology results have got a lot of attention. They got a lot of attention because they were surprising, they were against our intuition. It’s like subtle things about the environment can change our behavior a lot. In as much as they are not being replicated, maybe actually we just had a good sense that actually, no those things don’t matter so much to begin with.

Spencer Greenberg: Yeah. It’s certainly true that a bunch of findings haven’t replicated and a lot of people were really surprised by. There’s one that I want to talk to you about in particular which is that of power posing.

Robert Wiblin: Okay. Yeah, yeah.

Spencer Greenberg: So as many of you may know, there was a really famous Ted Talk, I think it was one of the most viewed Ted Talks of all time about this idea of power posing that adopting certain postures can make you feel more powerful. And you know, so imagine the posture that Superman might adopt, or that kind of thing. And what happened is there was a bunch of critiques of that research that came out, and then people tried to actually replicate the study.

As far as I know, I think there were six pre registered replications where people trying to replicate it in advance said, “Here’s the method we’re going to do, here’s the process,” then they went and did it, and they tried to replicate it. And I believe, if I recall correctly that by the standard way of deciding whether P less than 0.05 statistical significance, I think four of them did not replicate even the effects of people feeling more powerful, and I think two of them did replicate the effects of people feeling more powerful.

But, in the original study, they claimed not that just that it makes you feel more powerful. They’d also claimed that it then changes your cortisol levels, it changes your risk-taking behavior and so on. And those, as far as I know, those effects really did not replicate. And so there’s this really interesting thing that’s happened. Where all these people now have come and attacked to original research saying there were flaws in the original research. “This stuff doesn’t replicate. Power posing is fake, stop doing it before you go on stage or before a meeting.” But the irony to me, is I actually think that power posing works.

Robert Wiblin: Oh wow, okay.

Spencer Greenberg: And I’ll tell you why I think that.

Robert Wiblin: Yeah, hit me.

Spencer Greenberg: First of all, I happen to be a person that I think is very affected by my body posture. And so when I change postures, I actually can notice, like a fairly palpable effect. If there’s a large change in posture. So, it was very strange to me when the power posing stuff didn’t replicate that I’m like, “Wait, but I can just literally do an experiment on myself where I change my posture, I change to another, I feel an effect. I change to another.” So as someone who directly perceives that effect, I find it very strange.

That being said, attempting to be a good skeptic, I don’t necessarily, maybe I’m diluting myself, maybe I’m confused. So I went and ran a study that is the size of all the preregistered trials put together, n=1000. I preregistered it as well. And I tried to see whether people have a mood effect. Does power posing increase your mood? And so we’re still working on analyzing all the results, but the top line result is that, yes, we found a mood effect. Doing power poses seems to increase people’s mood. Seems to also increase their feelings of power.

There’s a data scientist who I’m friends with, who actually said he would go re-analyze the results, see if he agrees with us. So he’s checking them, see if he thinks we did the analysis properly. But the combination of this data, plus my first hand experience just alternating between different poses, really suggests to me that actually power posing might work. And maybe the critiques were accurate in the sense that they were accurately finding flaws in the original research, but maybe they actually misled people into thinking that this method doesn’t actually work.

Robert Wiblin: So why do you think those replications mostly didn’t find these effects that you’re finding? Are you measuring something different?

Spencer Greenberg: Here’s a really interesting thing. So, as far as I know, there were six preregistered replications. Two of them found an effect at P less than 0.05, four of them didn’t. Now, is that the pattern you’d expect if power posing didn’t work?

Robert Wiblin: Maybe it has like small effects? Is that the answer?

Spencer Greenberg: That’s what I think is going on. I think that what’s happening is power posing has small effects. It’s subtle. It’s not like profound change your life. It’s a subtle effect. And so I think what’s happening is that this pattern of replicate, didn’t replicate, replicate, didn’t replicate. To me that suggests a sort of relatively small effect. Those studies weren’t big enough to reliably detect it.

And actually, someone went out and did a Bayesian meta analysis, trying to combine all of the evidence from the six studies, and they concluded that it does actually have the effect. So I’m not the only one that thinks that. Now, here’s another thing about this. I suspect that people vary a lot on this dimension of how much body posture affects their mood. And so basically, what I suspect is that for some people, it actually has no effect on them, other people it’s kind of like a really small effect, and then some people it’s actually quite a large effect. And I think I’m in the kind of large, tend to be a larger effect group.

And so maybe that’s probably also why this is confusing because some people are utterly convinced by their own experience that this is totally useless, and other people are like, “What are you talking about? I can do this, and I actually feel a mood boost that seems significant to me.”

Robert Wiblin: Yeah, I was going to say my prior on this being true was pretty low. And then I think even when that study came out, because of all of the publication bias, it wouldn’t have updated me very much in favor of it. Maybe I think there’s a 5% chance that this is true. No, that’s a bit unfair. Start with a 10% chance, then the study comes out, then I inch up to 15% or something like that.

Spencer Greenberg: So what’s your update on the six pre-registrations, two of which seemed to find a statistically significant fact, and then my N equals 1000 study that found an effect? Apply Bayesian updating.

Robert Wiblin: Yeah. I think I would’ve moved upwards on it having some effect, like a non-zero effect. But maybe it’s shrunk on it having a very large effect. We’re kind of narrowing it down to something that’s a bit above zero but not a lot.

Spencer Greenberg: Yeah. And I think that’s a very reasonable way to look at it. Now, if I’m taking the devil’s advocate perspective you could say, “But maybe it’s a placebo effect.” And there it’s actually really interesting. It’s almost philosophical. What exactly do we mean by the placebo effect?

Robert Wiblin: The whole thing we were going for kind of is a placebo effect, right?

Spencer Greenberg: Yeah, in a certain sense. If we care about making people feel more powerful, the placebo is actually one mechanism by which it could do that. It wouldn’t imply that it’s not useful, it would just imply that the mechanism is believing that you’re gonna feel more powerful makes you feel more powerful. What would be really bad is if it was actually a reporting bias, in fact. In other words, people actually don’t feel more powerful but for some reason they report feeling more powerful when they’re in that posture. Because then you wouldn’t actually be producing the effect at all.

And, if I’m playing devil’s advocate against our own research, when we looked at people … At the end of the study we asked people, “Do you believe that body posture can affect mood?” And for people that said No, they got a much weaker effect than the people who said that Yes. But-

Robert Wiblin: But maybe they just know themselves.

Spencer Greenberg: Exactly. Maybe this really is a trait that there’s a high degree of variability and people whose body posture does affect mood have at some point in their life realized that. So, they’re like, “Yeah, it affects mood,” and then it also does actually affect their mood. It’s interesting to get in the weeds of this for continued analysis. I’m looking forward to getting this research out there.

Reference class forecasting

Robert Wiblin: Okay. A way that you can try to be more robust in coming up with your prior, or I guess in a sense of dating a prior other than just applying common sense is reference class forecasting. Looking at similar cases and seeing how often something is true, on average. Do you want to descr