0:33 Intro. [Recording date: July 1, 2019.] Russ Roberts: My guest is... Andy Matuschak.... Our topic for today is a fascinating essay of his called "Why books don't work."... What's wrong with books? I like them. This essay of yours is a little bit of an anti-book screed, and I'm quite, actually, sympathetic to it. But at first glance, the idea that books don't work is deeply disturbing. So, make the case. Andy Matuschak: Yeah. I think perhaps I need to roll back on the title a little bit. Perhaps a little bit of clickbait there. So let me begin by saying why I love books. Now, first off, and books are [?] the author of our cultural history, and certainly oral history has a great role to play there as well. But certainly insofar as we've been able to build on previous generations' knowledge reliably, and construct cultures that we're proud of, and indeed contribute to future cultures, much of that is occurring through books. Great! Separately, there's also the wonders of fiction: we're also able to live in other people's worlds, experience what could be. Absolutely fantastic. So, perhaps a better title for this essay would have been something like "Why Books Don't Reliably Contain Detailed Information." Russ Roberts: And, in particular, nonfiction. Although, we learn much from, I'd say, from fiction. But the type of learning we expect to do from nonfiction is different. Andy Matuschak: Sure. Now, sometimes we read nonfiction for entertainment purposes. But, sometimes we read nonfiction because we genuinely want to learn about the topic, and we really want to understand. Russ Roberts: And, we want to get smarter. And, I think one of the challenges of your essay, which is a challenge of this whole area, is: What do we really mean by 'learn'? Or, what do we really mean by 'remember'? Or, what do we mean by 'grasp'? But, at the heart of the challenge that you identify with books is something, a framework I had not heard before called transmissionism. Explain what that is. Andy Matuschak: Sure. It's a bit of a straw man really in that no one actually advocates for it, but essentially transmissionism is the false idea that you can come to know a thing by having that knowledge directly transmitted into you, as if you are a slate that I as the teacher can write upon. And so, in some sense, books are kind of imagining, or an author is kind of imagining that they can write an explanation of a thing, and that a reader can read that explanation, and then they'll come to know it. Now, sophisticated readers don't stop there: they know they have to do more. But it's hard to do it, and it actually the medium itself doesn't do much to convey what else must be done in order to learn. Russ Roberts: And you make a parallel which I find quite interesting about lectures, because I've given quite a few lectures in my life; and I've attended, mercifully, a smaller number, I think. And I'm struck by many of them as being of no obvious value whatsoever. It doesn't mean that there isn't value because there are subtle things that can be gained from listening to someone speak. But, as you point out, often a very short period of time after that lecture has passed, or if you've read the book, you don't remember a single thing. It's not that you didn't get everything the author or lecturer intended. You get, literally, nothing, at least that comes to your conscious mind. Andy Matuschak: Yeah. Especially months later. Lectures are maybe an easier thing to attack. I think we kind of have an intuition that, 'Oh, you attend a lecture,' and you might say something like, 'Oh, that was interesting.' Or 'that was helpful.' And then: Did it actually help you understand the material? Well, maybe, if after the lecture you did a whole bunch of stuff: You did some problems that suited some projects. But, if you just listen to the lecture, you probably don't walk away with all that much, except perhaps excitement, emotion, things like that. And that's because lectures, like books--as I was describing before--are sort of founded on this transmissionism notion: the notion is that 'I as the teacher can get up there and say a bunch of words; then you'll know stuff.' Russ Roberts: But it seems to me, to be fair to lectures and to give them their due, and I think also somewhat to books--and we won't spend too long on this part, saying nice things about them. But it seems to me that much of what we expect to get from them is exactly what we get. We're not really expecting to learn a massive amount. We're not expecting to learn how to master a subject. We want to be exposed to the working of a mind, someone's mind that appeals to us or that we think will inspire us. You know, I think about a weekly sermon as an example where, you're not really expected to learn anything. It's supposed to touch your heart, perhaps, change your behavior--it rarely does, I suspect. But perhaps the accumulation of that contact and that mode has an impact on you. And similarly hearing a lecture of a great scientist or a great thinker maybe inspires you to take your life more seriously, say--not to be fully aware of how the genetic process works or quantum computing, or anything like that. Andy Matuschak: Absolutely. I agree completely. And perhaps to state a mechanism even more specifically, I think that one idea that these things can convey that runs through those examples is that they can convey cultural norms, cultural values, or perhaps even the single individual's values really quite effectively.

6:10 Russ Roberts: Yeah. And that's a powerful thing. But, where I agree with you is that much of it is, I would call it, a pleasant way to pass time. We call that entertainment sometimes: I think that's somewhat demeaning, I think, to a great lecturer or to be exposed to a great thinker, but it's, as you, I think--you didn't use these exact words, but it was like, 'Yeah; I enjoyed that.' I think you said, 'It was interesting,' I think something we often say after a lecture. And it could be--that things were stimulated in our brain that we didn't, you know, fully appreciate or think about. But, as a way to convey--I would have us try to focus on two things: Information, one; and then wisdom. Obviously wisdom is much harder to convey, and the richest type of learning that I feel I do is--'wisdom' is a grandiose word for it, but, seeing connections between things I didn't see before; understanding that something I thought, I didn't realize applied to something else; seeing an analogy that might help me understand how to think about something--that's what I call, you know, wisdom, deep learning, whatever phrase you want to call it, as opposed to information. You know: The capital of California here is Sacramento. Or: The Boston Red Sox won the World Series last year. Those things, which are all accessible via Google or DuckDuckGo in a very, very short period of time--I view those as different. And I don't go to a lecture to get those facts. I go to a lecture to get something closer to wisdom. Andy Matuschak: Indeed. And it would be better, perhaps, if lecturers actually operated according to those principles. Now, one thing that we don't know--we, kind of referring to scientists as a field--is the degree to which that kind of declarative knowledge, the information as you call it, has to be deeply internalized before a person is susceptible to the wisdom. Uh, mmnnn, lots of differing views on that: it's kind of difficult to study empirically. But it does seem that to some degree, you need to be familiar with the core components of whatever topic of wisdom is attempted and to be conveyed. Russ Roberts: Yeah; you know, my wife's a math teacher, as listeners know. And I've been influenced by the psychologist Daniel Willingham. And he's a big fan--as am I--of, it's very hard to get wisdom without information. You need a base level of information: you need, for math, you need your multiplication tables. You need some historical facts. You can't just theorize about history without understanding some really tedious factual things. At the same time, factual things alone are not knowledge. They are just facts. And I think the challenge that we face as our time gets more valuable--and I know you are aware of this, so I'd like your opinion: The temptation to reduce everything to a multiple choice exam tends to force us toward spit-back information and move us away from wisdom, is, I think, part of the challenge. It's certainly the elementary-to-high-school educational problem. Andy Matuschak: It's a huge problem. Wittgenstein has this wonderful quote, 'a wheel that turns, although nothing turns along with it, is not part of the mechanism.' [Philosophical Investigations, Ludwig Wittgenstein--Econlib Ed.] And so, if you've simply memorized a whole bunch of piece of atomic information and yet they don't connect to each other, you are not able to combine them into anything. You are not going to get that far. Learning science gives us a couple of structures for thinking about this that I find helpful. One of them is Bloom's taxonomy for thinking about tasks. And, it begins with being able to recall things, reasonably; and it moves up through being able to, say, apply things, take some procedural knowledge, perform a procedure. Perhaps initially a very rote procedure and then perhaps one with some flexibility later. Finally it moves through being able to evaluate and make critical arguments. And ultimately to synthesize or create something anew. A different set of structures I'll bring up, and then hopefully that will be helpful for the conversation, is transfer. So, initially when we learn things we tend to apply them in a context where we're told exactly what to do: 'You need to perform this procedure on this data.' And then maybe we remove one of the phrases from the sentence: 'Perform some good procedure to achieve some good outcome on this data.' And then, the challenge becomes just knowing what kind of problem is it, and finally, what we call 'far transfer,' is a problem where you are--you the student, or you the learner--are forced to combine skills, things you are learning, in a completely new-to-you environment, that definitionally cannot be practiced via drills.

10:48 Russ Roberts: Yeah; I've probably mentioned this before: My favorite course evaluation I ever received was, I got a 1 out of 5 from a student who said, 'Professor Roberts is a terrible professor. He expects us to apply the material to things we've never seen before.' Of course, I warn people on Day 1, that's the essence of the class. And I think that is great teaching, when you can pull it off. It's not easily done. Not every student can do it. And I'm ashamed at how the techniques that I used when I was in the classroom to convey that process--my standard technique was: 'Here are some really hard problems. Think about them.' And I was not able to give the feedback that they needed often to help them see why their incorrect answers were incorrect. And that's extremely important, obviously is, to make progress. And we'll talk a little bit about that general challenge of feedback, because I think it's a fascinating one. And it's increasingly challenging as people's time gets more valuable. But, I want you to just talk a little bit about that open-ended versus multiple choice level of understanding. At one point in your essay you talk about textbooks. You'd think textbooks would be written and designed to be--to acquire knowledge, wisdom, etc. People write me sometimes, 'What econ textbooks should I read?' And the answer is, 'None of them.' They are not designed to be read. They are designed to complement an often extremely dry, multiple-choice-driven class. And they are explicitly not designed to convey information. And I think that's--it bothers me. Andy Matuschak: Nyeah. I'm happy to at least try to speak to that. Unfortunately, we don't have all the answers there. But, what we do understand of how what you call wisdom is acquired, or maybe the ability to, mmm, apply far transfer or to flexibly and fluidly take skills and use them in different contexts, that comes about through a variety of different mechanisms. For instance, explaining things that one has not seen before--using those concepts. Or, making critical arguments using those concepts. When I was a Khan Academy, my R&D group worked on this lengthy project to try to, instead of having multiple choice questions for everything, actually have open-ended activities for things. We were very focused on AP [Advanced Placement] Humanities. For instance, in history if you want to learn to make an argument for what historical causation is, that's a pretty complex thing. Historical causation includes things like, 'It couldn't have been otherwise.' It includes a sense of 'How important was this cause,' versus a different cause. And, so, if I were to ask you, 'What was the most important cause of World War I?' that's something that, mmm, if you were to give that as a multiple choice question, it wouldn't teach you very much in terms of wisdom for historical causation. Russ Roberts: I would say, 'None.' A typical question like that--not on the AP exam, although I will say, as I've ventured[?] before, that I wouldn't have done very well on the AP History exam, even in areas that I understood the history I think fairly well. And I certainly wouldn't have done well on the AP Economics exam, recently, as my kids struggle with them. And I said, 'These are terrible answers.' But on the history exam, even if I said, 'The assassination of Franz Ferdinand by Gavrilo Princip is the cause of World War I,' and then the next three were related to, let's say, Walt Disney movies--well, clearly a). is the right answer. But you've learned nothing. Now, there are subtler answers that you could give. But they wouldn't be right, either, even though they might apply to World War I. And the challenge is, as you point out, isn't teaching people what the right answer is. It's teaching them how to think and weigh evidence related to answers that are inherently not well-defined as correct or false. Andy Matuschak: That's right. And indeed that was actually the approach with this project. Before, you mentioned that most textbooks are defined not to be read on their own, but in fact to be used alongside a class. And, if that class is a multiple-choice oriented, very rote class, that's an unfortunate thing; and probably you are not going to walk away with a lot of wisdom. But, if that class is a seminar run by a competent facilitator, then you might get something pretty wonderful, where, the contents of that class, day to day, are discussions. And debates. Some one person raises an argument for what indeed was the cause of World War I; another person refines that and says, 'It's interesting that you point out this cause, because I think actually if that cause had been removed, nothing would have changed, for this and that reason.' And, together the class thing co-creates an understanding of what truly is important for causation. There's modeling happening. The teacher is interjecting, perhaps with a really good example of an argument at a given moment. A teacher might ask a probing question that is meant to highlight a way in which a student's conception is unproductive. Or a way in which it might be enhanced. And, indeed, the students will surprise each other. And so, it kind of got me--we kind of tried to create that kind of thing online.

15:59 Russ Roberts: And you wrote a paper about, which I read, and it's a fantastically provocatively idea-- Andy Matuschak: Thank you-- Russ Roberts: The reason I found it fascinating is a lot of it is built around the idea of peer-to-peer reaction. So, describe what that, how that model works, at least in this case, and whether you think it has, you know, potential to be more widely adopted. Andy Matuschak: Certainly. So, first I want to kind of describe what it's not. So, other people have tried to do open-ended learning activities online before. And often what that looks like is a peer-to-peer interaction. But, it's grading. So, it's back to the right-versus-wrong. It's: I wrote an essay, you grade my essay, and I grade your essay; we give each other scores. Maybe there's a rubric. The rubrics can help; it's not nothing. But we are really focused not on assessing understanding, but developing understanding. Those two things are different. And, we think that developing understanding looks a little bit more like that classroom, where there's an exchange of ideas and exchange of values. Now, we don't want, kind of the blind leading the blind type thing happening. So, we don't--actually many teachers do--just create a discussion forum--and say, 'Students please discuss between classes.' And sometimes that works. The evidence suggests that mostly it doesn't. So, we want some kind of facilitation, some kind of orchestration. When the teacher is there and they are skilled, they can supply it. But, if they are not there, or, indeed, if you want a conversation happening between every pair of students, then they can't really supply it. So, the solution that we build kind of offers structured scaffolds where students are presented with each others' arguments and then they are given almost like the little hand of cards, for, like, things to try. 'Oooh. I'm going to not disagree with you or point out a limitation of your argument, but actually one of the things I'm allowed to, card I'm allowed to play on your argument, is: I'm going to add a piece of evidence.' Or, 'I'm going to suggest a different implication of the argument that you are making that actually makes it even more powerful.' And, students are not only doing that with each others' work, they are also doing it with model work, that either the curriculum authors or the teacher supplied, so that they can interact as well, possibly without knowing, with examples which are structured to highlight either, mmm, particularly good manifestations of some facet of the skill, or faulty ones. Russ Roberts: Now, is anything happening with that project? Andy Matuschak: Ah, right. So, we finished that project mid last year. And, it's kind of on the queue. But, Kahn Academy is a pretty small nonprofit. At the moment my understanding is that they are focusing mostly on middle-school math, where this particular set of skills is not quite the priority. Which is interesting. Because, actually, in the new Common Core set of standards, there is this wonderful list of mathematical practices, ways of thinking like a mathematician that students are supposed to acquire, that are usually de-emphasized relative to the 'you should be able to multiply two digit numbers together.' And, my favorite one--of 8--is to be able to construct viable mathematical arguments and critique the reasoning of others. And, I wish luck to those constructing platforms using only multiple choice for that. Russ Roberts: Yeah; that's kind of tough.

19:10 Russ Roberts: So, one of the most interesting pedagogical insights I was ever exposed to was, I had the opportunity to spend an hour with Orson Scott Card, science fiction writer. And, he teaches creative writing. And at the time I was teaching economists how to convey economic ideas in writing. Which was, of course, I was teaching at George Mason, I think it was the only course in the country [?] like I'm pretty confident. And, I asked him for his advice. I was just starting to teach that class. And I didn't know quite how to do it. Or at least I thought I didn't know how to do it, but I thought he might know better. And he did. His suggestion was--and I didn't implement it as well as I might have, but I did the best I could--his suggestion was to have the students give each other feedback. Not literally grade, but critique the writing of the other students. And, the reason that's so deep--one thing is it saves the teacher work, which, of course, is always appealing at a certain level. But, I think it's better than that. It's a win-win, because, I think being a great writer requires being a great editor. And, it's very hard to learn how to edit one's own work. And, by editing the work of others, you learn how to edit your own work. And that, I think, is an incredibly powerful thing. Because, what you get instead is a teacher who writes in the margin in red pencil, 'Awkward,' or 'Confusing,' or 'Bad example.' And the student gets it back, and they never look--first of all, they rarely look at it other than to say, 'Boy, there's a lot of red marks here. I thought this was a better essay than my teacher thought.' But I think the idea of learning how to fix an awkward sentence--and of course, in an even better class would be, would be, just write 'Awkward,' you write your better version of the sentence that would improve it. Because then it teaches you how to fix your own sentences. And I just think that is--that's an incredible insight. And I just want to take it one step further. Which is, to pretend that EconTalk is an educational activity. And one of the things I like to think of is that your dialog with mine is echoing a dialog that a listener would have with you. Or with me. And, that process is one of the ways we learn--is through the challenge and the back-and-forth. And I think how little of that is in our current education system, how little of it is in our books, lectures. It's all--it's transmissionism is all the way down. Now, for better or for worse, in high school classes, transmissionism is, I think, getting reduced because kids can't even sit still to pretend they are being transmitted information to. And so, it is much more interactive. It seems to me that interaction and grappling with the material is the only way we really learn. Andy Matuschak: You'll get no disagreement from me there. Indeed, if you want to learn how to learn how to edit prose effectively, editing others' prose and your own prose, editing the best prose in the world, it's difficult to imagine better ways to do that. And, to the great credit of so many [?] of teachers, particularly recently, there's been a huge shift to interactive classrooms. Yes, because attention spans are shrinking and it's just more interesting and fun; but also because there's a growing recognition that even when one is learning math, this kind of interaction is essential. People are not going to really understand what, say, a negative number is simply by learning how to perform a set of procedures related to it. They need to get a feel for it, the sort of term of art is sense-making. And that involves experimentation and play. Russ Roberts: You know, or worse--they are not going to learn what a negative number is by hearing a definition. And I think this idea of grappling with problem-solving--and by problem-solving, it isn't just getting the right answer. In math it often is, and I think that's extremely important. But in economics it might be just to understand the factors that might be the ones you want to consider. And there's no "right answer" as to why something is happening in the world right now. That, to me, is the economic way of thinking. You know, I read today that fair-trade coffee isn't helping the workers; it's helping the farmers. And I'm thinking, 'No kidding! That's what my problem set would have challenged the students to think about.' Right? If your skill level is only worth x in the marketplace, the fact that your boss has more money doesn't necessarily mean they are going to share it with you. They pay what the market can bear. So, those kind of insights--which, you know, as you point out, could be applied to a much wider array of things other than, say fair-trade coffee--that's the challenge of getting people to think that way. And, telling them that answer, which I just did, for you listening out there, I don't think is nearly as valuable as arguing about it for half an hour with somebody about whether that's a good idea or not. Andy Matuschak: Absolutely. Maybe a way to summarize the whole approach is: Effortful engagement with ideas--that's going to be criticism, okay. If you are able to criticize effectively. Now, you may not understand the skill and question well enough to do that effectively. And that's fine. There's different kinds of effortful engagement. For instance, listening to you and another economist having a debate on that topic may be a really great way to slightfully effort-fully engage how you can't stop there. But it may be one great step on a scaffold to being able to enter that argument yourself and to debate with Russ.

24:39 Russ Roberts: And of course, a lot of you out there listen to EconTalk more than wants to each episode. Which is fascinating. Let's talk about that--not about EconTalk per se, but I want to go back to books. I read a book of Jewish theology called God, Man, and History. I found it utterly fascinating. After I put it down, I realized I couldn't tell you a single thing, 6 weeks later. And I went and asked--it's written by both[?], Eliezer Berkovits, in, I want to say in the early 1950s, 1953 or so. And I went around to people I knew who were interested in theology; and I said, 'Have you read Berkovits?' And they'd say, 'Oh, sure.' And I'd say, 'What can you tell me about him?' And the answer was: nothing. Every time. Now, part of this was due to the nature of theology. It's hard to remember theology. You don't deal with it every day. Part of it is Berkovits's style. Some of it, a lot of it is the human brain. Right? Which is prone to forgetting. But I decided--I was not happy about this. So I decided to re-read the book, with one of my children. I'd actually read it the first time with one of my children. But I decided to re-read it with a second child--following along, underlining in the margins, summarizing. And I've now read it, I think four times. It's one of the few books I've read more than once. And I have a pretty good idea of what he's talking about. And I could probably tell you right now. But, it was a lot of work. It was a huge amount of work. I mean, it's a challenging topic, obviously. It's not--actually, that's not really true. Every nonfiction, serious nonfiction book, is usually grappling with a complex set of ideas. So, what are your thoughts on that? On multiple readings? And, you talk about repetition? So, talk about what you are advocating there in online learning. Andy Matuschak: Sure. So, one lovely thing is that I think that scientists actually kind of know a bunch of what's necessary to form memories reliably. They're just not necessarily what we'll naturally do, day to day. It's a little hard to pull off. Memories are formed more reliably, say, if you are exposed to things multiple times; if they anchored to powerful emotional experiences; if you have a variety of encodings for that memory--for instance, you know, both visual and auditory and perhaps relating to places. But, um, you know, the repetitions thing is a particularly easy and mechanistic approach. So, you know, we're probably all familiar with the notion that if you study a thing a few times, maybe it sticks somewhat better. There is sort of an efficient way to do that. If you study things a few times very, very far apart from each other, it's maybe unlikely to stick so effectively. It also perhaps unintuitively for many, college students especially, turns out not to be the case that--mmm, let me phrase that in the positive--if you study things back to back, say, within a couple of hours of each other, that's also not really going to work all that way. This is called the spacing effect. It turns out to be very helpful to space your study out. It's also possible to build systems which take advantage of that spacing effect. And this has been known for a long time, all the way back to shoeboxes[?] for language learning, a system by Leitner. But, modern systems are also being used. It's a little niche right now. Often by, for instance, med students trying to prepare for the MCAT [Medical College Admission Test] and still in particular by language learners. But, my colleague Michael Nielson and I have been working on trying to integrate these systems more deeply into kind of any informational nonfiction texts you could possibly imagine.

28:12 Russ Roberts: And let's talk about that a little bit more. I'm--I've probably revealed on here that, you know, I don't--the way I think about it is you read about 50 books a year if you read a lot. And, it's a book a week. And you might read for 50 years, as an adult. Maybe a little more. But, it's about 3000 books if you are lucky. And so, you should pick them wisely. And if you are not productive, you should put them down. The idea of reading a book twice--or three or four times--now, I did it with one of my children, which made it special; and that's a different--it adds something like--it's like the thrill of the lecture in person. But I do think it's an interesting question about how often we should return to material, and how we should--and how that process should take place. Andy Matuschak: So, one of the things that's sort of shocking about the spacing effect and the way that it's instantiated, is that normally, when we do things--say, professionally as knowledge workers or as thinkers--we get diminishing returns. So, if we're not taking any notes at all, and we start taking notes: Okay, some good stuff's going to happen. And then if we take more notes, somewhat more diligent, we'll do a little better. But we're going to get diminishing returns. It's going to be some kind of a logarithmic curve; and then the same is probably true for, say, you know, having a good discussion about that thing with others: if you have twice as many discussions you are probably not going to have twice as many returns, indefinitely. But the weird thing of the spacing effect is it's the opposite of that. It's actually exponential. So, both prior theory and actually our present data with a current prototype, with almost 600,000 data points, is showing that, uh, marginal time actually leads to exponential returns. So, in order to explain that I have to get a little more concrete about what this project is. Russ Roberts: Reminding listeners that 'exponential' doesn't just mean, like, 'enormous amount.' It means more than linear. So, go ahead. Andy Matuschak: Sure. Yeah. Okay. So, we made this book on quantum computation. And we've chosen that topic in part because that's a topic that has a lot of notation, a lot of details. You need to know that, you know, okay, the bottom left-hand corner of this particular transformation matrix has this value, in order to really make sense of the topic. So, the first chapter takes about 4 hours to read. And, it contains 112 details that we track your memory of. And, we kind of follow up; and it's an interactive reading experience where as you are reading, you get these little prompts--like, 'Are you remembering this?' If you are not, we are going to kind of come back to you a little sooner. And then we come back the next day and we kind of jog your memory on the stuff you are having trouble with. And so on. You kind of start this practice. What's interesting about it is that the practice actually spaces out over time. So, initially the every day; then every 3 days; then every week, and every month, and so on. Russ Roberts: When you say every day, you mean the prompts to check your knowledge, right? The feedback. Andy Matuschak: That's right. And that isn't a cost which is equivalent to re-reading the book. In fact, it's very quick. It's 5 or 6 seconds per question, so there's about 112 of them. And so, it costs, ehh, 10-15 minutes for readers to do each of those repetitions. And actually, just last week I did the last first, mmm, large analysis on the efficacy of this thing. And by the time people get to the fifth repetition, which costs, mmmnn, 60, 70, maybe 80 minutes, kind of depends on where you are--people are able to leave the book alone for a month, have a month pass where they are not engaging with it. They come back. And, after a month engage with those 112 questions and get them right. And, that's really exciting, really interesting. But maybe what's even more exciting about it me is that the marginal costs from the 4th to the 5th repetition is actually the same as the marginal cost from the 3rd to the 4th [third to the fourth). As the marginal cost from the second to the third. Actually, it's slightly less. So, not only does it take slightly less time to get from a week to two weeks, and two weeks to a month; sorry, not only does it take the same amount of time, to get from a week to two weeks and two weeks to a month, it actually takes slightly less time for each of those steps. Russ Roberts: Now, I looked at that material. And--because I was intrigued by the pedagogical aspects of it. I have no interest in quantum computing, except as a thoughtful member of the 21st century revolution of knowledge. And the questions at the beginning, at least, were things like: What's the space called that a cubic vector is in? And, it's a state space. I forgot that term. Okay; but I got the--the first thing I got right, the answer was 2; and then I got the second one. These are spit-back questions, which are not unimportant. When you say 112 things--that's jargon, I assume. It's--may be some connections of things. But it's not the deeper understanding. Am I right? Andy Matuschak: You're right. Now, some of the questions were--some of them were involved. They would ask things like: 'Why is it difficult to store a qubit in this particular kind of matter?' for instance. And so you kind of have to come up with a justification. Now, after you practice that a few times, it'll become a spit-back thing. Russ Roberts: Now, a qubit [also spelled qbit--Econlib Ed.] --I just want to show off and say a qubit is like a bit in classical computing; but this is in quantum computing. Correct? Andy Matuschak: That's great. Yes. Russ Roberts: I learned something already. Andy Matuschak: Wonderful. So, I guess, first off, some of those 112 details are somewhat less spitback. But I think the really important thing is what those 112 details let you think next. My colleague has this metaphor that I really enjoy, so I'll share it here. Reading a challenging technical textbook is often a little bit like beginning by reading a book in English, and then--let's assume you don't know Spanish--Spanish words start creeping in. And by the time you finish the first chapter, like everything is in Spanish. And you turn to the second chapter, and you're like, 'Whoa. Like, I thought I'd picked up a book off the English section. What happened?' And so, you're going to struggle with that second chapter. If you have those 112 details, which[?] we have a second chapter, you are going to have a lot easier time learning about the quantum search algorithm.

34:18 Russ Roberts: Yeah. But that's--Let me take you to a harder example. I'm going to give you two. Just for fun. Because they are both former EconTalk guests. So, I interviewed Yuval Harari a while back about his first book, which is Sapiens. My wife is reading it now. And, let's pretend you ask me: What do you remember about Sapiens? Which my wife taps into a little bit because we are talking about it. And it turns out we are remembering three things. Three things! Not good. It's a long book. I remember that he thinks that agriculture was a mistake and I didn't agree with him; it didn't sit well with me. I remember that his view of money is based on trust and I thought, took that idea a little too far even though there is a sense on which money is based on trust. As an economist I found that a little simplistic. And third, he's anti-religion. Those are the--that's the three things I remember. That's weird! That's depressing. Now, let me take a different book: Fooled by Randomness, by Nassim Taleb. Now, when that book came out--I read it--didn't write when it came out. I read it somewhat after, The Black Swan, his second book came out. And there were two views of that book, which I've mentioned here before. One view was: 'There is nothing original in this book. This guy is a fraud. He pretends he has figured out all this stuff.' And I said, 'You know, I agree with that. ' I didn't learn anything that I didn't know in this book before. I knew that probability is difficult. I knew that risk is a hard thing to wrap your mind around. I understand something about most of the ideas he talked about in the book. So, in some sense, I learned nothing. On the other hand, I learned something incredibly deep. That book really grabbed me by the guts and jerked me around. It forced me to confront some things that I "knew" but didn't really internalize. And I would put that as another category of learning. These are things that, um, I'm just going to give you another example, another EconTalk guest, A.J. Jacobs, who writes a book called Thanks, A Thousand. It's about being grateful. Being grateful is a really good idea. I already knew that before I read the book. But the book made me a little more grateful, maybe. But, even if I remember that that's idea of the book, and even if after reading, I thought, 'Yeah. I should be more grateful,' to get me to be more grateful--that's a very high level. And so, those are the sort of 3--you know, those are all nonfiction books. They are all kind of trying to convey some understanding that the author has of the world around us. And I have really, really, different grasps of all of them. Andy Matuschak: That's wonderful. And, to some extent, it illustrates the variety of purposes for which books are intended. If we look at, classical rhetoric, only a small part of that information piece. But, if, for instance, in the second book where you were yanked around--I'm not actually sure if it was maybe ethos or pathos. It kind of could have been either depending on your predilections around Taleb. But that's something that's not going to come just from a flashcard. Russ Roberts: Exactly. Andy Matuschak: And indeed, that's part of why, for quantum computing, we didn't just make a box of flashcards. We are experimenting with this thing we call the mnemonic medium, which is different from a bunch of flashcards; and it's also different from a book insofar as it's structured information embedded in a narrative. Because narrative can have ethos. Narrative can have pathos. And it can do those jobs that those wonderful books you described did. And, perhaps, also, do the job of leaving you with some detailed knowledge of what you read. Russ Roberts: Yeah. I'm a big fan of narrative, obviously. You know, I've written three economics novels. And I do think that narrative--story-telling--is an important way that human beings' brains absorb information. Do you think about that, much? Andy Matuschak: Absolutely. So, at a very simple level, I mentioned earlier that memory comes from emotional salience. But, jeez--I mean, like, let's leave that behind for a minute. Let's talk about meaning. Why are we reading any of this stuff, anyway? What's the point of learning all this stuff, anyway? For me, it's--it's about the moments that spark joy. The moments that fill the delight behind my eyes, where I feel connected to the eternal. I don't know--standing here in front of the Hoover Tower, I felt a moment about that; and I asked you about peace. That's like a heady question for a Monday afternoon. I felt connected to peace. And, I don't want to learn about quantum computing to learn about quantum computing. I want to learn about quantum computing because it could, you know, bring us to the cosmos or something.

38:58 Russ Roberts: But there is something beautiful about it all. So, I do think--I remember as a young man--I was probably 22--and I was reading a book by Robert Ardrey, who I really liked. I'd be amazed if anybody out there under the age of 50 has heard of him. And I'd be interested if anyone over the age of 50 has actually read one of his books. I read almost everything he wrote. He was a playwright, and he dabbled in anthropological creativity. His most famous book was African Genesis, which's theme was, as human beings we came from the savannah and we hurt each other. We were violent animals, not just this sort of peace-loving image that I think a lot of people have about human origins. But he wrote a bunch of books: African Genesis; another book called The Social Contract; and I can't remember the other one. But in The Social Contract, I remember reading that book and putting it down and thinking--it made my hair stand on end. So exhilarated to be in the presence of that experience, and to have captured something of what he was trying to convey. And it made me want to be a writer, actually, more than almost anything else. I thought, 'Boy, if I could make somebody feel this, that would be deeply gratifying.' I don't know if I ever have. But that was such a powerful moment of human connection with this man's brain. It wasn't like I walked out thinking, 'Oh: I know now why people interact the way they do.' It was, 'Wow. It's hard to figure stuff out, and that's a really interesting try.' Andy Matuschak: What a powerful thing. And, ideally, we're all learning things for this reason. We're not learning things--ideally, I hope--in order to check off a box for the Common Core State Standards. We are learning things because we feel connected to the [? neumonis? gnomonis?]. Russ Roberts: Yeah. For sure.

40:52 Russ Roberts: So, let's talk about how to make things better. You know, one of the things I learned from a really interesting part of my life when I was interested in time management was: When you learn something, you should tell it over to someone. Okay. So, that's a version of your idea of--what's the name of it again? Andy Matuschak: Transmissionism. Russ Roberts: No, no, the stretched out-- Andy Matuschak: Oh, oh, oh, oh, right, right--the spaced repetition curve. Russ Roberts: Right. It's one example of spaced repetition. And, when I talk about--I don't actually probably do this, but I talk about it with my kids, when I try to give them advice on how to give a presentation, I say, 'Tell them what you're going to tell them; then tell them; and then tell them what you told them.' Now, I don't really lecture that way, partly because not all my lectures are informational like that. And I tend to be more of a stream of consciousness guy--which I'm not--it could be laziness. I don't know. It probably is laziness. But also part of it is I'm not really trying to teach them something. I want them to be thinking in a particular kind of way. So, there's a big difference between--lectures have different purposes, obviously. But, let's talk about how, in designing a lecture or writing a book or having a podcast, how we might improve what people get from it other than just the passage of time. Andy Matuschak: I love it. So, in the essay I talk about ways in which it's possible to intentionally design media so that their grains bend in a particular way. And when you conduct a class-- Russ Roberts: By grain, you mean like in wood. Andy Matuschak: Yeah. Like the grain of wood. Russ Roberts: Not like rice. Andy Matuschak: Yeah. Thank you for clarifying. And, when you are conducting a class that is not so much informational but which you want to use to help students think in a particular way, you are shaping the grain of the discussion in that class by asking questions in a certain way. By using both your body language and your responses to tamp down certain lines of inquiry, steer them in a certain ways, to amplify others. You do that in your podcast all the time: You conduct the grain of this podcast in a certain way. And books can do this as well. And in creating new media we can do this. It's very interesting to think about constructing such environments in educational contexts. One example that I'll share that I'm just totally in love with: When I talk about seminar-oriented classes, people often say that, 'That's great for college,' or, 'That's really great for the Humanities; it's not going to work for Arithmetic.' And, there's this wonderful professor, Constance Kamii, who wrote a book called Young Children Reinvent Arithmetic, wherein she documents her experiments of having students do exactly that. She had a class of 5-year-olds in First Grade where she simply gave them a large library of games they could play, which she invented, such that the grain of the games kind of involved the students' learning; and in fact inventing for themselves arithmetic. And then they, you know, they took a standardized test at the end of the year; and they did fine. Etc., etc. It's a very inspiring story for me. Russ Roberts: Yeah. And I think there is a challenging mix between engagement and content acquisition. I think about it all the time, in the more edgy stuff that I've done--example being the Keynes-Hayek rap videos, with John Papola. So, we tried to make them entertaining, but we really tried hard to make them educational. And, by having them rhyme--which of course Lin-Manuel Miranda also understood when he did Hamilton--it changes the ability of people to remember, and to have a shorthand form of communication. I probably used the word 'jargon' earlier in our conversation, and I probably put it in sort of sneer-quotes. But jargon is incredibly valuable, as a way to do shortcuts in communication. I think a lot of these innovations in terms of teaching education and, there's a huge challenge in figuring out--you know, games are great. Kids love games. If you can get them involved in games, they are going to get more engaged. The more time you spend on the gaming part, the less time they spend, maybe, learning. But maybe what they do learn is acquired more deeply. So, I think there's just a lot of tradeoffs there that we really haven't started to think about enough. We'll probably need to worry about it a little bit. Andy Matuschak: Absolutely. It's really understudied. And one way I think about this--I hope you'll forgive me for using the term--is, basically I think about efficient frontiers, where, for a given amount of enjoyableness, how effective can we make the thing--the kind of learning interaction? And likewise, if we want a given degree of efficacy, how enjoyable can we make it? And, films, for instance, at a reasonably low level of efficacy are reasonably enjoyable. You can kind of sit back and watch an educational film; it doesn't take much effort, but you probably won't get a ton out of it. It's very enjoyable. Likewise--but books are a little bit different. It's hard to not engage all that effort fully with a book. But, kind of, at the maximum level, it goes somewhat higher. If you are willing to kind of have a not-so-enjoyable experience with a book--maybe it becomes enjoyable because you are fascinated--but you kind of really delve deep, then you might get something that's much, much, much more effective than a short YouTube video can convey.

46:40 Russ Roberts: I'd be interested to know if you can talk about this; if you don't want to, just say so. But, you know, Kahn Academy, I think, in the early days made a conscious decision to have what I would call low production values--not fancy, very straightforward, not particularly entertaining. The main entertainment opportunity was to increase the speed for some kids who were kind of already getting it, maybe, or who could acquire it a little more quickly. One thing we haven't talked much about is how people differ a lot in what they get out of something: The three things you are going to get out of Sapiens maybe aren't the same three things I'm going to get. And maybe aren't the three things I want to get out of it. Which is another challenge. But, I'm curious, Kahn Academy's impact on mathematics, I would say, is by far its biggest impact. The more ambitious stuff that you have worked on is uncertain as to how it's going to turn out, but I think in math, it's been extraordinary. Is it measurable in any sense? And, the extraordinary part is that feedback. I think that students don't just watch it and hear someone tell them how to do it. They are actually forced to interact with the material. What are your thoughts on that? Andy Matuschak: Sure. That notion has been part of Kahn Academy from the start. It famously began with him as Uncle Sal, tutoring his nieces; and he was doing it over Skype or some equivalent, with a tablet. And, after doing the session with drawing on the screen, you know, give the kids problems online with like a little website he wrote. This idea of instant feedback--it goes way, way, way back. And, it's very valuable for this kind of declarative knowledge and rote procedural knowledge. And, it's been hugely helpful. There actually are great efficacy studies on Kahn Academy, demonstrating pretty interesting benefits. It's not a savior; it's not a silver bullet, at all. It's done some good. It's great. I think that model is very valuable. I think there's a lot of work to do to scale it to things like we discussed earlier, like, you know, 'What was the primary cause of World War I?' Getting there, rather than how to multiply two-digit numbers, is going to be an extra thing. One other challenge for that model that we engaged a lot with over the years and haven't yet cracked is building the conceptual understanding. Like, you might understand how to multiply those two-digit numbers. But, do you really know what's going on? I'll share this example that really struck fear into my heart from research a couple of years ago. We were doing some kind of explanation-oriented stuff, where we wanted kids to generate explanations for how stuff worked. And, not just, you know, use a procedure. And so, we chose a bunch of kids who had gotten every single question right about a particular topic. Like, as far as the system was concerned, they knew everything there was to know about this topic. And then we asked the simple follow-up question like, 'Can you explain, you know, why this formula gives you the area of this kite?' A kite is like a geometric shape. And, we got this, like, horrifying set of answers: 'IDK': 'I Don't Know.' 'Math is weird.' 'I dunno[?]' 'Blank, blank, mmm, blank.' Zero students--it was several dozen students who had answered every question correctly--could explain that. Russ Roberts: So, the question there that I think is so interesting, and I talk about this with my wife all the time, which is: Getting all the answers right is great, as long as you don't have to apply it to something new. And then, you might be--it's possible you have the underlying understanding that lets you apply it to the new stuff. But if you don't, you're cooked. And you won't be a good engineer. You're going to struggle. You know, some of it--I think part of the advantages of learning math, which is just such an interesting question, just because so little of it actually applies to everyday life, and yet, in terms of the amount of hours that are spent just, say, on geometry, that it causes some people to say this is a waste of time. I wonder about building grooves[?] in the brain: I think there's probably neural connections, probably a good thing. But I think there's also a feeling of triumph and achievement that comes from mastering math that's different, that's special, that's different than "mastering the cause of World War I." It's just a very, to me, a very deep and interesting area to think about. But the idea of how much underlying stuff you should understand. My wife teaches calculus. And she teaches it very much as a 'let's understand it' class. And her students get to college; they get As in their college calculus class; and they find it unbearably boring because it's all cookbook. It's all punch out the right answer, and not understanding it very well--just not important. Andy Matuschak: I think what you are accessing now is maybe the greatest tragedy of all to me, which is that: Why learn about math? Let's put kind of political disagreements about forced education aside. Math is sooo beautiful. Maybe I'm not going to force you to learn it; but, oh, my gosh--understanding something really deeply--I'm not talking about arithmetic, but, for instance, why does e show up in so many places? It's so beautiful. And that's because you are connecting to something fundamental and true. These are Platonic objects. Okay--that is a controversial idea. And none of that, none of that is what is accessed in traditional K-12 education. You are learning it because you have to, or because I told you so, or perhaps because it will help you get a job; or perhaps, in a very enlightened environment, because it will help you think analytically and critically and abstractly, and those are skills which are useful in a variety of domains. Russ Roberts: One of my favorite moments as an undergrad was taking a graduate class in bio-statistics, which was hopelessly difficult. It was a really interesting aside, which was, after the midterm, the teacher called us all together; and he said, 'You all failed.' There were about 40 people in the class. And he was going to change his whole instructional strategy. He was deeply disturbed by this result. Which is interesting, because he could have said, 'Well, just bad students this year.' But he decided that he had not taught the material well, and he changed his classroom interactions. He actually called on us instead of doing pure transmissionism. I don't think he knew any of our names, but we thought he did. He would call out someone's name, and when someone looked up, he would look at them; and we thought, 'Wow, he memorized all our names. That is so cool,' after the midterm. He may have memorized some of the names, but he didn't learn our faces for at least a while. But, it was a very cool emotional technique. But, in that class, we learned to prove the Central Limit Theorem in, I think, three different ways. And, you could argue any one of them was a waste of time. Just: Here's the Central Limit Theorem; just believe it. It was one of the greatest intellectual experiences of my life. I didn't become a statistician. I've forgotten a lot of my statistics. We used, I think, characteristic functions that are called in that proof; I don't remember what they are; they have a capital Greek letter associated with them. I can draw it; I don't remember the name of it. Anyway, it was a beautiful thing. That's all I think I have to say. Andy Matuschak: That's wonderful. I'm so glad you had that experience. Russ Roberts: Yeah; no; it's cool. The analogy that you're going to be hung in a fortnight concentrates the mind wonderfully, as, I think Samuel Johnson said. And I thought if I didn't get an A in this class I wasn't going to get into grad school. So, I was pretty motivated. That may have had something to do with it.

54:43 Russ Roberts: Anyway, let's go back to podcasts. And let's think out loud for a few minutes in the time we have left what make EconTalk more valuable to listeners. Now, I know some of you out there get together every week with your friends, or your family, and talk about it. Not very many of you. But that there's more than 2 is deeply wonderful to me. But, would it be useful to have EconTalk clubs? And maybe it shouldn't be every week for that episode. Maybe it should be 3 weeks before. Maybe they'd be staggered, based on what you said earlier. But, what might make--and we don't have to just talk about EconTalk: obviously there are a lot of podcasts out there, and a lot of why people listen to them is just for entertainment--but, a lot of it is to learn stuff. What could we do that would be more useful? Just for fun today I asked--the Adam Cifu episode came out today on Medical Conservatism, and I asked people what's one thing you learned from this episode, drawing on A. J. Jacobs' idea that you should have a 'One Thing' document where you list one thing that you might want to remember. And, you know, this is a great idea. You should do this for every episode. But, of course, your one thing is not going to be the same as other people's. And a list of one things maybe is not so useful. But I don't know. So, what are your thoughts on that? Andy Matuschak: I'll give it a shot; maybe we can brainstorm together. Here's one variant of 'one thing' you might find interesting. There's kind of a classic textbook on instructional design called Understanding by Design. And it introduces this concept that I found very helpful called Essential Questions. And, it argues that we should try to structure our curricula and structure our courses to really focus on accessing essential questions. What those are, are the questions which are central to the discipline. They are the questions which are alive. They are the questions which are not resolved. They are the questions which you, the novice, and I, the expert, we are all engaging in. Russ Roberts: Love that. Andy Matuschak: We can discuss it at all levels; we can apply it, perhaps, outside of the discipline. So, for instance, essential questions around compound interest. For instance, 'Why does it appear so many places?'--let's just stop there--can apply in so many domains. So, one 'One Thing' that might be fun for your listeners is if you or your guests introduce an essential question from their field or from their work, which then the listeners can discuss amongst themselves or think of individually. Russ Roberts: So, that's an interesting--I love that idea. So, let me try to rephrase it, to see if I understand it, check for understanding here. I love this idea that I ask a guest what in their area of expertise is up for grabs. Right? And, of course, the idea that listeners might know as much as they do is deeply troubling to most experts. The other challenge, though, and this is where I'm going to ask you to try to come up with this, is: What's the forum? So, we have a feature on the website called Extras, where we have--we raise questions related to the episode. Most people don't go to them. Almost by definition. Most people are listening in their car, commuting. They don't go on the website actually at all. I think we've done a--we've not done a very good job in the Internet era of figuring out physical, web-based spaces that are physically designed--user-interfaced spaces that are effective. You know, a comment board is just not great. It's a long stream of monologue. Separate monologues. Occasionally, there's back and forth; we've added that in, in our recent redesign. And it's a little bit better. I think it's a plus. But, a place to chat--maybe it's physical. Literally, in the sense that maybe it's a club. Or maybe it's a place in time online that you know you can come and engage with other listeners at that point about a question that was raised toward the end of an episode. Andy Matuschak: That's wonderful. I'll share a couple of thoughts. First, one thing we learned from designing the sort of open-ended activity tool at Kahn Academy was that, not all discussions are equally productive or constructive. And, you don't want someone to have to like, wade through those to figure it out. And so, places like Reddit and HackerNews have kind of figured out mechanism which allow the most fruitful or generative discussions to rise to the top. Sometimes that can mean the most controversial, or clickbait-y; and there are ways of dealing with that. But, for instance, if there were a thread: What is the relationship between one's emotional connection to knowledge and one's detailed understanding of that knowledge? That's a question which a learning scientist can engage with extremely deeply. But it's also a question which everyone can engage with. Everyone has experienced, everyone has interesting things to say about that. So, maybe if I go to a thread that includes that question, at the top might be some of the most interesting responses, and I might feel inspired before I come up with my own answer to that question, to engage with one of those responses--because it was very striking to me. And, indeed, the momentum that I built in responding to that might then inspire me to try to answer the question myself. Russ Roberts: Yeah. I think that's a--let me ask you about the forum for that. Is there a--do you know of any places to interface with material that is, um, better than just--so, there's the comment that just goes down and down and down. Then there's the up-vote, so you get maybe the more interesting ones. But those are the same technology: it's slightly improved. Is there a chance to do something else? Something--maybe it's moderated? I'm not sure. Andy Matuschak: Yeah. Sure. Just a couple other models that come to mind. So, one of them is moderated-ish. And, it's used by a lot of bloggers. So, many bloggers who write series of essays on their blog will write something; it will include some provocations. The comment section will have a lot of discussions going on. The person who wrote the thing might engage and facilitate those discussions slightly. But the more important thing that they'll do is kind of pull some of those comments, the important ones up into the body itself. Or, to address them in the next essay. So, the equivalent here might be , either to have volunteers for the community be highlighting particularly interesting questions or discussions, or for you on a subsequent episode to kind of react to the ideas from the community that you found most interesting.