I have no idea what I’m doing, but that’s not necessarily a bad thing. To explain why, I’ll touch on evolutionary biology, the impostor syndrome, the Dunning–Kruger effect, video games, test-driven development, material exploration, beavers, judo and Douglas Adams. I hope to convince you that wilful ignorance is a vital ingredient of human creativity, and therefore of programmer creativity, because a programmer is a type of human.

In the end I decided, yes, it was an idea for a conference talk. This is that talk.

So what did it mean? Was it a cry for help? A product idea? A topic for a blog post? Or could it be an idea for a conference talk?

I was familiar with the internet meme that superimposes that phrase on photographs of dogs. For example:

What did it mean? Its message is perfectly accurate, but it was weird to see it written down like that, staring me in the face, challenging and mocking me.

It says “I HAVE NO IDEA WHAT I’M DOING”. I don’t remember writing it, but it was in my house on my desk in my handwriting, so I obviously did.

In the end I did make all the early examples pass, but I didn’t feel as though I’d properly understood anything. I hadn’t learned anything deep about Scheme or obtained any insight into the problem I was trying to solve. My interpreter would soon be so complex that I wouldn’t be able to make any progress, and I still wouldn’t have grasped anything interesting about Scheme. Again, that was disappointing.

The red line is how many lines of examples I had in spec/ , and the green line is how many lines of implementation code I had in lib/ . The distressing trend is that lib/ kept getting larger: every time I added examples of new features, my interpreter grew to support them, and it soon became really big and messy.

So as I wrote these examples in spec/ I was also putting implementation code into lib/ , and by making the examples pass I was gradually implementing a Scheme interpreter, because the book essentially explains how to evaluate Scheme.

I converted the first few chapters into RSpec examples in advance of our first meeting, so that we could immediately get started on making them pass. As I did the conversion I quickly tried to make each example pass on my own, just to check that I hadn’t made any careless mistakes in transcribing the information from the book.

Since we were all Ruby developers, we thought it might be fun to turn these questions and answers into RSpec examples. The idea was to work through the book and make those examples pass as our understanding of Scheme improved.

The Little Schemer has an interesting format. Each page is divided into two columns: the left column asks questions about Scheme expressions, and the right column answers them. The entire book is a dialogue between the two sides of the page.

Some friends and I decided to start a book club for The Little Schemer ; it’s a classic of computer science but none of us had read it, so we started meeting up regularly to work through it together.

I did lots of hoovering, I did lots of washing up — my flat has never been so clean — I stared out the window, and I didn’t know what to do. That was a disappointment as well.

I wrote pages and pages of notes to try to get my thoughts into some kind of order, which didn’t really work. I bought a stack of index cards, filled them with ideas, and spent hours arranging and rearranging them on the floor to see if they’d fit together; that didn’t really work either.

Another example: last year I was supposed to write a book that used Ruby to explain computation theory . I was initially very excited about this — the contract was signed, and I was really looking forward to being an O’Reilly author — but eventually the moment came when I was sitting alone in front of a blank text editor. I panicked a bit when I realised I’d not written many things longer than a tweet before, so I certainly didn’t know how to write an entire book.

I walked away from the computer and gave up. I didn’t have any idea how to write this talk or what it should be about. That was disappointing.

I sat and stared at it for a long time and didn’t know what to do. Eventually I started making notes. I diagrammed some vague ideas on paper, then I opened OmniOutliner and started making an outline. I wrote a list of big-picture ideas and broke each of them down into smaller ideas, filling out each small idea with as much information as I could think of. In my desperation I kept adding more material — ideas, drawings, pictures, quotations, pieces of research — until the outline became vast and unmanageable.

The most relevant example is the process of trying to write this talk. Most of the talks I give are teaching in one form or another, so in a typical talk I take someone else’s idea and explain it by deconstructing it into pieces that are easier to understand. But this talk’s different, because I didn’t know in advance what I wanted to say; I didn’t really have any ideas, just an oblique title written on an index card.

Those results are interesting and might be relevant to you, but ultimately the impostor syndrome and the Dunning–Kruger effect are problems, not solutions. Just knowing that you’re suffering from them doesn’t necessarily help. I’m sorry about that.

By educating someone who doesn’t know what they’re doing, you give them not only the specific skill in question but also the metacognitive skill of being able to more readily spot their own inadequacies; you make them better at what they do, but an unintended side-effect is that they’ll spend more time thinking that they’re not especially good, because they’ll assume that everyone else is just as good as them.

That means that if you start out with low competence in a particular domain you’re likely to be mistaken about your own ability, but as you improve you become more likely to be mistaken about other people’s abilities instead. Instead of assuming that you’re comparatively good, you begin assuming that everyone else is good as well, because as you become more capable you start to think “well, surely everyone else knows this too”.

The miscalibration of the incompetent stems from an error about the self, whereas the miscalibration of the highly competent stems from an error about others.

Conversely, better-performing individuals did improve the accuracy of their self-evaluations, because seeing other people’s papers helped them to realise that others hadn’t done as well as them:

Despite seeing the superior performances of their peers, bottom-quartile participants continued to hold the mistaken impression that they had performed just fine.

The study found that people who performed poorly on the test not only overestimated their own competence initially, but also maintained that overestimate even when they’d seen other people’s answers:

The grammar test took things a little further. After each participant had answered the test questions and estimated how well they’d performed, they were shown some other students’ answers and given the opportunity to re-estimate their own relative performance. The intention of this was to find out how someone’s assessment of their own competence would change after they’d seen how well (or how badly) others had done.

On the other hand, in the upper quartile you can see that the highest-performing participants have gone in the opposite direction and underestimated their own performance. (The paper’s authors argue convincingly that this isn’t a regression effect .)

The lighter line shows the actual results, and the darker line shows the participants’ perceived performance. You can see that most people’s perceived performance is way above their actual performance: the people in the lower quartile thought they’d performed better than perhaps sixty percent of other participants, whereas in reality they’d only done better than between zero and twenty-five percent of them.

(The first thing the paper reports, incidentally, is that “gender failed to qualify any results in this or any of the studies reported in this article, and thus receives no further mention”. So maybe gender essentialism is nonsense, and gender has no bearing on competence. Who knew?)

The students were given tests which assessed their sense of humour, logical reasoning skills and English grammar skills, and then asked to guess how they’d performed relative to the other participants — they had to decide whether they’d performed better than ten percent of other people who took the same test, or fifty percent, or ninety-nine percent, or whatever.

The paper’s quite well-known now; it was relatively widely publicised at the time because it won the Ig Nobel prize for useless psychological research. It summarises and analyses experiments conducted by David Dunning and Justin Kruger in which they asked university students to evaluate their own competence.

The impostor syndrome isn’t the only way to misjudge your own competence. You might also think things like “there’s so much that I don’t know”, or “any idiot can write a Ruby program”, or “everyone here’s a better programmer than I am”. Those thoughts can be a result of the Dunning–Kruger effect , which was identified in 1999 in a paper called Unskilled and unaware of it: how difficulties in recognising one’s own incompetence lead to inflated self-assessments .

Despite external evidence of their competence, those with the syndrome remain convinced that they are frauds and do not deserve the success they have achieved. Proof of success is dismissed as luck, timing, or as a result of deceiving others into thinking they are more intelligent and competent than they believe themselves to be.

That means that when you succeed, you’re not necessarily predisposed to incorporate that information into your self-image; you keep the success at a distance, and your self-image remains unchanged.

I’m talking about you specifically here; you know what I’m talking about. Sometimes you think things like “I’m just making stuff up as I go along”, or “I don’t belong in this group of people”, or “I’m going to be found out”.

But enough about me. As you’ve seen, I frequently find myself in situations where I don’t know what I’m doing, and I usually hit a brick wall and feel disappointed. But it’s not just me! You have no idea what you’re doing either.

If you only take one thing away from this talk, make it this: beavers are idiots. They have no idea why they’re building these huge structures; they just blindly do it.

Dams are really impressive structures. They’re a couple of metres high and they can be tens or hundreds of metres long. Building one is quite difficult — it involves understanding the weather and the seasons and the way that water flows — but of course beavers don’t know anything about meteorology or hydrodynamics or planetary axial tilt.

Beaver dams are made from wood, mud and stones. Beavers cut down trees with their teeth and use the timber to divert a stream and build a dam across it, creating a pond. The pond protects the beaver against predators, provides them with food, and prevents the underwater entrance to their beaver lodge from blocking up with ice in the winter.

It’s not the just the bodies of animals that fit snugly into their environment, it’s their behaviour too. This is a North American beaver :

A giraffe has a very long neck, which allows it to browse the tops of trees free from competition from other leaf-eaters. But no giraffe appreciates that advantage, or even understands it; they just get on with eating leaves.

All of these animals look remarkably like something in their environment, but none of them has any meaningful understanding of why they look that way. They don’t know what they’re imitating. The stick insect doesn’t know anything about eucalyptus; that’s just what it looks like.

The left one is on a birch tree, and the right one is on a willow tree. Again, these have remarkable attention to detail in their camouflage; the right one has little markings that look just like the markings on a willow branch. They both strongly resemble the trees that they live on.

It’s difficult to see because it’s on a tree and it has extremely detailed camouflage. You can just make out its head, front legs and tail, but it’s otherwise almost completely invisible.

I carelessly thought that it might be fun to have Arthur Dent and Ford Prefect thrown out of the airlock of a Vogon ship without spacesuits, just to see what would happen. Unfortunately, of course, if anything was going to happen, I was going to have to think of it. I got very stuck.

In the second episode of the series, its protagonists are thrown out of an airlock without spacesuits on. This situation is quite perilous and dramatic, but unfortunately Douglas Adams didn’t have a plan for how he was going to get the characters out of it.

He wrote the radio series by the skin of his teeth: budgets were small and deadlines were tight, so he was writing each episode immediately before it was recorded, and each episode was recorded immediately before it got broadcast.

The first person I wanted to mention is Douglas Adams , who wrote a radio series called The Hitchhiker’s Guide to the Galaxy which subsequently became a book, TV series and film.

I’ve talked about me and I’ve talked about you. Now let’s look at other people who have no idea what they’re doing.

Solutions

I’ve outlined a few problems: we’ve established that you and I and Douglas Adams and all those animals have no idea what we’re doing. But what can we do about it? I’m going to suggest a few solutions.

Shoshin One possible solution is shoshin, a concept from Zen Buddhism. Shoshin means “beginner’s mind”. The idea is to approach a problem as if you knew nothing about it, like a beginner learning about that problem for the first time. Shoshin is about abandoning your preconceptions, being receptive and eager and open to possibilities without making premature decisions about what’s possible. A Zen teacher said: “in the beginner’s mind there are many possibilities, in the expert’s mind there are few”. Sometimes being an expert can work against you, because it leads you to decide on a solution before you’ve had a chance to properly think about the problem. Beginner’s mind allows you to explore a problem more authentically and pay more attention to its characteristics, rather than constantly jump to conclusions based on your experience. Experience is valuable, but not as valuable as the ability to pay proper attention to the problem in front of you.

Judo Another useful strategy is judo. (I don’t mean the Basecamp lingo.) Judo is how Douglas Adams solved his problem with Ford Prefect and Arthur Dent being thrown into deep space: The problem was the sheer improbability of every solution I came up with. [The answer] came about through watching a TV programme about Judo. If you have a problem, said the instructor on the programme, such as for instance a nineteen stone [man] in pyjamas trying to beat you into a pulp, the trick is to use this problem to solve itself. If you can trip or throw or deflect [him] as he hurtles towards you, then the fact that he weighs nineteen stone quickly becomes his worry instead of yours. So — I thought — if my problem is one of improbability, let’s use Improbability to solve the problem, so just for the heck of it I invented the Infinite Improbability Drive, and gave myself a whole new thing to write about. The Hitchhiker’s Guide to the Galaxy, BBC By putting himself in a difficult situation without thinking about how he was going to get out of it, Douglas Adams forced himself to think creatively and use the problem to solve itself. It was only by realising that the real problem was the improbability of all of the easy solutions that he was able to come up with the idea of the Infinite Improbability Drive, which was funny and creative and turned out to be a good source of plots later in the series.

Rolling up There’s a video game called Katamari Damacy in which you play a character, the Prince, whose job is to roll a magical ball called a Katamari around a three-dimensional world. Katamari Damacy, Namco The Katamari starts out small and grows by picking up objects as you roll it. When it’s small the only things that stick to it are tin cans and stationery and bits of rubbish, but larger objects like animals and trees can become stuck to it as it grows. Over time it gets big enough that cars and buildings and even continents can stick to it. I find this game fun because the Katamari grows by rolling up the environment — it’s made of drawing pins and cows and clouds. Pieces of the world that start out as obstacles can become beneficial once the Katamari gets large enough to roll them up, and the more of the environment the Katamari absorbs, the more you’re able to do with it. The way a Katamari grows is reminiscent of how biological evolution works. Jean-Baptiste Lamarck, an eighteenth-century French naturalist, made lots of important contributions to evolutionary theory but unfortunately popularised one idea that turned out to be wrong, and now that’s all anybody remembers him for. His theory, Lamarckian inheritance, was the idea that an organism can acquire characteristics during its lifetime and then pass them on to its offspring. (This is sometimes called inheritance of acquired characteristics.) Lamarckian inheritance can be used to explain the evolution of giraffes’ long necks. The story goes that ancestors of modern giraffes, with their normal-length necks, were constantly reaching upwards to try and eat leaves from trees, exercising their neck muscles and causing them to strengthen and elongate over the course of a lifetime. Then, when two of these ancestral giraffes mated to produce a baby giraffe, the baby would be a kind of copy of the parents (the mechanisms of inheritance weren’t well understood in Lamarck’s time), so the baby would be born with a slightly elongated neck just like its parents. As the baby grew into an adult it would repeat the same pattern, reaching for leaves and lengthening its neck further, so that its own offspring would have longer necks still, and over many generations its descendants would eventually develop extremely long necks. Unfortunately for Lamarck, this is nonsense. The year that he published this idea, Charles Darwin was born, and it was Darwin who came up with the correct explanation. Darwin realised that adaptations like long necks are the result of differential reproductive success of individual organisms in the presence of random variation: firstly, the random recombination of genomes that happens when two animals mate to produce offspring, and secondly the occasional introduction of random mutations into that process. Mutations are usually disadvantageous and produce uncompetitive offspring, but occasionally, purely by chance, a mutation occurs that makes an animal slightly better adapted to its environment, giving it a slightly higher chance of surviving and a slightly higher chance of reproducing. That reproductive advantage allows the mutation to become more widespread in the gene pool, and as that process is repeated with each successive generation, more and more information from the environment gets rolled up into the genome of a species. Over evolutionary time, organisms work like sponges. They absorb information from their environment, they become better and better adapted to that environment, and their genome begins to encode more and more information about it. That’s how you end with insects that look a lot like willow branches or eucalyptus twigs, and beavers that know how to modify wetlands for their own benefit. In the case of the peppered moth caterpillars, birds are the conduit for this information. When a bird looks at a tree, it’s less likely to spot and eat a caterpillar that looks a bit more like a branch than the other caterpillars on the tree. The brains of birds have transmitted this information from the caterpillar’s physical environment into its genome over hundreds of thousands or millions of years.

Test-driven development Rolling up is also a big part of test-driven development (and I suppose agile development generally). TDD encourages you to work iteratively by making small changes, and to impose selective pressure on your implementation by writing tests. When you’re doing TDD, you write tests to characterise new features or to reproduce existing bugs. Acceptance tests roll up information from stakeholders about what the software is supposed to do; regression tests reproduce problems by rolling up information from the world about weird edge cases, acting as a sponge that soaks up knowledge from bug reports and stores it somewhere useful. This is one reason why rewriting software from scratch can be so dangerous. Old software has all of this knowledge already soaked into it; it bristles with fussy little fixes and weird little features that each represent a piece of information from the environment in which it’s being used. If you throw old software away and replace it with shiny new software, you run the risk of jettisoning all of that knowledge that’s been painstakingly rolled up into it. Putting that knowledge into a test suite helps to lock it up in a place where it can live longer than the implementation it’s testing. TDD encourages beginner’s mind as well, because when you’re working test-first you can write the tests and then make them pass without thinking too far ahead and without getting too invested in any particular implementation strategy. TDD practitioners sometimes speak about first writing the test while thinking hard about the problem, then switching into the different mode of focusing on simply trying to make the test pass. That mode switch makes it easier to have beginner’s mind with respect to the larger problem you’re trying to solve, because you can focus on the small task of getting a test to pass and not become distracted by your expert preconceptions about the big picture. That process encourages material exploration as well. It allows you to be guided by the grain of the problem, rather than just charging ahead and hacking out the solution you were already thinking of. It gives you a chance to take things more slowly and to feel your way through the obstacles that have been placed in your path by the test, helping you to channel your energy in what is more likely to be a productive and creative way. And there’s a little bit of judo in TDD, I think. If you use tests to characterise a problem, that makes it easier to use the problem against itself when you’re writing the implementation, letting each failing test guide you incrementally towards a solution. Once you’ve put the problem’s constraints in place, they might help you to see how you can solve it in an elegant way, leading you towards a more natural solution which mirrors the structure of the problem.

Refactoring toward deeper insight “Refactoring toward deeper insight” is a phrase from Eric Evans’ book Domain-Driven Design, and it’s how I fixed my Scheme interpreter. To recap, this was the situation I found myself in: I got a bit dispirited at this point, so I put the project away for a bit and came back a few weeks later with a clearer head. This time I tried to be a little more patient and calm and pay more attention to what was going on in the code; previously it had been a mad rush to try to get the examples written in time for the book club. When I came back to it I did some more thoughtful, more patient refactoring, and an interesting thing happened: The point where the green line drops off is where I did some refactoring and discovered some hidden structure in my implementation. The refactoring wasn’t just moving furniture around; I was doing mindful stuff like eliminating duplication, introducing new abstractions, looking for interesting patterns, and to a certain extent thinking about what the book was trying to say. As I thought more about it, and tried to arrange the code in a way that better reflected what the book was trying to say, suddenly different pieces of my implementation began clicking together. Whole classes collapsed down to single concepts, and I had an amazing few days where I was able to delete a lot of code as more and more concepts clicked into place. That was the point when I started getting some insight. Now that I’ve gone through that process, I feel like I finally understand what’s amazing about Scheme. Writing the interpreter had felt like a real pain at first, and it wasn’t until I explored the code by refactoring it — and saw how it was supposed to work — that I felt like I was getting something out of the exercise.