This page is about gwern.net ; for information about me, see Links.

“Of all the books I have delivered to the presses, none, I think, is as personal as the straggling collection mustered for this hodgepodge, precisely because it abounds in reflections and interpolations. Few things have happened to me, and I have read a great many. Or rather, few things have happened to me more worth remembering than Schopenhauer’s thought or the music of England’s words.” “A man sets himself the task of portraying the world. Through the years he peoples a space with images of provinces, kingdoms, mountains, bays, ships, islands, fishes, rooms, instruments, stars, horses, and people. Shortly before his death, he discovers that that patient labyrinth of lines traces the image of his face.” Jorge Luis Borges, Dreamtigers Epilogue

The content here varies from statistics to psychology to self-experiments/Quantified Self to philosophy to poetry to programming to anime to investigations of online drug markets or leaked movie scripts (or two topics at once: anime & statistics or anime & criticism or heck anime & statistics & criticism!).

I believe that someone who has been well-educated will think of something worth writing at least once a week; to a surprising extent, this has been true. (I added ~130 documents to this repository over the first 3 years.)

Target Audience “Special knowledge can be a terrible disadvantage if it leads you too far along a path you cannot explain anymore.” Brian Herbert (Dune: House Harkonnen) I don’t write simply to find things out, although curiosity is my primary motivator, as I find I want to read something which hasn’t been written—“…I realised that I wanted to read about them what I myself knew. More than this—what only I knew. Deprived of this possibility, I decided to write about them. Hence this book.” There are many benefits to keeping notes as they allow one to accumulate confirming and especially contradictory evidence , and even drafts can be useful so you Don’t Repeat Yourself or simply decently respect the opinions of mankind. The goal of these pages is not to be a model of concision, maximizing entertainment value per word, or to preach to a choir by elegantly repeating a conclusion. Rather, I am attempting to explain things to my future self, who is intelligent and interested, but has forgotten. What I am doing is explaining why I decided what I did to myself and noting down everything I found interesting about it for future reference. I hope my other readers, whomever they may be, might find the topic as interesting as I found it, and the essay useful or at least entertaining–but the intended audience is my future self.

Development “I hate the water that thinks that it boiled itself on its own. I hate the seasons that think they cycle naturally. I hate the sun that thinks it rose on its own.” Sodachi Oikura, Owarimonogatari (Sodachi Riddle, Part One) It is everything I felt worth writing that didn’t fit somewhere like Wikipedia or was already written I never expected to write so much, but I discovered that once I had a hammer, nails were everywhere, and that supply creates its own demand .

Long Site “The Internet is self destructing paper. A place where anything written is soon destroyed by rapacious competition and the only preservation is to forever copy writing from sheet to sheet faster than they can burn. If it’s worth writing, it’s worth keeping. If it can be kept, it might be worth writing…If you store your writing on a third party site like Blogger, Livejournal or even on your own site, but in the complex format used by blog/wiki software du jour you will lose it forever as soon as hypersonic wings of Internet labor flows direct people’s energies elsewhere. For most information published on the Internet, perhaps that is not a moment too soon, but how can the muse of originality soar when immolating transience brushes every feather?” Julian Assange (“Self destructing paper”, 2006-12-05) One of my personal interests is applying the idea of the Long Now. What and how do you write a personal site with the long-term in mind? We live most of our lives in the future, and the actuarial tables give me until the 2070–2080s, excluding any benefits from caloric restriction/intermittent fasting or projects like SENS. It is a common-place in science fiction that longevity would cause widespread risk aversion. But on the other hand, it could do the opposite: the longer you live, the more long-shots you can afford to invest in. Someone with a timespan of 70 years has reason to protect against black swans—but also time to look for them. It’s worth noting that old people make many short-term choices, as reflected in increased suicide rates and reduced investment in education or new hobbies, and this is not due solely to the ravages of age but the proximity of death—the HIV-infected (but otherwise in perfect health) act similarly short-term. What sort of writing could you create if you worked on it (be it ever so rarely) for the next 60 years? What could you do if you started now? Keeping the site running that long is a challenge, and leads to the recommendations for Resilient Haskell Software: 100% FLOSS software , open standards for data, textual human-readability, avoiding external dependencies , and staticness . Preserving the content is another challenge. Keeping the content in a DVCS like git protects against file corruption and makes it easier to mirror the content; regular backups help. I have taken additional measures: WebCitation has archived most pages and almost all external links; the Internet Archive is also archiving pages & external links . (For details, read Archiving URLs.) One could continue in this vein, devising ever more powerful & robust storage methods (perhaps combine the DVCS with forward error correction through PAR2, a la bup), but what is one to fill the storage with?

Finding my ideas To the extent I personally have any method for ‘getting started’ on writing something, it’s to pay attention to anytime you find yourself thinking, “how irritating that there’s no good webpage/Wikipedia article on X” or “I wonder if Y” or “has anyone done Z” or “huh, I just realized that A!” or “this is the third time I’ve had to explain this, jeez.” The DNB FAQ started because I was irritated people were repeating themselves on the dual n-back mailing list; the modafinil article started because it was a pain to figure out where one could order modafinil; the trio of Death Note articles (Anonymity, Ending, Script) all started because I had an amusing thought about information theory; the Silk Road 1 page was commissioned after I growsed about how deeply sensationalist & shallow & ill-informed all the mainstream media articles on the Silk Road drug marketplace were (similarly for Bitcoin is Worse is Better); my Google survival analysis was based on thinking it was a pity that Arthur’s Guardian analysis was trivially & fatally flawed; and so on and so forth. None of these seems special to me. Anyone could’ve compiled the DNB FAQ; anyone could’ve kept a list of online pharmacies where one could buy modafinil; someone tried something similar to my Google shutdown analysis before me (and the fancier statistics were all standard tools). If I have done anything meritorious with them, it was perhaps simply putting more work into them than someone else would have; to quote Teller: “I think you’ll see what I mean if I teach you a few principles magicians employ when they want to alter your perceptions…Make the secret a lot more trouble than the trick seems worth. You will be fooled by a trick if it involves more time, money and practice than you (or any other sane onlooker) would be willing to invest.” “My partner, Penn, and I once produced 500 live cockroaches from a top hat on the desk of talk-show host David Letterman. To prepare this took weeks. We hired an entomologist who provided slow-moving, camera-friendly cockroaches (the kind from under your stove don’t hang around for close-ups) and taught us to pick the bugs up without screaming like preadolescent girls. Then we built a secret compartment out of foam-core (one of the few materials cockroaches can’t cling to) and worked out a devious routine for sneaking the compartment into the hat. More trouble than the trick was worth? To you, probably. But not to magicians.” Besides that, I think after a while writing/research can be a virtuous circle or autocatalytic. If one were to look at my repo statistics, you see that I haven’t always been writing as much. What seems to happen is that as I write more: I learn more tools eg. I learned basic meta-analysis in R to answer what all the positive & negative n-back studies summed to, but then I was able to use it for iodine; I learned linear models for analyzing MoR reviews but now I can use them anywhere I want to, like in my Touhou draft material. The “Feynman method” has been facetiously described as “find a problem; think very hard; write down the answer”, but Gian-Carlo Rota gives the real one: Richard Feynman was fond of giving the following advice on how to be a genius. You have to keep a dozen of your favorite problems constantly present in your mind, although by and large they will lay in a dormant state. Every time you hear or read a new trick or a new result, test it against each of your twelve problems to see whether it helps. Every once in a while there will be a hit, and people will say: “How did he do it? He must be a genius!”

I internalize a habit of noticing interesting questions that flit across my brain eg. in March 2013 while meditating: “I wonder if more doujin music gets released when unemployment goes up and people may have more spare time or fail to find jobs? Hey! That giant Touhou music torrent I downloaded, with its 45000 songs all tagged with release year, could probably answer that!” (One could argue that these questions probably should be ignored and not investigated in depth—Teller again—nevertheless, this is how things work for me.)

if you aren’t writing, you’ll ignore useful links or quotes; but if you stick them in small asides or footnotes as you notice them, eventually you’ll have something bigger. I grab things I see on Google Alerts & Scholar, Pubmed, Reddit, Hacker News, my RSS feeds, books I read, and note them somewhere until they amount to something. (An example would be my slowly accreting citations on IQ and economics.)

people leave comments, ping me on IRC, send me emails, or leave anonymous messages, all of which help Some examples of this come from my most popular page, on Silk Road 1: an anonymous message led me to investigate a vendor in depth and ponder the accusation leveled against them; I wrote it up and gave my opinions and thus I got another short essay to add to my SR page which I would not have had otherwise (and I think there’s a <20% chance that in a few years this will pay off and become a very interesting essay). CMU ’s Nicholas Christin, who wrote a paper by scraping SR for many months and giving all sorts of overall statistics, emailed me to point out I was citing inaccurate figures from the first version of his paper. I thanked him for the correction and while I was replying, mentioned I had a hard time believing his paper’s claims about the extreme rarity of scams on SR as estimated through buyer feedback. After some back and forth and suggesting specific mechanisms how the estimates could be positively biased, he was able to check his database and confirmed that there was at least one very large omission of scams in the scraped data and there was probably a general undersampling; so now I have a more accurate feedback estimate for my SR page (important for estimating risk of ordering) and he said he’ll acknowledge me in the/a paper, which is nice.

