Fixing the Peer Review Process by Crowdsourcing It? (Continued)

We call the fallout to any article the “comments,” but since they are often filled with solid arguments, smart corrections and new facts, the thing needs a nobler name. Maybe “gloss.” In the Middle Ages, students often wrote notes in the margins of well-regarded manuscripts. These glosses, along with other forms of marginalia, took on a life of their own, becoming their own form of knowledge, as important as, say, midrash is to Jewish scriptures. The best glosses were compiled into, of course, glossaries and later published — serving as some of the very first dictionaries in Europe.

Any article, journalistic or scientific, that sparks a debate typically winds up looking more like a good manuscript 700 years ago than a magazine piece only 10 years ago. The truth is that every decent article now aspires to become the wiki of its own headline.

Sure, there is still the authority that comes of being a scientist publishing a peer-reviewed paper, or a journalist who’s reported a story in depth, but both such publications are going to be crowd-reviewed, crowd-corrected and, in many cases, crowd-improved. (And sometimes, crowd-overturned.) Granted, it does require curating this discussion, since yahoos and obscenity mavens tend to congregate in comment sections.

That’s from a New York Times op-ed in last weekend’s Sunday Review by Jack Hitt, who is also a frequent contributor to This American Life (here is my favorite This American Life story by Jack Hitt).

Hitt’s point should be be taken more seriously by academics. In all fairness, however, in some corners of academia, the idea is being taken seriously: the AEJs — the four new journals of the American Economic Association — have comments section for every published article (I don’t know why the AEA has not also done so for its flagship journal, the American Economic Review.)

Unfortunately, readers of the AEJs seem to be slow to embrace that change, as few articles appear to have garnered any comments. Moreover, a quick look at the latest issue of each AEJ indicates no comments at all. Perhaps the problem is that one needs to be a member of the AEA to comment.

If those comments thread ever take off, and if other journals start offering similar comment sections, this would be a cheap, quick way of building canonical knowledge within any discipline, as I discussed in my previous post on this topic.