When you peer-review a paper, you can make one of a few basic recommendations to the editor. One option is this: do not publish the paper.

So what criteria should you use to make such a recommendation? In this post, I argue that some criteria are better than others.

1. Is the paper convincing?

A friend of mine mentioned this criterion the other day: “…[philosophy] papers ought to be convincing.” Call this the Convince Me standard or CM.

Maybe you think that CM sounds like a reasonable standard for peer-review. I don’t. I think CM let’s in too much noise. There are just too many unimportant factors that can have an impact on our being convinced (or not being convinced).† We can be convinced by bad arguments. Further, we can fail to be convinced by good arguments. This suggests that the quality of an argument is not always the only determining factor in our being convinced (or not being convinced) by an argument. So what other factors might influence us? Perhaps we are disposed to accept or reject certain arguments. Or perhaps we are less disposed to reflect on the quality of arguments, in general. For example, religious believers seem to be less reflective than atheists and agnostics (Pennycook, Ross, Koehler, and Fugelsang 2016).

2. Are the arguments logically valid?

If CM won’t work, then what other standard should we use? Analytic philosophers will probably agree about one thing: arguments should be logically valid. (Not sure what logical validity is? Check out the video below).

Fair enough. So what counts as “evidence”?

3. Do the premises seem intuitively correct?

If a claim is intuitive, is that claim thereby supported by evidence? Consider premise P:

P: gratuitously torturing human babies is bad.

You might think this premise is obviously true. Philosophers sometimes count on you having such intuitions. Consider this line: “[my claim] is so intuitive that most will need no more proof than its statement” (Wenar 2008). So, when a philosopher points out that you intuitively accept P, has she thereby presented evidence for P?

To answer that question, we must first answer a prior question: Is there a well-confirmed developmental theory about intuition that would explain

4. Are the premises well-supported by evidence?

5. What if the evidence isn’t clear?

There’s a catch: philosophers often use premises about which there aren’t any well-confirmed generalizations, one way or the other. The reason might be that the relevant science is too amniotic and undeveloped to have discovered well-confirmed empirical generalizations or that, in some domains, empirical generalizations just aren’t a thing (e.g., math). This suggests that there might be arguments to which WCG does not apply.

So how do we evaluate claims about a domain in which there are no well-confirmed generalizations? Perhaps we have to tentatively fall back on assumptions. But not any assumption will do. I will propose that such assumptions should be (i) widely shared and/or (ii) pragmatically useful. Call this the practical speculation criteria, or PS.

Conclusion

There are too many irrelevant factors involved in our being convinced (or not convinced) by an argument, so we cannot include the Convince Me, or CM, in our criteria for publication. We probably want to include logical validity in our criteria, but we will need more than that if we want to prevent bad (albeit valid) arguments from being published. This is why we need the well-confirmed generalization criterion, or WCG. And when that criterion doesn’t apply, we might look to intuitions. But even if we grant that intuitions count as evidence, they might not be widely shared and/or pragmatically useful, so I would prefer the practical speculation, or PS criterion to the intuitionist, or I, criterion.

This leaves us with a combination of criteria: the logical validity (V) criterion and either the well-confirmed generalization (WCG) criterion or (if that doesn’t apply, then) the practical speculation criterion (PS). Call this the Validity plus Evidence or else Practical Speculation criteria, or V⋅WCG∨PS for short.

On this criteria, reviewers can reject a paper if the paper lacks a logically valid argument and if the premises are not supported by widely accepted and/or pragmatic assumptions (or by well-confirmed empirical generalizations, where applicable). But reviewers should not reject a paper merely because the paper didn’t convince them or because certain premises were counterintuitive.

Disclaimer: There might be additional standards and criteria that we wish to impose as peer-reviewers — feel free to suggest some in the comments. I only mean to argue that we need not impose CM and I and that, at the very least, we should impose V⋅WCG∨PS.

Notes

† I am not claiming that the quality of an argument has no effect on whether one is convinced by the argument; I am only saying that the quality of an argument is only one among a suite of factors that can have an effect on whether one is convinced by the argument.

Featured image: “Urval av de bocker…” by Johannes Jansson, CC BY 2.5 Denmark