Many of you may be aware of the fact that a leading psych journal is set to publish an article that allegedly supports the thesis that ESP exists.

It has set off a debate in the scientific community about data analysis. See the link below.

http://www.nytimes.com/2011/01/11/science/11esp.html?_r=1&src=tptw

The main issue is that some statisticians have long been critical of a commonly accepted methodology that they think is too permissive when determining statistical significance with respect to small sample sizes. They are now rejoicing that this ESP paper is going through, because they think it will shed light on what they have long thought is a widely-accepted, bad methodology.

I think this has some bearing on the recent experimental philosophy (xphi) movement. I’ve expressed concerns here about some very popular xphi papers that allege to show that cultural background influences intuitions about philosophical thought experiments. My main concern was the incredibly small sample size and whether the studies included statistically significant differences. I was assured by some active xphi philosophers that these sample sizes plus the standards for determining statistical significance are widely accepted, and so I took them at their word.

Note, however that these statisticians are criticizing precisely this fact. They claim that this method is widely accepted and that it is bad that this is this case. Also note, that they have as one of their targets certain kinds of social science studies that have an xphi flavor – namely, looking at what sorts of factors might bias behavior.

My question is: if much of current xphi is modeling itself after these these social science methods: is xphi as currently practiced in trouble?

Update: Here’s one place where I raised concern about the sample size of a popular xPhi paper