Xkcd recently did a cartoon poking fun at what happens to science when people ascribe special meaning to certain numbers when calculating a p-value:
Prominent statistician Andrew Gelman rightly pointed out that similar issues could and would easily occur if everyone switched from p-values to Bayes factors, so the joke isn’t really about p-values necessarily.
I definitely agree switching to Bayes factors wouldn’t make this problem go away, but it would help matters somewhat. No matter how many times well meaning statisticians remind everyone that a p-value isn’t the probability of the null hypothesis, people can’t seem to get away from that idea. This is probably because the p-value is the standard statistician’s answer to a scientist’s request for a posterior probability. For certain assumptions about the prior information, you can calculate both of these quantities, and what typically happens is that a p-value of 0.05 occurs when the posterior probability of the null hypothesis is somewhere around 50% (this depends on assumptions, which is an opportunity to learn about logic, not something to be afraid of).
So if we did all go Bayesian, the problems of publication bias, researcher degrees of freedom, etc would be reduced, simply because all those “significant” results with p between about 0.01 and 0.05 would be seen for what they really are: evidence in favour of the null hypothesis, or at best, very slight evidence against it.
While it seems unlikely that everyone will go Bayesian soon, I propose a temporary fix up to p-values. All p-values should be multiplied by 10. Then everyone’s intuitions about strength of evidence will be made much more accurate.
Edit: Here’s a nice post with similar conclusions from a medical perspective, with a good explanation for those who like to think in terms of false positive rates etc, rather than my emphasis on plausibilities and so on.