pull down to refresh
34 sats \ 8 replies \ @SimpleStacker 3 May \ parent \ on: Replication crisis in biology: p-hacked hypothesises science
I'm actually a bit more sympathetic to "exploring hypotheses" and don't consider it p-hacking per se.
Sometimes you get a big dataset and you're looking to uncover patterns and causal relationships. Exploring various hypotheses, some of which will fail and some of which will succeed, is entirely appropriate
In my opinion the problem of p-hacking is promoting a hypothesis that isn't robust by only presenting the specifications that give you a good enough p-value
Another problem is publication bias in not publishing null results, which makes scientific consensus seem more certain that it really is
I completely agree. I don't think it's necessary to confine yourself to the prespecification, but I think it's good to have it and to report what is found when doing it.
Since nothing ever goes according to plan, though, I'm fine with exploring other possibilities from there.
In one sense, you could just raise the standards for statistical significance to account for how many specifications you explored. The problem is looking at 100 things and then reporting as though the 5% significance on the coefficients from one specification is very meaningful.
reply
Agree that preregistered studies would go a long way
We're far from getting anything close to nuance though given how lacking statistical education is. I literally had a biology phd teach my kid that a p-value of 0.05 or below means that X causes Y 🤦🏻♂️
reply
It's truly amazing how poorly most empirical disciplines understand empirical science.
reply
This is one area that I found the natural sciences surprisingly lacking in. I'm really glad for my econometrics education in that regard
reply
We have the most endogeneity issues to worry about, so we think about this stuff the most.
reply