“There are two things you are better off not watching in the making: sausages and econometric estimates. This is a sad and decidedly unscientific state of affairs we find ourselves in. Hardly anyone takes data analyses seriously. Or perhaps more accurately, hardly anyone takes anyone else’s data analyses seriously.”

That is the scathing critique that economist Ed Leamer leveled at empirical research in his famed 1983 article “Lets Take the Con Out of Econometrics”. At the time, he meant that researchers knew not to trust other researchers’ estimates much because they were sensitive to arbitrary choices made throughout the research process. But for most of the decades since Leamer’s critique, the educated public has tended to take peer-reviewed studies seriously.

This started to change with physician John Ioannidis’ 2005 hit article “Why Most Published Research Findings Are False”. Concerns grew rapidly through the “replication crisis” of the 2010s, assisted by the growth of social media. Psychology was hit first and hardest, starting with the 2011 article “False Positive Psychology”. But economics and the rest of the social sciences haven’t been spared.

A core premise of science is that research should be replicable. If one scientist creates an experiment to measure a physical constant like the speed of light, and they document their experiment well enough, other scientists should be able to perform the same experiment and find the same result. If one lab’s results can’t be replicated anywhere else, then like cold fusion, they probably aren’t real.

Outside of hard sciences like physics we don’t expect to get the same precision. Perhaps one trial finds a drug reduces heart attacks by 17%, while another finds 14%. But for research to usefully inform our actions, it needs to be at least somewhat replicable. If one trial found a drug worked but every subsequent trial found it did nothing, people probably shouldn’t take the drug.

...read more at econlib.org