I don't know if I've shared my econ hierarchy with you before, but I'd be curious what you think about it.
If we order methodologies as follows:
  1. Austrian: deductive reasoning from first principles
  2. Mathematical modelling: a few simplifying assumptions plus mathematical logic
  3. Experimental econ: well designed experiments testing points of ambiguity in theory
  4. Econometrics: statistical analysis of real world data
I basically have more confidence in the results from higher on the list than the results from lower and really I only believe results from lower if they can be made consistent with results from higher.
Ironically, all of my professional work is from the lowest rung of the ladder.
I'd basically agree with your ordering, except that I'd say that #1 and #2 usually end up with "it depends" (e.g. on some key elasticity), which then necessitates #3 and #4.
#3's problem is that this is pretty much impossible for most questions of economic significance. Small scale experiments aren't sufficient because they don't reflect real world decision-making conditions.
And #4 of course is full of problems, so the credibility really just depends on the research design and the circumstances of the study. IMO one paper is insufficient to conclude anything, but if you have lots of papers with different methodologies, across different settings, all showing the same thing, you can have more confidence. Still have to be careful, due to various biases in the publication process.
reply
I should have had the caveat "when appropriate" for the ones below being consistent with the ones above.
Where I think this hierarchy helps is that it limits the range of interpretations of results from the weaker methodologies. Instead of thinking a regression overturned some foundational economic principle, the researcher would instead try to figure out what other mechanism might be at play.
reply
I mean, the types of questions that people are answering these days with their econometric tools are so small-scale compared to what Austrians like to talk about, that it's not even necessarily a methodological gap it's also simply a difference in the scope of analysis.
I suppose I'm coming at it from a microeconomist's perspective. I think I'd agree with the Austrians more in their critique of math/stats when it comes to macro. (As you can tell, I'm not really a fan of modern macro.)
reply