Economist Writing Every Day

1. Trust literatures over single papers
2. Common sense and Bayes’ Rule agree: extraordinary claims require extraordinary evidence
3. Trust more when papers publicly share their data and code
4. Trust higher-ranked journals more up to the level of top subfields (e.g. Journal of Health Economics, Journal of Labor Economics), but top general-interest journals can be prone to relaxing standards for sensationalist or ideologically favored claims (e.g. The Lancet, PNAS, Science/Nature when covering social science)
5. More recent is better for empirical papers, data and methods have tended to improve with time
6. Overall effects are more trustworthy than interaction or subgroup effects, the latter two are easier to p-hack and necessarily have lower statistical power
7. Trust large experiments most, then quasi-experiments, then small experiments, then traditional regression (add some controls and hope for the best)
8. The real effect size is half what the paper claims

That last is inspired by a special issue of Nature out today on the replicability of social science research. An exception to rule #4, this is an excellent project I will write more about soon.