As someone who has dabbled in lab experiments for over a decade, I’m familiar with complaints about external validity. If an experiment is run with only college students, then how can we know if the finding will generalize to other populations? It’s a question worth asking, but many questions are worth asking and it doesn’t mean that controlled experimentation can’t add value to the economics literature. In the age of general suspicion of small studies, people say that replications are needed. We should only trust a conclusion that is supported by multiple studies. The thing about replications is that the process has to start somewhere. Empirical work has to get read and published. Replications are composed of individual studies.
I just met John List at the Alabama stop on his epic national book tour. He directed me to his work of art: Ungated Link. He wrote a play in response to the attacks on his work concerning external validity. He employs a rhetorical strategy of making your critics look obtuse. Even though the play is absolutely silly (thoroughly entertaining), he builds a strong defense for doing experiments. It is literally presented as the arguments of a defense lawyer. Before the trial begins, a “reporter” summarizes the conflict that has created the need for a formal trial:
Court Reporter Clifton Hillegass: Thank you Judge Learner. While it is never easy to convey succinctly the key points of a debate, this dispute has crystallized in a manner that leaves no middle ground. The prosecution, led by Mr. Naiv Ete, argues that all empirical work in economics must pass a set of necessary external validity conditions before being published in academic journals or used by policymakers. To date, in this courtroom no empirical work has passed his conditions, effectively rendering the question of generalizability beyond dispute, or as Livius Andronicus reminded us, Non est Disputandum de Generalizability. Ms. Minerva, Lead Defense, has argued that this line of reasoning leaves only theoretical exercises and thought experiments to advance science and guide policymaking, an approach that she fears will return us to the dark ages.
The paper is called “NON EST DISPUTANDUM DE GENERALIZABILITY?” It’s a good refresher on the history of science, not just economics.
Maybe the first best is for you to spend your weekend reading dense technical papers. But if you aren’t feeling up to that, then this play will make you feel like you learned something without even trying.
I’ll link this up to some of the posts I wrote last year about experiments and critics:
Economist Writing Every Day 