Replication

How to Set Working Directory in R for Replication Packages

JoyLeave a comment

The AEA Data Editor kicked it all off with this tweet:

“Please stop using “cd” (in Stata) or “setwd()” (in R) all over the place. Once (maybe, not really), that’s enough.”

Replies proliferated on #EconTwitter this week. In this blog post I am collecting solutions for R.   These days you might share the code used to generate your results for an empirical paper. That code would ideally be easy for other people to run on their own computers. File paths are hard (as I blogged previously).

A project for a single paper might have multiple code files. The code interacts with data stored somewhere. Part of the task of the code is to point the statistical program to the data set. It is frustrating if an outsider is trying to replicate a result and must alter the code in multiple places to point to their own location of the data.

Here is a concise summary of good practice, for any code language: “cd and setwd() specify the directory. When you share code and run on a different computer, they don’t work. Therefore, good practice to only specify once, at the beginning”

Continue reading →

SCORE Replications- Final Call

James Bailey1 Comment

The Systematizing Confidence in Open Research and Evidence (SCORE) project is an attempt to replicate hundreds of social science papers, and to search for patterns that predict what types of papers are more likely to replicate. You can read all about it at their site, and get a sense of its bigger picture importance in this great post by someone who participated in their prediction markets.

I’ve been involved in the project replicating and reviewing papers, and I plan to do a long post about what it taught me later this year. For now I just wanted to highlight that you can still join the project now, but the chance is ending soon, probably at the end of the month. I think its a great opportunity to advance science, work with the Center for Open Science on a DARPA-funded project, and get paid:

We are recruiting researchers and data analysts across the social-behavioral sciences for replication projects that use existing data that was not part of the original study. For these projects, you will select an original finding to replicate, receive or find alternative data to test the original claim, plan the preregistration of your analysis, receive peer review of your plan, and report your findings in a structured format. You will receive $3,000-$7,800 for each replication study, and you will also be eligible for co-authorship on the report of all replication studies.

New: Journal of Comments and Replications in Economics

James BaileyLeave a comment

I was pleased to see yesterday the announcement of a new journal, the Journal of Comments and Replications in Economics. As the name implies, it will publish articles that comment on or attempt to replicate previously published economics papers.

While empirical economics papers have in some ways become more believable over time, it is still rare for anyone to verify whether the results can actually be replicated, and formal comments on potential problems in published papers have actually become less common over time (though Econ Journal Watch has been a good outlet for comments).

The ability to independently verify and replicate findings should be at the core of science. But economists, like most other disciplines, are generally too focused on publishing original work to test whether already-published papers hold up. When we do try to replicate existing work, the results aren’t very encouraging; at best 80% of economics papers replicate.

If we want people to trust and rely on our work, we need to do better than that. The US Department of Defense agrees, and funded a huge project to determine what types of social science research hold up to scrutiny. I’ve been a bit involved in this and hope to sum up some of the results once this semester is over. For now, I’ll just say I’m happy to see the new Journal of Comments and Replications in Economics (and that it is both free and open-access, a rare combo) and I hope this represents one more small step towards economics being a real science.