Economist Writing Every Day

You’ve probably heard the phrase that US states are often “laboratories of democracy.” The phrase comes from a Supreme Court case. It’s well known enough that it has a short Wikipedia page. The basic idea is simple: states can try out different policies. If it works, other states can copy it. If it doesn’t work, it only hurts that state.

The 2020-21 pandemic has provided a number of possibilities for the “states as laboratories” concept. Here’s three big ones I can think of (please add more in the comments!):

1. Do states that impose stricter pandemic policies (“lockdowns”) have better or worse outcomes? This could be about health, the economy, both, or some other outcome.
2. Do states that end unemployment benefits sooner have quicker labor market recoveries? Or are these not the main drag on the labor market?
3. Do states that offer incentives for vaccination have higher vaccination rates? And what sort of incentives work best?

These are all good questions, but let me throw some cold water on this whole concept: we might not be able to learn anything from these “experiments”! The primary reason: the treatments aren’t randomly assigned. States choose to implement them.

Let’s think through the potential problems with each of these three areas:

1. States that imposed stricter pandemic policies were also probably hit harder and earlier by the virus. Even if, say, New Jersey has more deaths than the average state, that doesn’t mean the policies didn’t reduce deaths. It’s possible they would have been even higher! And even if NJ has worse economic outcomes, it doesn’t mean the strict policies caused these outcomes (or all of the outcome). The bad economic outcomes could have been caused by people reacting to the spread of the virus. Policy and people are reacting to the same things. It’s hard to separate these effects.
2. States that end UI benefits early might just be different than other states. Perhaps they have lower unemployment rates to start. They certainly have Republican governors. Maybe they just have more flexible labor markets in general because of existing policy differences. Of course, you can try to control for all these factors! But you might still be missing some unobservable differences.
3. As with UI benefits, perhaps states that offer incentives had lower vaccination rates. But lower vaccination rates mean that you have more room to grow! If the trajectory is a different direction after the incentive is offered, that may seem like good proof. But once again, it’s hard to know if it is the incentive itself that caused it (though this may be the one of the three where we might have the most confidence).

So should we junk the idea of the states as laboratories? No! And here’s why: for many types of policy, this is the best evidence we will get. For most types of public policy, we will never get truly randomized experiments. So it’s either use these inferior types of evidence, or just use our big brains to guess what is right.

There are also better and worse ways to use the states as laboratories. For example, looking at differences around state borders can be a good approach. There are often metro areas right on state borders (borders are often rivers and historically a lot of cities were located on rivers), with a commuting area that can spread over 2-3 states. Plus, in the case of a pandemic virus, it will hit the entire metro area at once. Viruses have no idea what a state border is.

One good line of research is by Goolsbee and Syverson. They look at whether state shelter-in-place orders reduced movement of people, or if people reduced their movement even without a shelter-in-place order. And they look at commuting areas at state borders. They find some effect of shelter-in-place orders, but most of the effect is from people just deciding to stay home. Even on the side of the border without the SIP order, mobility dramatically declined.

Ultimately though, what can we learn from this shelter-in-place “experiment.” On the one hand, it appears that this policy can’t be responsible for most of the bad economic effects. It was people’s voluntary decisions. On the other hand, this policy also can’t be praised much for reducing any spread of the virus. Again, it was people making choices. So even when we have pretty good evidence from these experiments, it’s often unclear what to do with it.

There have been a number of policies where looking at different near state borders can help us to answer some important questions. For example, do higher taxes deter business activity? Rohlin, Rosenthal, and Ross provide evidence that “state-level tax policies do affect the location decisions of entrepreneurs and new business activity.” I have a paper in progress with Alicia Plemmons which also uses this approach to study state-level taxes, and she has used it to study the effects of occupational licensing too. Right-to-work laws and minimum wage laws have also been studied by looking at state borders. Scott Cunningham’s book discusses more advanced techniques for doing these kinds of comparisons.

Does the state border approach save the “laboratories of democracy” argument? Does it overcome all the problems? It certainly helps. It’s certainly better than just naively comparing states. Remember again that if we don’t compare states in some way, we don’t really have much else to go on in terms of empirical evidence. Cross-country evidence is even trickier, since so many other factors differ and may not be controlled for.

So let’s give two cheers for federalism! But for the third cheer, let’s make sure we are actually making comparisons that make sense, and aren’t being fooled by common statistical errors like reverse causality, omitted variable bias, or survivorship bias.