Weigh costs, benefits, and evidence quality

Living means making decisions with imperfect information. But Covid provides many examples of how people and institutions are often still bad at this. A few common errors:

  1. Imperfect evidence = perfect evidence. “Studies show Asprin prevents Covid”. OK, were the studies any good? Did any other studies find otherwise?
  2. Imperfect evidence = “no evidence” or “evidence against”. In early 2020, major institutions like the WHO said “masks don’t work” when they meant “there are no large randomized controlled trials on the effectiveness of masks”
  3. Imperfect evidence = don’t do it until you’re sure Inaction is a choice, and often a bad one. If the costs of action are low and the potential benefits of action high, you might want to do it anyway. Think masks in 2020 when the evidence for them was mediocre, or perhaps Vitamin D now.
  4. Imperfect evidence = do it, we have to do something Even in a pandemic, it is possible to over-react if the costs are high enough and/or the evidence of benefits bad enough (possibly lockdowns, definitely taking up smoking)

Any intro microeconomics class will explain the importance of weighing both costs and benefits. But how do we know what the costs and benefits are? For many everyday purchases they are usually obvious, but in other situations like medical treatments and public policies they aren’t, particularly the benefits. We have to estimate the benefits using evidence of varying quality. This creates more dimensions of tradeoffs- do you choose something with good evidence for its benefits, but high cost? Or something with worse evidence but lower costs? Graphing this properly should take at least 3 dimensions, but to keep things simple lets assume we know what the costs are, and combine benefits and evidence into a single axis called “good evidence of substantial benefit”. This yields a graph like:

Applied to Covid strategies, this yields a graph something like this:

This is not medical advice- I say this not merely as a legal disclaimer, but because my real point is the idea that we should weigh both evidence quality and costs, NOT that my estimates of the evidence quality or costs of particular strategies are better than yours

Judging the strength of the evidence for various strategies is inherently difficult, and might go beyond simply evaluating the strength of published research. But when evaluating empirical studies on Covid, my general outlook on the evidence is:

Of course, details matter, theory matters, the number of studies and how mixed their results are matters, potential fraud and bias matters, and there’s a lot it makes sense to do without seeing an academic study on it.

Dear reader, perhaps this is all obvious to you, and indeed the idea of adjusting your evidence threshold based on the cost of an intervention goes back at least to the beginnings of modern statistics in deciding how to brew Guinness. But common sense isn’t always so common, and this is my attempt to summarize it in a few pictures.

One thought on “Weigh costs, benefits, and evidence quality

  1. capitalistfirst October 7, 2021 / 11:07 pm

    Masks don’t work. The virus is airborne. Your N95 mask may block 95 percent of particles, viruses are smaller. You breath 100,000 virus cells in a minute. With the best of mask 5000 viruses still get through to infect.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s