IPUMS – Economist Writing Every Day

Triumph of the Data Hoarders 2: The Institutions

October 9, 2025James BaileyLeave a comment

Datasets can be pulled offline for all sorts of reasons. As I wrote in February, this shows the value of being a data hoarder– just downloading now any data you think you might want later:

Several major datasets produced by the federal government went offline this week…. This serves as a reminder of the value of redundancy- keeping datasets on multiple sites as well as in local storage. Because you never really know when one site will go down- whether due to ideological changes, mistakes, natural disasters, or key personnel moving on.

The US Federal government shutdown this month provides another reminder of this. So far most datasets are still up, but I’ve seen some availability issues:

The good news is that a number of institutions have stepped up in 2025 to host at-risk datasets (joining those like IPUMS, NBER, and Archive.org that have been hosting datasets for many years, but are scaling up to meet the moment):

Restore CDC hosts all CDC data as it was in January 2025.

Harvard Library’s project is less user-friendly but more powerful, archiving all 16 terabytes of Data.gov.

A map of archived USAID-funded education data.

The Data Rescue Project provides tools and suggestions for how other institutions can save data at scale, plus links to other projects.

The Data Foundation’s SAFE-Track project is aiming to document the effects of Federal data changes.

State Demographics 1962-2024

July 31, 2025July 30, 2025James BaileyLeave a comment

I provide a simple, clean panel dataset of the historical demographics of US states here.

I made this state-year level dataset from the individual-level responses in the Current Population Survey‘s Annual Social and Economic Supplement. It includes key demographic variables commonly used as controls in regressions: age, race, sex, marital status, income, education, health insurance.

This was no great feat, in that I’m sure hundreds of researchers have generated very similar datasets before- but as far as I can tell, none of them share it publicly like this. I hope that posting this will save the time of researchers who would otherwise spend a couple hours re-doing the same work themselves, and offer easy-to-use Stata and Excel files for students who might not be able to make one themselves.

I’ve often seen my own students start an empirical project where they have state-level data on their dependent and independent variables of interest, then realize they need some control variables and find that they are surprisingly hard to get. Census’ website is difficult to navigate and mostly offers its data one year at a time. IPUMS is great for getting individual-level responses; they have a tool that can export state-level data directly, but it is not built or advertised in a way that students end up actually using it for that, and it is limited to exporting 8000 cells of data at a time (the dataset I share has 53,856). The National Welfare Data is what I have been recommending to students for state-level controls, but its year coverage is more limited (1980-2023), and it contains welfare variables that are mostly extraneous for this. County-level data is available back to 1969, which could be aggregated to the state level, but it doesn’t have income or education.

Disclaimers: prior to 1977, some states are missing or are combined with nearby states (the data makes clear which ones). Some variables change how they are coded by Census / IPUMS over time. Age in particular sees several big changes to its universe and its top-coding, and race gets recoded in 2003. Therefore, this is not always a good dataset for measuring national trends in a variable over time, but it should still work well for making comparisons across states in a given year. If you are using this data in a regression, I strongly recommend controlling for year fixed effects to mitigate this issue. If you want to double-check my cleaning code, it is available here.

Once again, the state demographic data is available here. If you think there is a better source out there for historical state demographics, let us know below. See my data page for more cleaned-up versions of public datasets.

Triumph of the Data Hoarders

February 6, 2025James Bailey3 Comments

Several major datasets produced by the federal government went offline this week. Some, like the Behavioral Risk Factor Surveillance Survey and the American Community Survey, are now back online; probably most others will soon join them. But some datasets that the current administration considers too DEI-inflected could stay down indefinitely.

This serves as a reminder of the value of redundancy- keeping datasets on multiple sites as well as in local storage. Because you never really know when one site will go down- whether due to ideological changes, mistakes, natural disasters, or key personnel moving on.

External hard drives are an affordable option for anyone who wants to build up their own local data hoard going forward. The Open Science Foundation site allows you to upload datasets up to 50 GB to share publicly; that’s how I’ve been sharing cleaned-up versions of the BRFSS, state-levle NSDUH, National Health Expenditure Accounts, Statistics of US Business, and more. If you have a dataset that isn’t online anywhere, or one that you’ve cleaned or improved to the point it is better than the versions currently online, I encourage you to post it on OSF.

If you are currently looking for a federal dataset that got taken down, some good places to check are IPUMS, NBER, Archive.org, or my data page. PolicyMap has posted some of the federal datasets that seem particularly likely to stay down; if you know of other pages hosting federal datasets that have been taken down, please share them in the comments.

IPUMS Data Intensive Workshop & Conference

August 2, 2024August 2, 2024Zachary BartschLeave a comment

I just returned from the Full Count IPUMS data workshop at the Data-Intensive Research Conference that was hosted by the Network on Data Intensive Research on Aging and IPUMS. The theme of this conference was “Linking Records”.

It was the best workshop and conference that I’ve ever attended. I’d attended the conference remotely in the past. But attending the workshop was exceptional. Myself and about 20 other people were flown to the Minneapolis Population Center and put up in a hotel during our stay (that made the conference a low-stress affair). The whole workshop was well organized, the speakers built on one another’s content, and there was a hands-on lab for us to complete. I felt my human capital growing by the hour.

Continue reading →

Almost Observable Human Capital

May 17, 2024May 17, 2024Zachary BartschLeave a comment

I’ve written about IPUMS before. It’s great. Among individual details are their occupations and industry of their occupation. That’s convenient because we can observe how technology spread across America by observing employment in those industries. We can also identify whether demographic subgroups differed or not by occupation. There’s plenty of ways to slice the data: sex, race, age, nativity, etc.

But what do we know about historical occupations and what they entailed? At first blush, we just have our intuition. But it turns out that we have more. There is a super boring 1949 report published by the Department of Labor called the “Dictionary of Occupational Titles”. The title says it all. But, the DOL published another report in 1956 that’s conceptually more interesting called “Estimates of Worker Trait Requirements for 4,000 Jobs as Defined in the Dictionary of Occupational Titles: An Alphabetical Index”. The report lists thousands of occupations and identifies typical worker aptitudes, worker temperaments, worker interests, worker physical capacities, and working conditions. Below is a sample of the how the table is organized:

Continue reading →

Most Improved Data

December 29, 2022December 5, 2025James Bailey9 Comments

The US government is great at collecting data, but not so good at sharing it in easy-to-use ways. When people try to access these datasets they either get discouraged and give up, or spend hours getting the data into a usable form. One of the crazy things about this is all the duplicated effort- hundreds of people might end up spending hours cleaning the data in mostly the same way. Ideally the government would just post a better version of the data on their official page. But barring that, researchers and other “data heroes” can provide a huge public service by publicly posting datasets that they have already cleaned up- and some have done so.

I just added a data page to my website that highlights some of these “most improved datasets”:

the IPUMS versions of the American Community Survey, Current Population Survey, and Medical Expenditure Panel Survey
The County Business Patterns Database, harmonized by Fabian Eckert, Teresa C. Fort, Peter K. Schott, and Natalie J. Yang
Code for accessing the Quarterly Census of Employment and Wages by Gabriel Chodorow-Reich
The merged Statistics of US Business, my own attempt to contribute

I hope to keep adding to this page as I find other good sources of unofficial/improved data, and as I create them (one of my post-tenure goals). See the page for more detail on these datasets, and comment here if you know of existing improved datasets worth adding, or if you know of needlessly terrible datasets you think someone should clean up.