Economist Writing Every Day

Ajeya Cotra writes the following in “Language models surprised us” (recommended, with more details on benchmarks)

In 2021, most people were systematically and severely underestimating progress in language models. After a big leap forward in 2022, it looks like ML experts improved in their predictions of benchmarks like MMLU and MATH — but many still failed to anticipate the qualitative milestones achieved by ChatGPT and then GPT-4, especially in reasoning and programming.

Joy’s thoughts: A possible reason for underestimating the rate of progress is not just a misunderstanding of the technology but a missed estimate on how much money would get poured in. When Americans want to buy progress, they can (see also SpaceX).

I compare this to the Manhattan project. People said it couldn’t be done, not because it was physically impossible but because it would be too expensive.

After a briefing regarding the Manhattan Project, Nobel Laureate Niels Bohr said to physicist Edward Teller, “I told you it couldn’t be done without turning the whole country into a factory.” (https://www.energy.gov/lm/articles/ohio-and-manhattan-project)

We are doing it again. We are turning the country into a factory for AI. Without all that investment, the progress wouldn’t be so fast.