Author: Joy Buchanan

The re-Americanization of the American Economic Association Meetings

Joy Buchanan1 Comment

According to a chart circulating this week, attendance at the AEA winter meeting exceeded 13,000 economists at the peak. I had some job interviews at Chicago in 2017, and I spoke on a panel at San Diego in 2020. Chicago was freezing cold. San Diego was lovely in January. Cold or not, probably every economist on the job market showed up in person to those conferences. There are harrowing stories of people who missed job interviews because of snow and cancelled flights.

In 2020 (just before Covid) I wrote:

A privilege of being in this profession is the chance to meet people from around the world. At my lunch table currently there are people from Chile and Southern China to my right and Kansas and Hong Kong to my left. #ASSA2020

It is (or was) a huge win for the US to be the meeting place. Now that job interviews have moved to Zoom, attendance has fallen by more than half. I assume that the attendees here in 2024 skew more America-based.

A European economist predicts attendance in 2024 will fall even lower because “the interview stage of the market has moved online, and at a random date between October and January. No point in flying to ASSA, especially if you are a candidate based in Europe hoping for a job in Europe”

Instead of asking why the economics job market unraveled, maybe we should be surprised that it was ever so centralized. People really did travel from all over the world to an American city in January to interview for almost every serious job available in the economics profession.

Small points:

  • I’ve seen several people complain that it’s a bad time of year. For me, it’s a good time of year. Since I do a lot of teaching, it’s good to have a conference that happens before my teaching starts.
  • I was happy to see that childcare is available here (for a fee).
  • I talked to a job market candidate who was trying to weigh a tenure-track job offer from a teaching school against the possible offer of a postdoc. He would prefer the postdoc. Because of the new system, job offers come in sequence. He risks losing the teaching offer before he finds out if he will get the postdoc offer in writing. In the old days, candidates would have been more likely to know what their best option was going to be.
  • The online JOE job board still serves for coordination. It is still administered by the AEA for jobs globally.
  • James provided evidence that the AEA will start letting more people get on the program to present research. (By contrast, the Southern Economic Association meeting has actually increased in attendance since Covid. They allow accept more people to present research on the program.)
  • When my grad school research group went out for lunch, I was sometimes the only American at the table. I loved it. I’m grateful that those brilliant people came here to start their careers.

Updated List of Top Blogs for 2023

Joy Buchanan1 Comment

Back in August I listed the most-read posts of 2023. Here I will finish out the year by listing a few more highlights. This has been another big year for our website.

From Jeremy in October: Kids Are Much Less Likely to Be Killed by Cars Than in the Past This is good news (goods new is underreported). Jeremy also dives into the trade off between safety and letting kids roam free.

This connects with Jeremy’s post from March: The Leading Causes of Death Among Elementary-Age Children Happy New Year, everyone. We are glad you made it through childhood to share this next year with us. Tyler says he is “thankful for being able to watch and to try to understand the procession of history before us”. Agreed.

Thanksgiving 2023 is the Second Cheapest Ever (Relative to Earnings) Jeremy didn’t get a ton of blog traffic from this (good news is under clicked), but the White House Press Secretary picked it up. Jeremy also contributed a helpful table to the raging debate over How the Economy is Doing vs. How People Think the Economy is Doing.

Zachary has been giving out good advice for economics teachers, backed up by his data. All professors can read: 5 Easy Steps to Improve Your Course Evals. Econ professors check: Update on Game Theory Teaching. It’s about how to teach Game Theory, but I also see it as a testament to how much a course can improve if you allow a teacher to iterate multiple times at the same school. Administrators, take note.

What We Are Learning about Paper Books is jointly my reflection on AI generative books and a review of Tyler Cowen’s new book GOAT. I’m a techno-optimist, but I think there is value in an old-fashioned paper book, mostly from a behavioral or neuro perspective.

Warren Buffett’s Secret Sauce: Investing the Insurance “Float”

Where Can You Still Buy an Affordable Home in the US?

James on investing. Historical Price to Earnings Ratios By Industry

Two from Mike: Stop and Frisk was an Unmitigated Disaster and On minimum wages and the devil’s discount

What if you could get your phone or tablet to read Kindle or other text aloud to you? Search engine traffic has landed tens of thousands of visitors on this practical post by Scott: How to Read Aloud Kindle and Other Text on iPhone, iPad, and Android

ChatGPT on Advent

Joy Buchanan2 Comments

I have a paper that emphasizes ChatGPT errors. It is important to recognize that LLMs can make mistakes. However, someone could look at our data and emphasize the opposite potential interpretation. On many points, and even when coming up with citations, the LLM generated correct sentences. More than half of the content was good.

You can read ChatGPT’s take on a wide variety of topics within economics, in the appendix of our paper. The journal hosts it at https://journals.sagepub.com/doi/suppl/10.1177/05694345231218454/suppl_file/sj-pdf-1-aex-10.1177_05694345231218454.pdf If that link does not work then the appendix has been up on SSRN since June in the form of the old version of the paper.

Apparently, LLMs just solved an unsolvable math problem. Is there anything they can’t do? Considering how much of human expression and culture revolves around religion, we can expect AI’s to get involved in that aspect of life.

Alex thinks it will be a short hop from Personal Jesus Chatbot to a whole new AI religion. We’ll see. People have had “LLMs” in the form of human pastors, shaman, or rabbis for a long time, and yet sticking to one sacred text for reference has been stable. I think people might feel the same way in the AI era – stick to the canon for a common point of reference. Text written before the AI era will be considered special for a long time, I predict. Even AI’s ought to be suspicious of AI-generated content, just in the way that humans are now (or are they?).

Many religious traditions have lots of training literature. (In our ChatGPT errors paper, we expect LLMs to produce reliable content on topics for which there is plentiful training literature.)

I gave ChatGPT this prompt:

Can you write a Bible study? I’d like this to be appropriate for the season of Advent, but I’d like most of the Bible readings to be from the book of Job. I’d like to consider what Job was going through, because he was trying to understand the human condition and our relationship to God before the idea of Jesus. Job had a conception of the goodness of God, but he didn’t have the hope of the Gospel. Can you work with that?

Continue reading →

Chapman Economists Revise Forecast

Joy Buchanan1 Comment

Back in June, I watched the livestream of the Chapman Economic Forecast with Dr. Jim Doti (who was president when I was a student at Chapman). Typically, this is a valuable informative event, and the team has an excellent record of performance. They have often outdone other forecasters in predicting the future.

That is why I feel a little bad for making this post in the summer and tweeting out Doti’s prediction that we would have a recession by now.

To be fair to Doti, there has been a lot of uproar over this issue. Lots of people thought the economy would be bad. And lots of people feel like the economy is bad (the “vibecession”) even though it is objectively not. Many tweets have gone by about it.

Doti opened by saying his prediction had turned out to be wrong. He had an explanation for it (pictured below). You can watch it free here (recorded on Dec 14).

Doti said that he had expected a large fiscal stimulus in the form of deficit spending, however he had not expected the deficit to be so large. Debt-financed spending propped up an economy that was otherwise poised to contract. At least, that is a plausible story.

Looking forward, Doti does not predict a recession next year, but he does predict weak growth and possibly one quarter of GPD decline (not two).

The next part of talk was about the long-term consequences of deficit spending. Nothing is free. TANSTAAFL

In addition to vibecession, anyone following economics in 2023 needs to know what a “soft landing” is.

Do People Trust ChatGPT Writing?

Joy Buchanan1 Comment

My new working paper with Will Hickman is up on SSRN: Do People Trust Humans More Than ChatGPT?

We study whether people will pay for a fact-check on AI writing. ChatGPT can be very useful, but human readers should not trust every fact that it reports. Yesterday’s post was about ChatGPT writing false things that look real.

The reason participants in our experiment might pay for a fact-check is that they earn bonus payments based on whether they correctly identify errors in a paragraph. If participants believe that the paragraph does not contain any errors, they should not pay for a fact-check. However, if they have doubts, it is rational to pay for a fact-check and earn a smaller bonus, for certain.

Abstract: We explore whether people trust the accuracy of statements produced by large language models (LLMs) versus those written by humans. While LLMs have showcased impressive capabilities in generating text, concerns have been raised regarding the potential for misinformation, bias, or false responses. In this experiment, participants rate the accuracy of statements under different information conditions. Participants who are not explicitly informed of authorship tend to trust statements they believe are human-written more than those attributed to ChatGPT. However, when informed about authorship, participants show equal skepticism towards both human and AI writers. There is an increase in the rate of costly fact-checking by participants who are explicitly informed. These outcomes suggest that trust in AI-generated content is context-dependent.

Our original hypothesis was that people would be more trusting of human writers. That turned out to be only partially true. Participants who are not explicitly informed of authorship tend to trust statements they believe are human-written more than those attributed to ChatGPT.

We presented information to participants in different ways. Sometimes we explicitly told them about authorship (informed treatment) and sometimes we asked them to guess about authorship (uninformed treatment).

This graph (figure 5 in our paper) shows that the overall rate of fact-checking increased when subjects were given more explicit information. Something about being told that a paragraph was written by a human might have aroused suspicion in our participants. (The kids today would say it is “sus.”) They became less confident in their own ability to rate accuracy and therefore more willing to pay for a fact-check. This effect is independent of whether participants trust humans more than AI.

We are thinking of fact-checking as often a good thing, in the context of our previous work on ChatGPT hallucinations. So, one policy implication is that certain types of labels can cause readers to think critically. For example, Twitter labels automated accounts so that readers know when content has been chosen or created by a bot.

Our working paper is currently trending on SSRN top ten lists such as this one.

Suggested Citation:
Buchanan, Joy and Hickman, William, Do People Trust Humans More Than ChatGPT? (November 16, 2023). GMU Working Paper in Economics No. 23-38, Available at SSRN: https://ssrn.com/abstract=4635674

GPT-4 Generates Fake Citations

Joy Buchanan2 Comments

I am happy to share my latest publication at The American Economist: ChatGPT Hallucinates Non-existent Citations: Evidence from Economics

Citation: Buchanan, J., Hill, S., & Shapoval, O. (2024). ChatGPT Hallucinates Non-existent Citations: Evidence from Economics. The American Economist. 69(1), 80-87  https://doi.org/10.1177/05694345231218454

Blog followers will know that we reported this issue earlier with the free version of ChatGPT using GPT-3.5 (covered in the WSJ). We have updated this new article by running the same prompts through the paid version using GPT-4. Did the problems go away with the more powerful LLM?

The error rate went down slightly, but our two main results held up. It’s important that any fake citations at all are being presented as real. The proportion of nonexistent citations was over 30% with GPT-3.5, and it is over 20% with our trial of GPT-4 several months later. See figure 2 from our paper below for the average accuracy rates. The proportion of real citations is always under 90%. GPT-4, when asked about a very specific narrow topic, hallucinates almost half of the citations (57% are real for level 3, as shown in the graph).

The second result from our study is that the error rate of the LLM increases significantly when the prompt is more specific. If you ask GPT-4 about a niche topic for which there is less training data, then a higher proportion of the citations it produces are false. (This has been replicated in different domains, such as knowledge of geography.)

What does Joy Buchanan really think?: I expect that this problem with the fake citations will be solved quickly. It’s very brazen. When people understand this problem, they are shocked. Just… fake citations? Like… it printed out reference for papers that do not actually exist? Yes, it really did that. We were the only ones who quantified and reported it, but the phenomenon was noticed by millions of researchers around the world who experimented with ChatGPT in 2023. These errors are so easy to catch that I expect ChatGPT will clean up its own mess on this particular issue quickly. However, that does not mean that the more general issue of hallucinations is going away.

Not only can ChatGPT make mistakes, as any human worker can mess up, but it can make a different kind of mistake without meaning to. Hallucinations are not intentional lies (which is not to say that an LLM cannot lie). This paper will serve as bright clear evidence that GPT can hallucinate in ways that detract from the quality of the output or even pose safety concerns in some use cases. This generalizes far beyond academic citations. The error rate might decrease to the point where hallucinations are less of a problem than the errors that humans are prone to make; however, the errors made by LLMs will always be of a different quality than the errors made by a human. A human research assistant would not cite nonexistent citations. LLM doctors are going to make a type of mistake that would not be made by human doctors. We should be on the lookout for those mistakes.

ChatGPT is great for some of the inputs to research, but it is not as helpful for original scientific writing. As prolific writer Noah Smith says, “I still can’t use ChatGPT for writing, even with GPT-4, because the risk of inserting even a small number of fake facts… “

Follow-Up Research: Will Hickman and I have an incentivized experiment on trust that you can read on SSRN: Do People Trust Humans More Than ChatGPT?

@IMurtazashvili has pointed me to a great resource for AI-era literature review work. “AI-Based Literature Review Tools” from Texas A&M

Joy’s Fashion Globalization Article with Cato

Joy Buchanan5 Comments

I am published by Cato this week:

Fast Fashion, Global Trade, and Sustainable Abundance

This is part of a 10-part series called “Defending Globalization: Society and Culture”

Imagine trying to explain the world today to a person who time traveled forward from 300 years ago. How could someone who lived in France in the year 1600 understand our modern problems?

Person from the Past: So, how is it with 8 billion people?

Me Today: It’s bad. We have too many clothes.

PftP: Right. With 8 billion you wouldn’t have enough clothes for everyone.

MT: Too many.

PftP: Not enough?

MT: I said we have TOO MANY clothes. Not even the poorest people in the world want them. Shirts pile up on the beaches and pollute the ocean.

PftP: …

My article highlights the fact that we live in an era of unprecedented clothing abundance. First, that was not always true.

Most of human history has been characterized by privation and low‐​productivity toil. As one American sharecropper exclaimed in John Steinbeck’s Depression‐​era novel The Grapes of Wrath, “We got no clothes, torn an’ ragged. If all the neighbors weren’t the same, we’d be ashamed to go to meeting.”

https://www.cato.org/publications/globalization-fashion

Secondly, not everyone is celebrating.

The United Nations Economic Commission for Europe called the fashion industry an “environmental and social emergency” because clothing production has roughly doubled since the year 2000. Their main concerns are fast fashion’s environmental impact and working conditions. 

Some of my article is a response to the critics of modern low-cost mass production.

Thirdly, I explain how we could keep most of the benefits of cheap clothes with less litter in the environment. The item I am most optimistic about is using our new artificial intelligence tools to re-sort the world’s junk. We would produce and throw away fewer clothes if we had a better system for rearranging the stock of goods that we already have. The problem I see today is that I have “perfectly good” clothes in my house that I don’t really want; however, attention and time are so scarce that no one will pay me for them. Even if I donate them, I worry that half will end up in the trash. Someone on this earth could use them but identifying that someone and making the trade still has high prohibitively high transaction costs. Very smart AI could come to my house and scan my stuff and pay me for it because very smart AI could get it to someone with a positive value for it.

If you’d like to see a trail of blogs that I wrote while in the research phase for this article, use https://economistwritingeveryday.com/?s=fashion

Lastly, we thank Tyler for the Marginal Revolution link.

Thanksgiving Links

Joy BuchananLeave a comment

Asked for its methodology, the White House pointed us to a Nov. 15 blog post by Jeremy Horpedahl, an associate economics professor at the University of Central Arkansas.

https://www.politifact.com/factchecks/2023/nov/21/karine-jean-pierre/fact-checking-karine-jean-pierre-on-this-years-tha

“When people ask what Civilization VI’s “cultural victory” condition would look like in the real world, this is it. Write it in the manual.”

And new from Cato: Food Globalization Puts the World on Your Plate

The current alliance to counter digital piracy

Joy BuchananLeave a comment

U.S. Immigration and Customs Enforcement’s (ICE) Homeland Security Investigations (HSI) has joined forces with the Motion Picture Association (MPA) to launch a new initiative aimed at countering digital piracy and protecting a vital sector of the U.S economy.

The initiative is called Operation Intangibles.

Digital streaming services provide quick and easy access to creative works, such as music, television, movies. However, the growth of digital streaming services has presented new challenges when it comes to law enforcement’s ability to ensure vital copyright protection for the industry. This technology that has provided millions of people access to their favorite shows, has also enabled criminals to turn piracy into a crime that is no longer restricted to the hand-to-hand sale of illegally pirated media.

Digital piracy negatively impacts millions of jobs, results in less taxes being paid, and threatens innovation and creativity. Its effects are felt across multiple industries and includes the cost of corollary crimes on consumers such as the potential damage caused by hidden or embedded malware, as well as identity theft and financial crimes, such as credit card fraud.

There exists a National Intellectual Property Rights Coordination Center.

Recall, Napster was shut down in 2001. Limewire was shut down in 2010. Illicit downloading is happening through other channels.

According to this graph, people spent almost as much on vinyl records as they did on CDs in 2018. The Economist provided a nice chart (last updated in 2019) on the rise of streaming revenue and the collapse of traditional records sales.

Axios Survey of Americans on AI Regulation

Joy BuchananLeave a comment

Axios just surveyed over 2,000 U.S. adults to find that “Americans rank the importance of regulating AI below government shutdowns, health care, gun reform…” Without pressure from the public to pass new legislation, Congress might do nothing for now, which will lead to the rapid advance of LLM chatbots into life and work.

The participants seem more worried about AI taking jobs than they are excited about AI making life better. There is some concern about misinformation.** So, they don’t think AI will have no impact on society, but they also don’t see enacting regulation as a top priority for government.

At my university, the policy realm I know best, we will probably not be “regulating” AI. We have had task forces talking about it, so it’s not because no one has considered it.

The Axios poll revealed gender gaps in attitudes toward AI. Women said they would be less permissive about kids using AI than men. Also, “Parents in urban areas were far more open to their children using AI than parents in the suburbs or rural areas.” Despite those gaps in what people say, I expect that the gaps in what their children are currently doing with technology are smaller. Experimental economists are suspicious of self-reported data.

**Our results did not change much when we ran the exact same prompts through GPT-4. A version of my paper on AI errors that I blogged about before is up on SSRN, but a new manuscript incorporating GPT-4 is under review: Buchanan, Joy, Stephen Hill, and Olga Shapoval (2023). “ChatGPT Hallucinates Nonexistent Citations: Evidence from Economics”. Working Paper.