Back in June, I watched the livestream of the Chapman Economic Forecast with Dr. Jim Doti (who was president when I was a student at Chapman). Typically, this is a valuable informative event, and the team has an excellent record of performance. They have often outdone other forecasters in predicting the future.
That is why I feel a little bad for making this post in the summer and tweeting out Doti’s prediction that we would have a recession by now.
To be fair to Doti, there has been a lot of uproar over this issue. Lots of people thought the economy would be bad. And lots of people feel like the economy is bad (the “vibecession”) even though it is objectively not. Many tweets have gone by about it.
Doti opened by saying his prediction had turned out to be wrong. He had an explanation for it (pictured below). You can watch it free here (recorded on Dec 14).
Doti said that he had expected a large fiscal stimulus in the form of deficit spending, however he had not expected the deficit to be so large. Debt-financed spending propped up an economy that was otherwise poised to contract. At least, that is a plausible story.
Looking forward, Doti does not predict a recession next year, but he does predict weak growth and possibly one quarter of GPD decline (not two).
The next part of talk was about the long-term consequences of deficit spending. Nothing is free. TANSTAAFL
In addition to vibecession, anyone following economics in 2023 needs to know what a “soft landing” is.
Fed Chair Powell: "There's little basis for thinking the economy is in a recession now."
The latest Fed projections show a soft landing in 2024.
Growth slowing, but positive +1.4% Inflation cooling to ~2.4% Unemployment rising to 4.1% (so just below recession trigger) pic.twitter.com/6cvgC6l5Nd
We study whether people will pay for a fact-check on AI writing. ChatGPT can be very useful, but human readers should not trust every fact that it reports. Yesterday’s post was about ChatGPT writing false things that look real.
The reason participants in our experiment might pay for a fact-check is that they earn bonus payments based on whether they correctly identify errors in a paragraph. If participants believe that the paragraph does not contain any errors, they should not pay for a fact-check. However, if they have doubts, it is rational to pay for a fact-check and earn a smaller bonus, for certain.
Abstract: We explore whether people trust the accuracy of statements produced by large language models (LLMs) versus those written by humans. While LLMs have showcased impressive capabilities in generating text, concerns have been raised regarding the potential for misinformation, bias, or false responses. In this experiment, participants rate the accuracy of statements under different information conditions. Participants who are not explicitly informed of authorship tend to trust statements they believe are human-written more than those attributed to ChatGPT. However, when informed about authorship, participants show equal skepticism towards both human and AI writers. There is an increase in the rate of costly fact-checking by participants who are explicitly informed. These outcomes suggest that trust in AI-generated content is context-dependent.
Our original hypothesis was that people would be more trusting of human writers. That turned out to be only partially true. Participants who are not explicitly informed of authorship tend to trust statements they believe are human-written more than those attributed to ChatGPT.
We presented information to participants in different ways. Sometimes we explicitly told them about authorship (informed treatment) and sometimes we asked them to guess about authorship (uninformed treatment).
This graph (figure 5 in our paper) shows that the overall rate of fact-checking increased when subjects were given more explicit information. Something about being told that a paragraph was written by a human might have aroused suspicion in our participants. (The kids today would say it is “sus.”) They became less confident in their own ability to rate accuracy and therefore more willing to pay for a fact-check. This effect is independent of whether participants trust humans more than AI.
We are thinking of fact-checking as often a good thing, in the context of our previous work on ChatGPT hallucinations. So, one policy implication is that certain types of labels can cause readers to think critically. For example, Twitter labels automated accounts so that readers know when content has been chosen or created by a bot.
Suggested Citation: Buchanan, Joy and Hickman, William, Do People Trust Humans More Than ChatGPT? (November 16, 2023). GMU Working Paper in Economics No. 23-38, Available at SSRN: https://ssrn.com/abstract=4635674
Citation: Buchanan, J., Hill, S., & Shapoval, O. (2024). ChatGPT Hallucinates Non-existent Citations: Evidence from Economics. The American Economist. 69(1), 80-87 https://doi.org/10.1177/05694345231218454
Blog followers will know that we reported this issue earlier with the free version of ChatGPT using GPT-3.5 (covered in the WSJ). We have updated this new article by running the same prompts through the paid version using GPT-4. Did the problems go away with the more powerful LLM?
The error rate went down slightly, but our two main results held up. It’s important that any fake citations at all are being presented as real. The proportion of nonexistent citations was over 30% with GPT-3.5, and it is over 20% with our trial of GPT-4 several months later. See figure 2 from our paper below for the average accuracy rates. The proportion of real citations is always under 90%. GPT-4, when asked about a very specific narrow topic, hallucinates almost half of the citations (57% are real for level 3, as shown in the graph).
The second result from our study is that the error rate of the LLM increases significantly when the prompt is more specific. If you ask GPT-4 about a niche topic for which there is less training data, then a higher proportion of the citations it produces are false. (This has been replicated in different domains, such as knowledge of geography.)
What does Joy Buchanan really think?: I expect that this problem with the fake citations will be solved quickly. It’s very brazen. When people understand this problem, they are shocked. Just… fake citations? Like… it printed out reference for papers that do not actually exist? Yes, it really did that. We were the only ones who quantified and reported it, but the phenomenon was noticed by millions of researchers around the world who experimented with ChatGPT in 2023. These errors are so easy to catch that I expect ChatGPT will clean up its own mess on this particular issue quickly. However, that does not mean that the more general issue of hallucinations is going away.
Not only can ChatGPT make mistakes, as any human worker can mess up, but it can make a different kind of mistake without meaning to. Hallucinations are not intentional lies (which is not to say that an LLM cannot lie). This paper will serve as bright clear evidence that GPT can hallucinate in ways that detract from the quality of the output or even pose safety concerns in some use cases. This generalizes far beyond academic citations. The error rate might decrease to the point where hallucinations are less of a problem than the errors that humans are prone to make; however, the errors made by LLMs will always be of a different quality than the errors made by a human. A human research assistant would not cite nonexistent citations. LLM doctors are going to make a type of mistake that would not be made by human doctors. We should be on the lookout for those mistakes.
ChatGPT is great for some of the inputs to research, but it is not as helpful for original scientific writing. As prolific writer Noah Smith says, “I still can’t use ChatGPT for writing, even with GPT-4, because the risk of inserting even a small number of fake facts… “
I still can't use ChatGPT for writing, even with GPT-4, because the risk of inserting even a small number of fake facts or bad interpretations into a blog post is unacceptable, meaning that it requires so much time to fact-check that it doesn't save effort.
Imagine trying to explain the world today to a person who time traveled forward from 300 years ago. How could someone who lived in France in the year 1600 understand our modern problems?
Person from the Past: So, how is it with 8 billion people?
Me Today: It’s bad. We have too many clothes.
PftP: Right. With 8 billion you wouldn’t have enough clothes for everyone.
MT: Too many.
PftP: Not enough?
MT: I said we have TOO MANY clothes. Not even the poorest people in the world want them. Shirts pile up on the beaches and pollute the ocean.
PftP: …
My article highlights the fact that we live in an era of unprecedented clothing abundance. First, that was not always true.
Most of human history has been characterized by privation and low‐productivity toil. As one American sharecropper exclaimed in John Steinbeck’s Depression‐era novel The Grapes of Wrath, “We got no clothes, torn an’ ragged. If all the neighbors weren’t the same, we’d be ashamed to go to meeting.”
The United Nations Economic Commission for Europe called the fashion industry an “environmental and social emergency” because clothing production has roughly doubled since the year 2000. Their main concerns are fast fashion’s environmental impact and working conditions.
Some of my article is a response to the critics of modern low-cost mass production.
Thirdly, I explain how we could keep most of the benefits of cheap clothes with less litter in the environment. The item I am most optimistic about is using our new artificial intelligence tools to re-sort the world’s junk. We would produce and throw away fewer clothes if we had a better system for rearranging the stock of goods that we already have. The problem I see today is that I have “perfectly good” clothes in my house that I don’t really want; however, attention and time are so scarce that no one will pay me for them. Even if I donate them, I worry that half will end up in the trash. Someone on this earth could use them but identifying that someone and making the trade still has high prohibitively high transaction costs. Very smart AI could come to my house and scan my stuff and pay me for it because very smart AI could get it to someone with a positive value for it.
Asked for its methodology, the White House pointed us to a Nov. 15 blog post by Jeremy Horpedahl, an associate economics professor at the University of Central Arkansas.
“When people ask what Civilization VI’s “cultural victory” condition would look like in the real world, this is it. Write it in the manual.”
When people ask what Civilization VI’s “cultural victory” condition would look like in the real world, this is it. Write it in the manual. https://t.co/i88RmJF7qr
U.S. Immigration and Customs Enforcement’s (ICE) Homeland Security Investigations (HSI) has joined forces with the Motion Picture Association (MPA) to launch a new initiative aimed at countering digital piracy and protecting a vital sector of the U.S economy.
Digital streaming services provide quick and easy access to creative works, such as music, television, movies. However, the growth of digital streaming services has presented new challenges when it comes to law enforcement’s ability to ensure vital copyright protection for the industry. This technology that has provided millions of people access to their favorite shows, has also enabled criminals to turn piracy into a crime that is no longer restricted to the hand-to-hand sale of illegally pirated media.
Digital piracy negatively impacts millions of jobs, results in less taxes being paid, and threatens innovation and creativity. Its effects are felt across multiple industries and includes the cost of corollary crimes on consumers such as the potential damage caused by hidden or embedded malware, as well as identity theft and financial crimes, such as credit card fraud.
Recall, Napster was shut down in 2001. Limewire was shut down in 2010. Illicit downloading is happening through other channels.
According to this graph, people spent almost as much on vinyl records as they did on CDs in 2018. The Economist provided a nice chart (last updated in 2019) on the rise of streaming revenue and the collapse of traditional records sales.
Axios just surveyed over 2,000 U.S. adults to find that “Americans rank the importance of regulating AI below government shutdowns, health care, gun reform…” Without pressure from the public to pass new legislation, Congress might do nothing for now, which will lead to the rapid advance of LLM chatbots into life and work.
The participants seem more worried about AI taking jobs than they are excited about AI making life better. There is some concern about misinformation.** So, they don’t think AI will have no impact on society, but they also don’t see enacting regulation as a top priority for government.
At my university, the policy realm I know best, we will probably not be “regulating” AI. We have had task forces talking about it, so it’s not because no one has considered it.
The Axios poll revealed gender gaps in attitudes toward AI. Women said they would be less permissive about kids using AI than men. Also, “Parents in urban areas were far more open to their children using AI than parents in the suburbs or rural areas.” Despite those gaps in what people say, I expect that the gaps in what their children are currently doing with technology are smaller. Experimental economists are suspicious of self-reported data.
**Our results did not change much when we ran the exact same prompts through GPT-4. A version of my paper on AI errors that I blogged about before is up on SSRN, but a new manuscript incorporating GPT-4 is under review: Buchanan, Joy, Stephen Hill, and Olga Shapoval (2023). “ChatGPT Hallucinates Nonexistent Citations: Evidence from Economics”. Working Paper.
This is the 4th year in a row that the crew has put together some recommendations on products or books that we actually use, for your consideration in holiday gift buying. I’m going to put things into categories of Stuff for Adults, Kid’s Toys, Books for Adults, and Kid’s Books
Stuff for Adults (Men can be hard to shop for, so this might save Christmas!)
Scott says these scissors are amazing: “Fiskars 9 Inch Serrated Titanium Nitride Shop Shears”, available from Amazon here. Unlike some thick, heavy, or stubby heavy-duty shears, these have the feel of regular scissors, with fairly long, narrow blades. The handles are fairly substantial, and very comfortably contoured to the hand/thumb. The real magic is in the blades. They are sharp, with a very hard titanium nitride coating. Also, they have fine serrations in the cutting edge, that tend to grip the material in place as you are cutting.
Zachary recommends 5 things that he really uses at home
Food makes great presents for adults. Just give me Doritos, thanks.
Kid’s Toys
A wonderful game that you might not already have is Protect the Penguin. It’s high-energy but much less work than something like Twister. I actually enjoy playing it, too.
I’ll re-up from last year that Spot It is incredible and fits in your coat pocket. Fun for all ages. Several versions of the game. Not everyone has to know how to read or add to play so good for events with lots of ages represented. LEGO sets are always fun. If you keep the difficulty level age-appropriate than your kid should be able to play independently for an hour. I’ll put up one link but of course there are many thousands to choose from that can be tailored to any interest. My son likes Minecraft-themed sets.
Books for Adults
If you want to see all the books we have read and reviewed, just click on the Books category or go to
Not all of those posts are going to give you quick gift ideas, so if I had to pick out one from the last year it would be:
Secondhand: Travels in the New Global Garage Sale is a great book in my summer stack on fast fashion. I have always been interested in the combination problem-blessing of too much stuff. Adam Minter explains perfectly what many of us have been curious about.
You might have a relative who is very environmentally conscious or works hard to reuse and recycle. They might think it’s interesting to learn about the secondhand markets in America and beyond.
Kid’s Books
I’m reading The Hobbit aloud to my son right now and I highly recommend the experience. It’s going to take us months to get through it by reading a few pages at a time around bedtime.
Camp Out!: A Graphix Chapters Book (Bug Scouts #2) Funny graphic novel series about a group of friends in a scout troop. Probably especially fun for a kid who is in some kind of scouts program. Calvin and Hobbes is another comic series that my kid genuinely looks forward to reading.
Joke books can lead to great conversations. If the kid wants to know why something is funny, you can end up talking for a long time about the complex world.
See what we recommended in previous years. We have always had a mix of kid and adult items.
I suppose books were, for a time, the cheapest way to convey bits of information, before anyone had heard of “bits”. I asked ChatGPT to do my boring work and rehearse the history of the term “bits” so I could get my facts straight for this post.
Based on my other research, I was not confident that this paper called “The Binary System” exists. I could find no evidence of it from 5 minutes of googling. When I asked ChatGPT about it, ChatGPT apologized and said it probably isn’t real.
That kind of error would have been less likely with paper books that had human editors and authors. We hardly thought about this benefit of the old system, until it was gone. Because books had some cost to write and print, it made economic sense to employ an editor to ensure quality. For a while, no one would have thought of spewing out this ocean of associated terms that we get from ChatGPT, because it was too expensive. So, the first underappreciated benefit of printed books was that they were relatively more accurate.
This is the first “generative book” that I had ever accessed. I did the worst thing, right away. I asked the Chatbot to give away the ending, and it did.
I wish I could have bought the paper book and read until the end, using the suspense as a device to get me to learn new details about famous economists. Books and movies used to be able to use suspense to keep the audience engaged. Before generative books, that just seemed like the only way.
I didn’t think of books as special until I used a phone to (try to) learn. Now I put more value on the hours I spent reading paper books when I was younger. Authors were manipulating me through that rigid medium. I was forced to wade through pages and pages to get to the point. But what value is the “point” without the context? Getting to the point through arguments and examples, instead of just seeing a tweet, made us smarter. The second benefit of books was that we were forced to work harder.
Generative books are a further step toward poastmodernism.
A middle ground I can imagine is asking the chatbot to play coach instead of search engine. What if the Chatbot could write you a shorter version of the book that cuts out the parts you would have skimmed in a paper book? Still, just knowing that you were skimming certain parts actually created context. I took that for granted, until now.
Something has happened in the world outside of our screens that feels both “full circle” and “anti-Network State”. The BBC announced that an American tech billionaire bought the English pub where J.R.R. Tolkien and C.S. Lewis famously met to talk with a group of friends.
We could just see pictures of that place on the internet. You can see my 2014 pilgrimage to it pictured below. But people want to go in person, if they can afford it. That used to just be the only way to do things. Now that we can see what it is like to live through digital media, we are discovering the value that existed all along in 3-D doings.
Speaking of Tolkien, I’m reading The Hobbit aloud to my son right now and I highly recommend the experience. I’m glad that he is capable of enjoying it, since the pacing is so different from the media and games on screens. To get an elementary-aged boy reading from paper, I also recommend Calvin and Hobbes.
We like to put up recommendations for holiday gifting at this time of year. That paperback of Calvin and Hobbes would make a great gift for a kid (or adult!).
Zachary told us that he has printed out our own blogs for the week to read at home on paper. That’s “full circle.” Maybe print has something for us, even if it is more expensive than scrolling screens.
Buying new hardcover books at a bookshop regularly can be prohibitively expensive. But if you look at used books, you can easily walk out of a rummage sale with 10 quality books for $10.
Also, sometimes ordering older print paperback books or DVDs on Amazon is extremely cheap. Here are some of the cheap Amazon media purchases I made in 2023. I might have found them even cheaper at a local garage sale, but getting it delivered to my house saves a lot of time. I recommend all of these items over scrolling mobile screens.
Some people might prefer Kindle to paperback because it allows them to maintain a larger “library”. If you have very limited space to store books, then I can understand that advantage. However, I know that I learn better from paper. Printed media might still be worth paying for.
Afterword: I printed out two chapters of Tyler’s new book and it’s excellent.
It’s mostly very smart and serious, but this paragraph made me laugh out loud.
Roger Forsgren is a retired Chief Knowledge Officer of NASA. He was a director of Academy of Program, Project and Engineering Leadership at NASA. Roger is also an undergraduate with a liberal arts degree, a mechanical engineering degree… the author of … “When Graduation is Over, Learning Begins – Lessons for STEM Students and Professionals”.
In this episode, Swami & Roger discuss the importance of having a liberal arts background for an engineer, foundational skills needed for successful engineers, how communication skills, decision-making skills, and working with people are as important as number crunching, and where empathy can help achieve efficiency in an imperfect world filled with vulnerable people.
Forsgren does not see education as primarily a signaling exercise. Engineers need to know math and they learn much of it in school. Forsgren thinks of communication as a skill that can be learned, although I don’t think he would say that a traditional classroom is the only place to do that learning. Extra curricular activities probably play a role in developing social skills, and traditional school can be a good place to get that practice.
Forsgren: “I’d be wasting my time if I tried to train NASA engineers in calculus… they know it already… In the past it’s always been called soft skills [and that turns people off]… what we do is we change that to ‘professional skills’ and this is something I think is ideal for an engineer because an engineer has a choice in their career… [you can just do the analytics and math] but if you want to move ahead in your career, if you want to become a manager… you really have to develop the post-professional skills of leadership and communication…”
He has a section on former president Herbert Hoover. Forsgren said Herbert Hoover was the Elon Musk of his day. Hoover was extremely successful as an engineer, and even in some government positions relating to war and logistics. Then Hoover was a bad American president because of his poor communication skills. “He couldn’t lie… he had a hard time talking to people…”
Forsgren has a lot of patience and compassion for people who don’t have naturally good social skills. If you don’t think you are great with people, then I recommend this short blog on Lucidity (HT: Tyler) by Leber. Leber’s portrait of a husband and wife misunderstanding each other could apply to people on either side of the Israel-Palestinian conflict right now.
Economist Writing Every Day 