Have you been told that economists only care about money? If anyone would tell Taylor Swift to focus on her own career, you might think it would be the most famous economist of all, Adam Smith. But AdamSmithWorks fans already know that Adam Smith was concerned with the whole person. So, would Adam Smith advise Taylor Swift to rush back to America after an exhausting concert just to cheer on a man in a football game?
I’m getting ready to give some public local talks about AI. Last week I shared some pictures that I think might help people understand ChatGPT, specifically:
My first thought is that GPT-4 was giving incorrect estimates of the heights of these mountains because it does not actually “know” the correct elevations. But then a nagging question came to mind.
GPT has a “creativity parameter.” Sometimes, it intentionally does not select the top-rated next word in a sentence, for example, in order to avoid being stiff and boring. Could GPT-4 know the exact elevation of these mountains, and it is just intentionally being “creative,” in this case?
I do not want to stand up in front of the local Rotary Club and say something wrong. So, I went to a true expert, Lenny Bogdonoff, to ask for help. Here is his reply:
Not quite. It’s not that it knows or doesn’t know, but based on the prompt, it’s likely unable to parse the specific details and is outputting results respectively. There is a component of stochastic behavior based on what part of the model weights are activated.
One common practice to help avoid this and see what the model does grasp, is to ask it to think step by step, and explain its reasoning. When doing this, you can see the fault in logic.
All that being said, the vision model is actually faulty in being able to grasp the relative position of information, so this kind of task will be more likely to hallucinate.
There are better vision models, that aren’t OpenAI based. For example Qwen-VL-Max is very good, from the Chinese company Alibaba. Another is LLaVA which uses different baselines of open source language models to add vision capabilities
Depending on what you are needing vision for, models can be spiky in capability. Good at OCR but bad at relative positioning. Good at classifying a specific UI element, but bad at detecting plants, etc etc.
Joy: So, I think I can tell the Rotary Club that GPT was “wrong” as opposed to “intentionally creative.” I think, as I originally concluded, you should not make ChatGPT the pilot of your airplane and go to sleep when approaching the Alps. ChatGPT should be used for what it is good at, such as writing the rough draft of a cover letter. (We have great “autopilot” software for flying planes, already, without involving large language models.)
Another expert, Gavin Leech, also weighed in with some helpful background information:
the creativity parameter is known as temperature. But you can actually radically change the output (intelligence, style, creativity) by using more complicated sampling schemes. The best analogy for changing the sampling scheme is that you’re giving it a psychiatric drug. Changing the prompt, conversely, is like CBT or one of those cute mindset interventions.
For each real-name model (e.g. “gpt-4-0613”), there’s 3 versions: the base model (which now no one except highly vetted researchers have access to), the instruction-tuned model, and the RLHF (or rather RLAIF) model. The base model is wildly creative, unhinged, but the RLHF one (which the linked researchers use) is heavily electroshocked into not intentionally making things up (as Lenny says).
It’s currently not usually possible to diagnose an error – the proverbial black box. My friends are working on this though
For more, note OpenAI admitting the “laziness” of their own models. the Turbo model line is intended to fix this.
Thank you, Lenny and Gavin, for donating your insights.
By now, everyone should consider using ChatGPT and be familiar with how it works. I’m going to highlight resources for that.
My paper about how ChatGPT generates academic citations should be useful to academics as a way to quickly grasp the strengths and weakness of ChatGPT. ChatGPT often works well, but sometimes fails. It’s important to anticipate how it fails. Our paper is so short and simple that your undergraduates could read it before using ChatGPT for their writing assignments.
A paper that does this in a different domain is “GPT4GEO: How a Language Model Sees the World’s Geography” (Again, consider showing it to your undergrads because of the neat pictures, but probably walk through it together in class instead of assigning it as reading.) They describe their project: “To characterise what GPT-4 knows about the world, we devise a set of progressively more challenging experiments… “
For example, they asked ChatGPT about the populations of countries and found that: “For populations, GPT-4 performs relatively well with a mean relative error (MRE) of 3.61%. However, significantly higher errors [occur] … for less populated countries.”
ChatGPT will often say SOMETHING, if prompted correctly. It is often, at least slightly, wrong. This graph shows that most estimates of national populations were not correct and the performance was worse on countries that are less well-known. That’s exactly what we found in our paper on citations. We found that very famous books are often cited correctly, because ChatGPT is mimicking other documents that correctly cite those books. However, if there are not many documents to train on, then ChatGPT will make things up.
I love this figure from the geography paper showing how ChatGPT estimates the elevations of mountains. This visual should be all over Twitter.
There are 3 lines because they did the prompt three times. ChatGPT threw out three different wrong mountains. Is that kind of work good enough for your tasks? Often it is. The shaded area in the graph is the actual topography of the earth in those places. ChatGPT “knows” that this area of the world is a mountain. But it will just put out incorrect estimates of the exact elevation, instead of stating that it does not know the exact elevation of those areas of the world.
The first thing to explain is that what ChatGPT is always fundamentally trying to do is to produce a “reasonable continuation” of whatever text it’s got so far, where by “reasonable” we mean “what one might expect someone to write after seeing what people have written on billions of webpages, etc.
If you feel like you already are proficient with using ChatGPT, then I would recommend Wolfram’s blog because you will learn a lot about math and computers.
Scott wrote “Generative AI Nano-Tutorial” here, which has the advantage of being much shorter than Wolfram’s blog.
I don’t have much time to write this week because I lost so many work hours to schools closing for “weather.”
Tyler has been saying that we should welcome more intelligence (in the form of LLMs – I’m not getting any smarter). What would we want intelligence for? How about reducing the error rate on school closing?
First, I will recognize that things are already getting better due to computers. The internet and texting and radar help. Compared to when I was a child in New Jersey, it’s more efficient to text all the parents the night before, as opposed to having people get up at 6am to scan the radio for news. Weather forecasting has presumably gotten better.
Now my rant: Right around what was already a three-day official weekend, school was closed three times. Even my kids were irate when that last day was announced. In my opinion, only one of those closures was justified for extreme weather.
There is a lot of dumb in a city. People complain about routine processes being suboptimal. It would be great if we humans could figure out ways to apply more intelligence to these local problems and make less mistakes.
This is a joke for any readers in cold climates. My Alabama kids thought it was fun to collect icicles because they have almost never seen them before.
In addition to all the usual items for a principles of macroeconomics class, I’m asking my students to listen to one podcast episode this semester. They have to write a short summary on a discussion board for credit.
It took me a bit of time to collect this list of links. I also give them some discretion to find their own episode, but I’m not posting my rules on that point here. This list is something you can copy, paste, and modify. The point is to have all the web links in one place so that students can just click around. There have been many great podcasts over past 2 decades, but I list relatively new content so that we get a bit of “current events” thrown in. So, even if you’ve assigned podcasts before, this new list might be helpful.
Are smartphones bad for kids? Cal walks through the data on this question, including how researchers came to be worried, their findings, critiques of their findings, and where we are today. He then gives recommendations for how to think about technology when it comes to your kids.
In May of 2023, Cal Newport shared well-informed opinions about whether smartphones harm young people. In the first half of the podcast, he talks about depression and loneliness data.
Minute 30 of the podcast: Screentime harms teenagers because they inhibit the development of critical thinking skills. Deep critical thinking skills require training. Reading an analog book is better than screens (see my review of Tyler’s AI generative book and poastmodernism).
See my summary of Emily Oster on video games for kids. She does not clutch her pearls over violent video games. However, she is concerned about what activities get crowded out by screentime. She is especially worried about sleep, because on that topic the data are clear.
Minute 31, Call Newport: Tweens and teens scroll on their phones for too long instead of going to sleep. A 13-year-old boy with a smart phone will “be up until 4 in the morning.” A tween told him that middle school girls arrive at school too exhausted to function because they have been on their phones all night.
FYI, if you are the parent in an Apple device network, you can set time limits on the devices in your family. I filed this report about smart watches last year, incidentally in the same week as the release of Newport’s podcast episode.
One mom told me that Apple Watch is considerably more durable than a Gizmo (I wouldn't know from experience). And, my son described "Watch Jail" where his teacher keeps smart watches overnight if there are usage infractions during the school day.
There is a new generative AI product called Phind. Does it make up fake citations like ChatGPT? I will report experimentation with the free-tier product from January 7, 2023.
Phind is an intelligent search engine and assistant for programmers.
Prompt: “In less than 10 sentences, explain how the ideas of Hayek are connected to bailing out failed banks, and include citations from published papers as a list. include author, year in parentheses, and journal for the citations.”
Friedrich Hayek, a prominent Austrian economist, advocated for a free market system where businesses and individuals make decisions based on individual knowledge and self-interest, rather than central planning by government or bureaucrats. His ideas suggest that interventions, such as bailing out failed banks, should be minimal and temporary, allowing the market to correct itself.
However, the financial crisis of 2008, including the failure of many banks, led to significant government intervention, including the bailout of banks. This action can be seen as a departure from Hayek’s principles, as it involves direct government intervention to stabilize the financial system.
According to a paper by Todd Zywicki (2009), the Federal Reserve’s artificial lowering of short-term interest rates and the resulting shift by consumers to adjustable rate mortgages (ARMs) triggered the housing bubble and subsequent crisis [Source 1](https://reason.org/wp-content/uploads/files/federal_reserve_monetary_policy_hayek_rule.pdf). This aligns with Hayek’s critique of central banking’s manipulation of interest rates.
Moreover, the Basel Accords, international capital adequacy and liquidity requirements for banks, were designed to encourage banks to hold securitized loans in their portfolios, thereby concentrating risk and leveraging up, thereby increasing their exposure to risk. This approach, contrary to Hayek’s preference for decentralization and competition, may have contributed to the financial crisis [Source 1](https://reason.org/wp-content/uploads/files/federal_reserve_monetary_policy_hayek_rule.pdf). In conclusion, while Hayek’s ideas of minimal government intervention and self-regulating markets provide a useful theoretical framework, practical implementation in complex and interconnected modern economies may require different strategies, including bank bailouts.
Are the citations real and correct? Phind simply is not trying to write with in-text citations and academic references. There is a mention of “Todd Zywicki (2009)” and also a weblink. The weblink is to a policy paper published by Reason. This policy paper is on the correct topic and the paper does exist. The Zywicki citations also exists, in the sense that if you go into the Reason paper it cites a real Wall Street Journal op-ed. So, if you follow the trail, you can see what document Phind learned to cite Zywicki from. It seems like it is stealing the citation more than reading the source documents. To be fair, sometimes human academics work that way, too.
What I was hoping for was in-text citations with a compiled bibliography at the end. Phind did not give me that.
I’ll try another test that I ran on ChatGPT and Bing in 2023.
The prompt is “In what paper did John Duffy propose that economists use experiments to study equilibrium selection?”
Phind does not attempt an in-text citation in the response. The link in the paragraph (link number 2) points to John Duffy’s list of papers, which is better than just making up fake citations but also does not create a literature review for me. Phind is, as with the Hayek test above, providing breadcrumbs of links through which I can discover existent papers.
Is there a paper called “The Transition from Stagnation to Growth: An Adaptive Learning Approach”? Yes. And it is by Duffy.
Phind lists weblinks to sources. Has Phind done more for me than Google, on this search? Not much, in terms of finding and synthesizing references.
According to a chart circulating this week, attendance at the AEA winter meeting exceeded 13,000 economists at the peak. I had some job interviews at Chicago in 2017, and I spoke on a panel at San Diego in 2020. Chicago was freezing cold. San Diego was lovely in January. Cold or not, probably every economist on the job market showed up in person to those conferences. There are harrowing stories of people who missed job interviews because of snow and cancelled flights.
A privilege of being in this profession is the chance to meet people from around the world. At my lunch table currently there are people from Chile and Southern China to my right and Kansas and Hong Kong to my left. #ASSA2020
A privilege of being in this profession is the chance to meet people from around the world. At my lunch table currently there are people from Chile and Southern China to my right and Kansas and Hong Kong to my left. #ASSA2020
It is (or was) a huge win for the US to be the meeting place. Now that job interviews have moved to Zoom, attendance has fallen by more than half. I assume that the attendees here in 2024 skew more America-based.
A European economist predicts attendance in 2024 will fall even lower because “the interview stage of the market has moved online, and at a random date between October and January. No point in flying to ASSA, especially if you are a candidate based in Europe hoping for a job in Europe”
Lower, the interview stage of the market has moved online, and at a random date between October and January. No point in flying to ASSA, especially if you are a candidate based in Europe hoping for a job in Europe
Instead of asking why the economics job market unraveled, maybe we should be surprised that it was ever so centralized. People really did travel from all over the world to an American city in January to interview for almost every serious job available in the economics profession.
Small points:
I’ve seen several people complain that it’s a bad time of year. For me, it’s a good time of year. Since I do a lot of teaching, it’s good to have a conference that happens before my teaching starts.
I was happy to see that childcare is available here (for a fee).
I talked to a job market candidate who was trying to weigh a tenure-track job offer from a teaching school against the possible offer of a postdoc. He would prefer the postdoc. Because of the new system, job offers come in sequence. He risks losing the teaching offer before he finds out if he will get the postdoc offer in writing. In the old days, candidates would have been more likely to know what their best option was going to be.
The online JOE job board still serves for coordination. It is still administered by the AEA for jobs globally.
James provided evidence that the AEA will start letting more people get on the program to present research. (By contrast, the Southern Economic Association meeting has actually increased in attendance since Covid. They allow accept more people to present research on the program.)
When my grad school research group went out for lunch, I was sometimes the only American at the table. I loved it. I’m grateful that those brilliant people came here to start their careers.
Back in August I listed the most-read posts of 2023. Here I will finish out the year by listing a few more highlights. This has been another big year for our website.
Zachary has been giving out good advice for economics teachers, backed up by his data. All professors can read: 5 Easy Steps to Improve Your Course Evals. Econ professors check: Update on Game Theory Teaching. It’s about how to teach Game Theory, but I also see it as a testament to how much a course can improve if you allow a teacher to iterate multiple times at the same school. Administrators, take note.
What We Are Learning about Paper Books is jointly my reflection on AI generative books and a review of Tyler Cowen’s new book GOAT. I’m a techno-optimist, but I think there is value in an old-fashioned paper book, mostly from a behavioral or neuro perspective.
I have a paper that emphasizes ChatGPT errors. It is important to recognize that LLMs can make mistakes. However, someone could look at our data and emphasize the opposite potential interpretation. On many points, and even when coming up with citations, the LLM generated correct sentences. More than half of the content was good.
Apparently, LLMs just solved an unsolvable math problem. Is there anything they can’t do? Considering how much of human expression and culture revolves around religion, we can expect AI’s to get involved in that aspect of life.
Alex thinks it will be a short hop from Personal Jesus Chatbot to a whole new AI religion. We’ll see. People have had “LLMs” in the form of human pastors, shaman, or rabbis for a long time, and yet sticking to one sacred text for reference has been stable. I think people might feel the same way in the AI era – stick to the canon for a common point of reference. Text written before the AI era will be considered special for a long time, I predict. Even AI’s ought to be suspicious of AI-generated content, just in the way that humans are now (or are they?).
Many religious traditions have lots of training literature. (In our ChatGPT errors paper, we expect LLMs to produce reliable content on topics for which there is plentiful training literature.)
I gave ChatGPT this prompt:
Can you write a Bible study? I’d like this to be appropriate for the season of Advent, but I’d like most of the Bible readings to be from the book of Job. I’d like to consider what Job was going through, because he was trying to understand the human condition and our relationship to God before the idea of Jesus. Job had a conception of the goodness of God, but he didn’t have the hope of the Gospel. Can you work with that?
Economist Writing Every Day 