research

We wrote “ChatGPT Cites Economics Papers That Do Not Exist”

I expect that problem to go away any day, so I gave it another try this week. For the record, they are currently calling it “ChatGPT Mar 23 Version” on the OpenAI website.

First, I asked ChatGPT for help with the following prompt:

ChatGPT is at it again. There is no such paper, as I will verify by showing John Duffy’s publications from that year:

ChatGPT makes up lies (“hallucinations”). It is also great for some tasks, and smart people are already using it to become more productive. My post last week was on how impressive ChatGPT seemed in the Jonathan Swift impersonation. I didn’t take any time to do fact checking and I would bet money that at least something was made-up-facts in there.

I posed the same question to the Bing plug-in for the Edge browser (Microsoft). Yup, I have opened Edge for the first time in forever to use Bing.

Bing handles the prompt by linking to a useful relevant paper – so if you click the link you will get to a helpful and not misleading answer. Just being a smart search engine instead of hallucinating randomly is better, for my purposes.

The actual paper I wanted returned was this one, by the way:

Duffy, John. “Experimental macroeconomics.” Behavioural and Experimental Economics (2010): 113-119.

There is no reason that ChatGPT should be better than an expert in a subfield of a field of economics. But that’s the genius of a good search engine. You ask it “Can I repair a broken fiddlewhat?” The search engine does not claim to know but rather directs you to the blog of the world expert in fiddlewhats.

I can’t find the link to it, but I’m going to toss in one more thing here. Tyler Cowen did an interview this Spring on AI. There was a newspaper reporter who had a “creepy” interaction with an AI that made for the topic of a viral internet article. Tyler made a very contrarian point by saying that he interprets this as a case of AI alignment. The reporter wanted something sensational and he got what he wanted.

So, it will probably be true for a long time that if you want to find a failure of AI, you can get what you want. Still, I’m putting this on the record here because I wonder if this particular problem will get solved quickly.

My co-blogger Mike Makowsky has a thoughtful post this week about the academic publishing process. I wanted to offer a slightly different perspective on the same topic. But my perspective comes from someone who is not at a research university, and someone who has recently survived the tenure process.

A little background for those not completely familiar with the academic world: schools are usually considered either teaching or research schools. At first this seems confusing: both Clemson (where Makowksy is) and the University of Central Arkansas (where I am) require that faculty engage in both research and teaching. The difference is subtle, but the big hint is that Clemson is considered an “R1” school (the highest research designation) and has a PhD program with many graduate students. At a school like Clemson, research is valued more than teaching. At UCA, teaching is valued more than research. (Much more could be said about the differences, perhaps in a future post.)

We both engage in both teaching and research (as well as service!), but the emphasis is different. For me at UCA, the expectations of which journals I will publish in and how frequently I will publish are lower than at a school like Clemson. At Clemson, some of your publications should be in the Top 5 (or at least Top 10) journals from time-to-time. At UCA, if you published in one of the top journals, the assumption would be that you are probably leaving soon to go to an R1 school

I’m glad both types of schools exist, and my point here is not to disparage either type of school. But the difference is important for thinking about the academic publishing process.

For someone at an R1 school, publications in top journals are positional goods. Makowsky doesn’t say this exactly, but that’s my takeaway from his post. There are only so many spots available in these journals, and they have value because there is only a fixed number available. And since there has been, over the years, a lot more economists doing a lot more research not all of the great papers will end up being published in one of the top journals.

Upshot: there are a lot of great papers being published in Top 50 or even Top 100 journals! Let me pick on myself. As I said, I recently successfully survived the tenure process. My publication record was good enough. You can inspect my publications over at Google Scholar. I’m proud of these publications. I think some of them are really great. But I’m fairly confident that I would never earn tenure at Clemson with these publications. Instead, you need a publication record like Makowsky.

What’s interesting here is that Mike and I occasionally publish in some of the same journals. Public Choice and Constitutional Political Economy jump out to me. These are, in my view, very fine journals. Lots of interesting research is published in these journals. I’m especially proud of this paper in Public Choice. But if someone published only in these two journals and journals like them, they wouldn’t get tenure at an R1 university.

So what do we do with this information?

Continue reading →