Economist Writing Every Day

A couple years ago, my Co-blogger Mike described his productive, but novice intern. The helper could summarize expert opinion, but they had no real understanding of their own. To boot, they were fast and tireless. Of course, he was talking about ChatGPT. Joy has also written in multiple places about the errors made by ChatGPT, including fake citations.

I use ChatGPT Pro, which has Web access and my experience is that it is not so tireless. Much like Mike, I have used ChatGPT to help me write Python code. I know the basics of python, and how to read a lot of of it. However, the multitude of methods and possible arguments are not nestled firmly in my skull. I’m much faster at reading, rather than writing Python code. Therefore, ChatGPT has been amazing… Mostly.

I have found that ChatGPT is more like an intern than many suppose:

1. Time of day matters. If I start using it at 8 AM on the East Coast, it’s fast and precise. However, similar prompts at 4 PM Eastern time are slow, buggy, and lazy. Maybe this is when all of the US time zones are trying to use it?
2. There’s reading and then there’s “reading“. Say that I’m hitting a simple error in python due to a mistake in my code. I can upload the entire file to ChatGPT, which will read it. But it is clearly cutting a lot of corners. It will say things like “you probably have some code somewhere similar to… “. A tireless intern would know precisely what code I have and don’t have because they have the file! Or it will tell me to make sure that I have imported a Python module. Given the code, it should know whether I have! ChatGPT is not the same as a standard computer. This is where the LLM methodology and computational limitations shine through. It will try to get me to do work rather than bearing the computational load.
3. Prompt noncompliance. Sometimes ChatGPT just thinks that it knows better. I might tell it to summarize an idea without using any commas and avoiding words that end in ‘ed’. Such a style makes clear who the subject is and helps maintain active voice. It often disobeys/fails without explanation. It seems to ignore the parts of prompts that deviate from popular requests.
4. Meandering answers. Sometimes I will ask a ‘yes, or no’ question, and I get paragraphs of answer. Like an intern, ChatGPT does not know what I think is important. Obviously, it can’t pick up on social cues. So I have to remind it periodically to update its memory in order to keep responses extremely concise so that I can quickly move on or probe if I need to.
5. Short term memory. Have you noticed that the longer a chatGPT conversation persists, the worse it gets? It’s like the student who gets so distracted by doing the math in a problem, then they forget what their goal is. Or, if you provide a premise, then chatGPT eventually forgets it — sometimes after not so long. When web scraping with selenium, I told it to never use webdriverwait because it’s buggy and interacts poorly with many website features. Even so, it still suggests it.

Much like an intern, I find that ChatGPT performs better if you hold its hand. It will fail to understand large chunks of code, but if walked through each line, it can do just fine. Similarly, like economists, it’s great at telling you “no”. So if you type your own version of some Python code and ask if it will do X, then it will tell you why your code won’t achieve the goal, and then provide an alternative. It’s much better at correcting you than writing its own code from scratch.

For me, ChatGPT is very much like having an intern. If I need a second set of eyes, then I have them. If I want a customized Wikipedia article about a specific feature of a particular topic, then I can have it. With the web enabled version, I can see the citations and the terminology used among specialists in order to explore more on my own. 

But if interns and AI are so similar, then why use ChatGPT? It’s the breadth and speed. For many scholars, human interns are too slow, or can only understand a subset of a prompt.  ChatGPT can handle the fire hose and the variety of targets quickly.