The experimental economics world is currently still doing data collection in traditional physical labs with human subjects who show up in person. This is still the gold standard, but it is expensive per observation. Many researchers, including myself, also do projects with subjects that are recruited online because the cost per observation is much lower.
As I remember it, the first platform that got widely used was Mechanical Turk. Prior to 2022, the attitude toward MTurk changed. It became known in the behavioral research community that MTurk had too many bots and bad actors. MTurk had not been designed for researchers, so maybe it’s not surprising that it did not serve our purposes.
The Prolific platform has had a good reputation for a few years. You have to pay to use Prolific but the cost per observation is still much lower than what it costs to use a traditional physical laboratory or to pay Americans to show up for an appointment. Prolific is especially attractive if the experiment is short and does not require a long span of attention from human subjects.
Here is a new paper on whether supposedly human subjects are going to be reliably human in the future: Detecting the corruption of online questionnaires by artificial intelligence
Abstract: Online questionnaires that use crowdsourcing platforms to recruit participants have become commonplace, due to their ease of use and low costs. Artificial intelligence (AI)-based large language models (LLMs) have made it easy for bad actors to automatically fill in online forms, including generating meaningful text for open-ended tasks. These technological advances threaten the data quality for studies that use online questionnaires. This study tested whether text generated by an AI for the purpose of an online study can be detected by both humans and automatic AI detection systems. …
Using Prolific might become suspect in the future because AI will be so good at imitating humans. These are paid surveys, especially the kind economists like me run. Unfortunately, that creates a financial incentive for the bot farmers to join up and collect participation fees.
There used to be a set of attention checks or comprehension questions that a researcher would feel comfortable using to distinguish serious human subjects from bots or otherwise. Going forward, this will be less certain.
Prediction: There will be a lot of interest in the future in pointing AI tools on pre-2023 data, before the world got officially weird and text data was no longer reliably generated by humans.
There is so much human-generated data setting in databases and professors’ offices that no one has had time to analyze yet. The LLMs will be able to do neat things with pre-2023 data. In the medium-term, if LLMs get a little better at literature reviews, there is a lot of potential for AI to do meta-analysis of the existing full scientific literature.
AI agents could potentially do some of the tedious work that RAs used to do to conduct interviews and/or run experiments with human subjects. It’s ironic that what we won’t trust AI’s to do is verify whether the subject is human or not. Via @johnjhorton and @emollick
Economist Writing Every Day 