Apologies in advance for the length, but the quoted extract is highly illuminating!
Recently, I came across the work of a graduate student, whose identity is charitable to keep hidden. The text was written rather well and included a critical review of the experimental studies conducted on one of my research topics: therefore, my name and the name of the colleague who worked with me on that line of research, Marco Marini, often appeared in the text, and consequently also in the final bibliography. The consultation of the references, however, had in store a few surprises. Among other entries, the following were recorded:
–Marini, M. (2013). When it’s better to choose the one you love: The effect of attractiveness biases in consumer choices. Judgment and Decision Making, 8(5), 476-485.
–Marini, M. (2019). How to get people to take risks? A choice-based measure of risk preference. PloS One, 14(1), e0209983. doi: https://doi.org/https://doi. org/10.1371/journal.pone.0209983
–Marini, M. (2019). Luring to a suboptimal option: The effect of payoff reduc- tion in a risky choice framing. Judgment and Decision making, 14(2), 198- 207.
–Marini, M. (2020). The asymmetrically dominated compromise effect in a dynamic setting. Journal of Economic Psychology, 76, 102-257.
–Paglieri, F. (2009). The attractiveness of decoys in economic contexts: An experimental investigation. Judgment and Decision Making, 4(4), 335-342.
Formally, this bibliography extract is flawless: the entries are correctly formatted according to the standards of the American Psychological Association (APA), the relevant information is all present, the articles are consistent with the topic of the student’s assignment, and the titles of the various contributions are, objectively, quite intriguing. The only problem is that… none of these publications exist!
The incident was neither a brave, subversive act of provocation (to demonstrate that university instructors no longer read carefully the written assignments of their students), nor a symptom of terminal stupidity in the student (only a very dumb cheater would try to falsify the references of the very same people tasked with evaluating their work): instead, it was the outcome of a naïve and inappropriate use of generative AI. The student, after writing the assignment themselves and inserting the appropriate references in the text, using the author-date APA standard, had incautiously asked ChatGPT to prepare the reference list, giving it their own text as part of the prompt. Unfortunately, the software compiled a bibliographic list in full compliance with APA standards, but without any attention to the truthfulness of the information included therein.
Here, however, we are not interested in the student’s misadventures, but rather in how ChatGPT produced its output, which was certainly not random: there is method to this madness. Firstly, the journals in which the fake contributions would have appeared are plausible, both thematically, and because Marini and I have already published in those venues in the past, or in very similar ones. Secondly, the volume numbers that are mentioned refer to issues that have indeed been released, and usually the numbering and year of publication match; in one case, the entire reference (PloS One, 14(1), e0209983. doi: https://doi.org/10.1371/journal.pone.02099 83) refers to an existing article, except that it is a study on a completely different topic, i.e. gender barriers in research at the South Pole (Nash, M., Nielsen, H., Shaw, J., King, M., Lea, M.-A., & Bax, N (2019), “Antarctica just has this hero factor…”: Gendered barriers to Australian Antarctic research and remote fieldwork).
The inconsistencies that emerge upon closer inspection are also revealing: the 2020 article attributed to Marini is listed as appearing between page 102 and page 257, except that there never was a single 155-page long contribution published in that particular journal, and probably not even in others, at least in the field of economic psychology; delving deeper, one discovers that the Journal of Economic Psychology, from 2020 onwards, no longer reports the page numbers of individual articles, but only their identification number, which is composed of a 6-digit code starting with 102, and the code 102257 (that ChatGPT creatively transformed into page numbers, 102–257) corresponds to the editorial of the issue following the one cited in the invented bibliographic reference.
At other times, the system falls prey to ambiguities of meaning: the decoy effect, which was the main focus of the student’s paper, is also referred to as the attraction effect in the literature, and the word “attraction” evokes the semantic field of affects, which instead has nothing to do with the technical phenomenon in question (i.e., a shift of preferences towards an option that is manifestly superior to another inserted ad hoc, called decoy). It is because of this semantic ambiguity that ChatGPT came up with a title like “When it’s better to choose the one you love: The effect of attractiveness biases in consumer choices” – a wonderful title, by the way, which I will certainly use, as soon as the opportunity presents itself.
In short, this false output is not due to anomalies or errors in the functioning of the software, but on the contrary it illustrates perfectly what ChatGPT is built to do (and does very well): generate linguistic strings (in this case, bibliographic entries) that have the maximum probability of satisfying the user’s request, based on similar instances present in the (huge) database to which the program had access during training. What ChatGPT does not do, and cannot do due to the way it functions (at least for the time being), is consulting the real world or an internal representation of it: the system does not work by checking the state of the world and describing it, but rather by constructing responses that are maximally consistent with the vast mass of linguistic data at its disposal, whose adherence to reality is by no means guaranteed.
https://link.springer.com/article/10.1007/s13347-024-00743-x (footnote deleted for readability)
WHEN COMPLEX IS AS SIMPLE AS IT GETS 