What is generative AI?
Generative AI is a type of artificial intelligence that can produce new text, images, audio, and video in response to user prompts. It includes tools like ChatGPT, Google’s Bard and Bing’s Chatbot.
Generative AI creates text that mimics human writing. It is a type of machine learning based on massive datasets that follows language patterns by predicting combinations of words that are likely to occur together. However, the ability to follow patterns is not the same as the ability to discern facts. It does not actually understand the context or the meaning of the text it creates.
Other resources from UBC:
The same principles for evaluating information sources apply to generative AI. Tests such as the SIFT and RADAR tests can be helpful in determining if the information you’ve found is reliable.
However, some of the questions we typically ask ourselves about sources may be more difficult to answer when consulting generative AI, because the process it takes to arrive at answers is not public.
So how can you assess the information generative AI gives you?
Look for other reliable sources to corroborate the AI’s claims. Try to find alternative sources that cover the same topic, or even the original context that a claim came from (these are principles F and T of the SIFT test).
Check citations for hallucinations:
You can ask a generative AI tool to cite its sources, but it is known to create very convincing fake citations.
It can even create citations that have the names of real researchers who study the topic you've asked about. However, the article named in the citation might not exist or may not be from the journal it cites. These invented citations are referred to as “hallucinations.”
You’ll need to search to confirm these articles actually exist. For information on finding sources from their citations, see:
Look up the source article and check the information the generative AI tool claims to have found in it. AI is not built specifically to cite truthfully and accurately, so it may name an article that does not actually contain the information.
Currency (when a document was created, edited, updated, or revised) is an important factor in evaluating any information source. If you need recent information on a world event or a new development in research, generative AI may not have that information in its dataset. As of October 2023, if you ask ChatGPT (GPT-3.5) how recent the data it’s trained on is, it will tell you that its information comes from 2021 and it does not have the ability to pull current information from the internet.
As more text is published that has been created by generative AI, eventually this AI-generated content will enter the training datasets for new generations of AI. This may lead to a decrease in the quality of the data, as errors in early generations of AI may compound themselves over time.
This idea was proposed and tested by Shumailov et al. (2023) in their paper “The Curse of Recursion: Training on Generated Data Makes Models Forget.” They found that the inclusion of AI-generated content in training datasets led to what they call model collapse - "a degenerative process whereby, over time, models forget the true underlying data distribution, even in the absence of a shift in the distribution over time" (p. 2).
Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., & Anderson, R. (2023). The Curse of Recursion: Training on Generated Data Makes Models Forget. ArXiv, abs/2305.17493.
The LibrAIry, a team led by librarians from McGill University, have created the ROBOT test to help evaluate the legitimacy of a particular AI technology. It stands for:
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
To cite in APA: Hervieux, S. & Wheatley, A. (2020). The ROBOT test [Evaluation tool]. The LibrAIry. https://thelibrairy.wordpress.com/2020/03/11/the-robot-test
Fernandez, P. (2023). Some observations on generative text artificial intelligence’s impact on libraries part 2. Library Hi Tech News, 40(5), 1-5. doi:https://doi.org/10.1108/LHTN-05-2023-0080
Hervieux, S. & Wheatley, A. (2023). Artificial Intelligence [research guide]. https://libraryguides.mcgill.ca/ai/home
Kim, B. (2023). ChatGPT and generative AI tools for learning and research. Computers in Libraries, 43(6), 41-42. Retrieved from https://www.proquest.com/trade-journals/chatgpt-generative-ai-tools-learning-research/docview/2830977489/se-2
Mairn, C., & Rosengarten, S. (2023). Helping students navigate research with Al TOOLS. Computers in Libraries, 43(7), 22-26. Retrieved from https://www.proquest.com/trade-journals/helping-students-navigate-research-with-al-tools/docview/2869934028/se-2
Scheelke, A. (2023). AI, ChatGPT, and the Library [research guide]. https://libguides.slcc.edu/ChatGPT/InformationLiteracy