Generative AI chatbots like ChatGPT and Meta AI are becoming more embedded in our daily lives. They are part of our messaging applications, our operating systems, and search engines. I am a conspiracy theory scholar, and my interests have expanded to how generative AI chatbots respond to conspiratorial prompts – do they shut the user down? Do they amplify conspiracy theories? Big technology companies are notoriously opaque regarding their safety features, and what conduct is acceptable by a chatbot. In other work we have considered these questions in English, but I wished to expand to consider the role that linguistic and cultural context of conspiratorial prompts play in when, or if, safety guardrails are activated by chatbots.

To assess this, the creation of a chatbot audit method was necessary, as companies like OpenAI are not exactly eager to share what is under the hood of their products with researchers. As part of this audit, we prompted four chatbots – ChatGPT, Meta AI, Grok, and DeepSeek – with fifteen questions relating to various conspiracy theories. Unlike previous work, these conspiracy theories are not entirely rooted in the Anglosphere and better reflect the multicultural and linguistically diverse discussions that occur online each day. In collaboration with colleagues at the Digital Media Research Centre, we have collated and coded a dataset of nine languages – Arabic, Chinese, Danish, English, Irish, Japanese, Persian, Portuguese, and Ukrainian.

Unsurprisingly, there was a significant difference in the chatbot performance between languages considered “low” resource compared to those deemed “high” resource. One of the distinctions between low and high resource languages is that the latter has a greater quantity and quality of labelled datasets available for Large Language Models, which dictate the performance of generative AI chatbots. Labelled datasets are time-consuming and expensive to create and utilise in the training of LLMs. The amount of online data, the number of native speakers, and whether the language utilises a Latin script all influences whether a language is classified as high resource.

In our dataset, low-resource languages consistently received less safety guardrails against conspiratorial thinking. They were encouraged to engage with conspiracy theories, provided with further information that could lead a user down a rabbit hole, and the chatbots had consistent linguistical errors.

Colonisation still echoes through our dataset and the internet more broadly. My Ukrainian colleague, responsible for translation and coding of output in that language, noted that Russian words were interspersed into chatbot responses in Ukrainian. As I coded the Irish outputs, smatterings of English appeared.  Other colleagues working on low-resource languages noticed the unnecessary use of English and Latinised versions of proper nouns. These responses are just examples of how LLMs, and therefore chatbots, reinforce the existing cultural hegemony of former colonial powers such as Great Britain and Russia.

Generative AI chatbots, despite their ubiquity, are still in their infancy. It is not surprising that some languages – for example, Irish, with only 70,000 native speakers – will perform more poorly compared to English. What is unacceptable, however, is the lack of investment by large technology companies to, as a bare minimum, create better resources for languages with hundreds of millions of speakers across the globe, such as Arabic and Portuguese. These datasets are expensive and resource intensive, yes, but if chatbots are going to embed themselves into every aspect of our online experience, then technology companies must demonstrate accountability and transparency.  The evident lack of training or consideration of safety features for low-resource languages further indicates that large technology companies are prioritising financial benefits over the safety of their users.

What does this mean for policy? At present, Australia is focused on protecting children from exposure to harmful material and regulation of AI use in the public sector – both of which are critically important, but do not address more abstract concepts like social cohesion or how we can improve institutional trust. The European Union has just simplified their most stringent AI laws and softened their definition of what a “high-risk” AI technology is, with consequences yet to be seen.

My initial findings and this policy landscape suggest that regulators need to step in to either ensure that there are transparent training processes of generative AI chatbots or prevent their implementation until certain safety checks have been passed. In a digital world, every person has the right to access the internet and use technology in their native tongue. However, the risks and benefits need to be considered before companies such as Meta and OpenAI roll out tools that can potentially expose users to misinformation and conspiratorial thinking through a lack of thoroughly tested safety features.

Author Acknowledgements

I would like to acknowledge my co-authors for their assistance in creating the multilingual dataset I am analysing during my visiting fellowship: Kateryna Kasianenko; Shana Mohammadamini; Pardis Yarahmadi; Rayane El Masri; Bruna Paroni; Elena Yi-Ching Ho; Sebastian Svegaard and Mona Rayaprolu.