Introduction to AI Assistants and News Content
Artificial intelligence (AI) assistants are becoming increasingly popular for gathering information, including news. However, a recent study by the European Broadcasting Union (EBU) and the BBC found that these assistants often misrepresent or mishandle news content. The study evaluated the free versions of several AI assistants, including ChatGPT, Copilot, Gemini, and Perplexity, across 14 languages and 22 public-service media organizations in 18 countries.
Key Findings of the Study
The research assessed 2,709 core responses from the AI assistants and found that 45% of them contained at least one significant issue, while 81% had some issue. The most common problem area was sourcing, which affected 31% of the responses at a significant level. The EBU noted that "AI’s systemic distortion of news is consistent across languages and territories." This means that the errors and issues found in the study were not limited to specific languages or regions but were a widespread problem.
Performance of Each AI Assistant
The study found that the performance of the AI assistants varied. Google Gemini showed the most issues, with 76% of its responses containing significant problems, driven by 72% with sourcing issues. The other assistants performed better, with 37% or fewer of their responses containing major issues overall and 25% or fewer with sourcing issues.
Examples of Errors in AI Responses
The study found several examples of errors in the AI responses, including outdated or incorrect information. For instance, several assistants identified Pope Francis as the current Pope in late May, despite his death in April. Gemini also incorrectly characterized changes to laws on disposable vapes. These errors highlight the need for users to verify the information provided by AI assistants against original sources.
Methodology of the Study
The study was conducted between May 24 and June 10, using a shared set of 30 core questions plus optional local questions. The researchers focused on the free versions of the AI assistants to reflect typical usage. Many organizations had technical blocks that normally restrict assistant access to their content, but these blocks were removed for the response-generation period and reinstated afterward.
Implications of the Study
The findings of the study have significant implications for users of AI assistants and for the media organizations that provide news content. When using AI assistants for research or content planning, it is essential to verify claims against original sources to ensure accuracy. The high rate of errors in AI responses also increases the risk of misattributed or unsupported statements appearing in summaries that cite specific content.
Looking Ahead to Improved AI Assistants
The EBU and BBC published a News Integrity in AI Assistants Toolkit alongside the report, offering guidance for technology companies, media organizations, and researchers. The toolkit aims to help improve the accuracy and reliability of AI assistants in providing news content. The EBU’s view is that growing reliance on assistants for news could undermine public trust if the issues found in the study are not addressed. As EBU Media Director Jean Philip De Tender put it, "When people don’t know what to trust, they end up trusting nothing at all, and that can deter democratic participation."
Conclusion
In conclusion, the study by the EBU and BBC highlights the need for caution when using AI assistants for news gathering and research. The high rate of errors and issues found in the study emphasizes the importance of verifying information against original sources. By understanding the limitations and potential biases of AI assistants, users can take steps to ensure the accuracy and reliability of the information they gather. As the use of AI assistants continues to grow, it is essential to address the issues found in the study to maintain public trust in news content and promote democratic participation.